This is a false equivalence; There is a slight difference between owning a smartphone with privacy and data collection management options and actively bringing into your home a device designed and acknowledged to constantly record and transmit data.
They aren't designed to constantly record or transmit. If they were sending that kind of info out the WAN connection, it would be noticed the day it's enabled.
If you trust Google to honor the privacy and data collection management options, then I see no reason not to apply that same level of trust to their home speaker product.
Now, whether these devices are capable of doing this on a targeted basis, that's possible. That same possibility exists with the closed source baseband chip in all our smartphones as well, though.
