The fundamental difference is that a smartphone will record and send the data remotely only when you have applications or malware to that effect. So you still have nominal control.

Smart speakers are sending the audio to the mothership by design, it's the way they work in normal operation.