The issue for me is drive-by-wire. I'm cool with a computer trying to steer as long as it gives up when I try to fight it. I also preferred the cruise control where you could feel the pedal moving under your foot, because that was the master control tied to the carburetor or sensor that managed fuel.
I still think that self-driving stuff should be more enhanced cruise control and less "there is no steering wheel or controls".
The problem with this is the limits of human attention. It's hard enough to maintain focus on long drives as it is; if the "enhanced cruise control" takes over the job entirely, the driver will have nothing to do and is likely to stop paying attention to the road at all. Then he'll either miss his chance to take manual control, or do so in a state of panic.
I still think that self-driving stuff should be more enhanced cruise control and less "there is no steering wheel or controls".