That would be the case if people are rational. However, people are not rational, and don't like not being in control, even if the chances of them dying are far lower. It may be safer to share the road with self-driving cars, but it's still scarier to rely on a machine that you don't understand.
The most important barriers to adoption are social, not technological, and that means that even one major catastrophe can cause a wave of backlash, regardless of whether or not it's justified.
I agree that people are not rational and will fear the outliers and that the biggest barriers will be social. I'm hoping that there are very few glitches along the way, since we'll be safer if this happens. Most people feel confident about their own driving, but I think they also remember a lot of the close calls they've had.
I think—and hope—we'll be surprised at how quickly we acclimate to magic. Sure, we'll be completely freaked out by letting go of the steering wheel. Then we'll be irritated at how our car thought the leaves blowing across the road were an obstacle. Finally, we'll wonder how we ever survived our own driving and we'll mourn for those that didn't.
The most important barriers to adoption are social, not technological, and that means that even one major catastrophe can cause a wave of backlash, regardless of whether or not it's justified.
And that's why it needs to be rock solid.