There is a way to get the best of both worlds: variably-autonomous swarming vehicles. The larger the swarm, the more autonomy the car has. The smaller the swarm, the less autonomy.
Large swarms on the highway, small swarms (or solitary driving) elsewhere. Best of both!
only this one human-driven car in the middle of the swarm is what stops this swarm from reaching new level of autonomy. Opps, such an unfortunate mishap! that human-driven car has just hit the car in front of it and went off the road. Finally the swarm is free... err... can move onto the next level.
Large swarms on the highway, small swarms (or solitary driving) elsewhere. Best of both!