For long trios that's what I do. I wear headphones to listen to music and audio books so I don't disturb my family. There's no way to tell Android Auto to only use my headphones, despite it being asked.