Refuse to leave the bus, point out that you’re willing to call police and wait for them. Casually mention that the driver will get home faster by finishing their route than forcing you to waste a lot of his time in the process of getting him fired.

Tl;Dr don’t put up with that shit. Ever.