A good friend of mine used to work for Oxford City. You do have action you can take.
I have been on a bus where it was unloaded due to bad behaviour. I know the behaviour Bus Drivers go through. It makes my job look like unicorns and rainbows.
I feel your pain but you can stop the bus and not continue if you wanted. You can call the police to attend (as happened to my good friend)
One of the best things I like about being a Train Driver is that I have little to no interaction with the public any more. Some passengers are awesome , friendly and understanding. Others are vitriolic.
I've seen Drivers flip out at passengers and they are still working. I have also seen a Driver quit his job because he couldn't really take it any more
If the safety of the bus was compromised I would have stopped but I didn't feel that was the case and obviously there were other passengers to consider.
The golden rule, just remember any abuse is not (normally) anything personal it is aimed at the company that you work for.
In my experience about 98% of passengers were fine.