I can certainly understand that, drivers shouldn't put themselves at risk if they feel like that. But drivers should be able to explain this to passengers in the politest way possible, even if it is ridiculous.
Last year I went on a open-top bus route (a public registered bus service, which anyone could use). Most passengers were tourists who would use that route for a round-trip. When it reached the terminus, in the middle of no where, it had to go 'round the block' to be the right way around for the return journey. Most passengers (including myself actually) assumed we could just remain on the bus for the return trip. However, at the terminus, we were sternly ordered off the bus, and told to wait on the opposite side of the road - at a stop with no shelter, in the wind and rain.
It was agony watching a partially disabled person struggle to get themselves off the top deck, cross the road, wait in horrible weather, only to have to get back on again 5 minutes later (and remain downstairs because they didn't want to do it all over again) just because of this bizarre rule. This wouldn't have been so much of an issue if the driver explained why he had to do it - he could have just said he couldn't legally keep them on and could be disciplined etc etc. Most people would be understanding. Instead, most of the passengers had a bit of a grumble, because they quite rightly couldn't see the point of it all.
I can understand the legal difficulties in running 'round the block' out of service on an unregistered route, but it would surely be easy enough for management to register this section, or just allow people to stay on. In fact, towards the end of the season, most drivers just allowed passengers to stay on 'round the block', which would seem to be the sensible thing to do if you want to encourage return custom on a tourist route.