The people who empty the machines don't touch the cash either, all they do is withdraw the full hopper from the machine and put a new one in. I've watched them do it on the S&B machines on the T&W Metro.
They don't even need to count up, because the TVM does that too. Human interaction with the cash is basically limited to the passenger who puts it in and, if applicable, a passenger who gets change after a purchase.
If a TOC were worried about the safety of cash handling, the TVM is the last thing you'd turn off. Which makes me think most of it is about saving money.
FWIW I agreed with bus companies deciding to go cashless. Bus Vannin did here. But, crucially, Bus Vannin made their pre-paid smartcards free of charge (usually they cost two quid) and accepted cash at their offices where you could buy or top up a pre-paid card.