No you're quite right, carriage keys are not controlled at all and indeed are lost on a worryingly regular basis by the numerous people who have them. As well as traincrew this includes various people around stations and even on board trolley stewards. They should, in my humble view, be replaced in all security/safety sensitive applications such as cab doors and control panel covers, with something far more secure and issued on a recorded basis. They'd still go walkabouts of course, but on a far less frequent basis if the people responsible for carrying them were obligated to take more care, perhaps encouraged to do so by a substantial cost for replacement items.
A colleague of mine told me a deeply worrying tale a while back, concerning a couple of what you may well term 'scroats'. These two gents were making sufficient nuisance of themselves for arrangements to have been made to stop the train at a signal outside it's next scheduled station call, in order to ensure that the police were present on the platform when the service pulled in. As the Guard is speaking with the Driver regarding communication from the Panel box, the train's doors are suddenly released and the scroats leap down onto the cess and scarper. Turns out they had a carriage key in their possession, and were aware of where to find the release controls and which buttons to hit. Needless to say they didn't bother with the relative complexity of a local door procedure, they just hit the release and popped the whole lot. Thankfully nobody else chose to open up any other doors