When setting back, you need to check, among other things, that any points are set correctly (trailing and facing), that no other trains are moving that might conflict, that nobody is in the way or might potentially be in the way (including trespass), that there are no obstructions likely to damage the train or track (if the train has used the emergency braking then it could have dislodged something), and that the rear of the train does not stop foul of anywhere it's not meant to.
Bear in mind that Signaller, driver and shunter all need to have a clear understanding of what each other are doing and what the train is doing and why. If somebody's job involves driving a different train they can't be fully focused on the move in hand. This isn't like waving a mate into a tight parking space. The driver of the train will not know anything is wrong for a long time, as (s)he has 600-800m of train between her/him and the problem. Some of these problems might be on tolerances of millimetres, such as the wagon scraping the platform edge - it might be fine, it might just be a scrape, but it could damage a really vital component or bump the coping stone out of gauge and that then tears a Titanic-style hole in the next LO unit.
Even at 5mph, A 3,600hp loco pushing 1600-1800 tonnes will do damage if things are in the way, and things are most likely to be in the way around the margins, where you need to be close to it to check whether or not it's safe to continue. There's a reason these things are tightly controlled - do them wrong and you'll damage, hurt or kill things.