It happens on LU largely because the busiest times *tend* to be heavily dominated by regular commuter-type users, who in the main are fairly well organised and know what to do. That said, there's plenty of locations on LU where 20-second dwell times are impossible even with regular users - Bank Northern Line was one such example, there's plenty of others.
Mainline trains used to be just as efficient in slam-door days, something like a VEP could manage to load or unload a massive amount of people in a very quick dwell time, on relative numbers of people this would probably have given LU a run for their money back in the day. Indeed a good proportion of people would be off the train before it had even come to a stand, this practice was essentially standard well into the 1990s and nobody batted an eyelid.
Even if we ignore the safety aspects, I don't think slam-door trains would work nowadays. Slam-door HSTs were absolutely painful in their last days, especially when on frequent-stop work like in Cornwall - mainly due to people failing to close the doors behind them, which requires a level of spatial awareness many people nowadays seem to lack. LU isn't all milk and honey either, things aren't as slick nowadays as they were in the days of guards, where it was standard practice to have the doors opening the instant the train stopped (if not sooner), and closed again at the earliest possible moment.
The mainline transitioned to sliding doors mainly at a time when passenger numbers were comparatively low, and things have largely remained the same ever since, despite passenger numbers increasing massively. This is probably the answer to the question.