I wonder if there is any defined point whereby someone decided that any door open in traffic was "not allowed". I remember in my youth class 150/1s regularly used to arrive/depart with the guard hanging out of the door, same with loco hauled sets into the 2000s and 2010s, various EMUs and before they were modded class 156s and that is without looking at driving cabs.
I don't however remember any point where someone said this "this should stop" uniformly across all operators or indeed a point where it did - as usual whether they're meant to or not it has been a piece meal change over the years.
I do remember in the aftermath of the James Street incident some of the documentation from the RSSB commented on the design feature on the 313/314/315/507/508 EMUs of having a non interlocked cab door to allow the guard to observe departure, but at the same time the comment was made that this was seen to be an unacceptable risk on the modern railway so it was forbidden.
I do remember a number of years ago a notice case item went out regarding being vigilant about leaving the rear cab door open on HST sets owing to a number of trains having to be stopped for it to be shut and the concern it caused platform staff, but even then there was nothing about the front cab. Class 222 if I remember rightly will give a message on the TMS if the *back* cab door isn't correctly shut but will quite happily fly along at 125 mph with the front cab doors open which suggests to me some thought was given to that being acceptable back in 2004 when they were entering traffic.
I have noticed it declining in frequency over the years though as the older hands have departed. As you say I've noted several Northern 195s failing to depart because the cab door is still slightly open so it must be on the interlock there.
It's a very good question, and having worked Mk1s and the like in the past I'm all too familiar with the benefits of having a sturdy guards' door to aid visibility of the dispatch corridor. It's not dissimilar from the question that was posed on here a bit back about drivers having to stop now for token exchanges.
In my opinion, there's a distinction to be made between an open cab/local door which is being 'supervised' if you will, ie, has a conductor standing at it, and one that is just open because it has not been closed. The former, whilst still being a potential risk of falling out, is only going to be open at relatively low speeds and as there's a platform next to it there won't be a significant distance to fall. The big risk with having an open door on departure is people on the platform running late being tempted to join in, and a sturdy conductor blocking the way is going to reduce some of that risk but does also introduce the risk of the conductor being bowled over by a panicking customer trying to board post-dispatch (back when we had slam door trains it was bad enough with laterunners trying to board through the droplights).
It's also a case of balancing the risks of the procedure with the risks you're trying to minimise and a lot of work in recent years has gone into reducing the effects of crowding (eg by relocating station furniture to less busy bits of the platform, incentivising customers to stand in bits of the platform further away from stairs etc) which can push people towards the platform/train interface, and how we influence customers behaviour at stations and on platforms (eg highlighting the risks of alcohol around the railway, visible staff to act as a deterrent to horseplay, 'lend a helping hand' type campaigns so people aren't struggling with luggage at stations which can lead to a perception they need to rush) - if they're not too close to the edge in the first place then they can't fall between the train and the platform.
I would expect, though I haven't looked, that what's happened is the rule has been there for a while to say 'local' doors shouldn't be open but for many years it's not really been enforced, and as safety culture has developed the industry has felt more confident in challenging it and asking whether actually it's what we should be doing. Additionally more and more conductors now will be joining the industry learning interlocked power door stock and then when faced with a train where you can theoretically lean out of while it's on the move, this isn't seen as something that they would have any reason to do. Indeed I often see newer conductors not even opening cab droplight windows departing stations now.