I am still at a loss as to why this happens.
In the "bad" old days, the rolling stock engineer built trains that conformed to C1, W8 etc, and the civil engineer made sure a certain route conformed to C1, W8 etc. No need for every new type to be passed for each route.
What went wrong?
Greed, with a hint of bad specification. Acknowledging that PG1/2/3 are supposed to address this, every few years someone buys a train that pushes the boundaries - class 158, class 373, IEP and quite likely HS2's Conventional Compatibles are all culprits. This is all exacerbated by NR not really having a grip on their infrastructure either, since VE became a god to be worshipped.
The W series is all about freight - in theory all passenger trains should fit a W6A route but it just isn't true: there are W12 routes cleared for 373/2 and there also ones that aren't ... not that 373/2 will ever operate in the UK ever again.
Summary - the train builders don't have to pay to tweak the infrastructure, so why would they bother with compromises?