At the most general level I have often reflected on just how 'bespoke' the rag bag of railway routes was many years ago. Various routes perhaps had very lightly laid track, tight loading gauge, sharp curves, punishingly steep gradients (with issues for both traction going up and braking coming down), smoke problems in tunnels, some areas remote from sources of good coal at affordable prices and so on. So 'standard' designs often just couldn't fit or work, or else were very 'difficult' or inefficient if they could go there.
So there would a constant demand from the operating and civil engineering functions for specific designs with shorter fixed wheelbases, more power, better acceleration, lighter weight, more water capacity, ability to burn 'slack' coal or whatever.
From another angle, in the absence of computer modelling and design, and general 'mass production' it was actually very hard to know exactly how any new design would work out in practice. Hence the 'let's tweak the boiler pressure/diameter, wheel sizes, piston stroke, valve gear, etc.' syndrome to see if it gave better results.