The all of two or three HSTs a day in either direction many years ago were not calling at Worcestershire Parkway, Pershore, Honeybourne, Hanborough and Reading - and there were also far fewer trains all the way to or from London to get in their way. The substantial sums of money generated in fares from the intermediate stations between Worcester and Oxford are what allow Worcester to have the frequency of services to and from London it now enjoys, so given a choice between fast trains for the Worcester market, or getting in the revenue at the intermediate stations, there will only be one winner..
Until the early 1990s, there were no other through trains to or from London the rest of the day, with a change at Oxford the rule. From 1993 until 2004, almost all of the London service was Turbo dmus - things have change a bit since then.
The single-line sections at either end of the Cotswold Line are still there, and the need to be able to timetable trains through those without services getting in each others' way is vital to ensure the timetable is robust and reliable. Lack of reliability was a reason why lots of people from the Worcester area kept claiming it made sense to drive all the way to Warwick Parkway to reach London, despite there being no journey time or cost advantage. Recent improvements in Cotswold Line reliability seem to have resulted in a rise in traffic from Worcester, so no one is going to risk that by implementing a timetable that does not work, however exciting the end-to-end journey times look on paper.
There is nothing much to be gained by raising speed limits, due in large part to the close spacing of the intermediate stations, except perhaps the 75mph between Evesham and Moreton-in-Marsh. But even here the 60mph restriction at Aston Magna curve will not change, so any journey time gain would be minimal.
The majority of services in both directions are in the range of 2hrs to 2hrs 15mins, with a couple of sub-2hr journeys as well.
The late afternoon trains out of Worcester are most important for people heading home to the Vale of Evesham and they (and the 05.50 from Paddington in the morning) all cross a series of very busy trains heading in the other direction, which it would not be a good idea to delay, hence the more generous timings and extended dwells in places. The number of people making end to end journeys on those trains is limited. The same goes for the 22.00 from Hereford to London, which has an extended schedule to allow for late-night engineering work on the GWML between Didcot and London.