However, I would argue that Bedford-Brighton is by far the most important TL service and should always be protected in service disruption, notably being the main service for Bedford, Luton and St Albans, the airport traffic to Luton Parkway/Gatwick and the lack of alternatives compared to the inner TL routes (St Albans-Sutton/Sevenoaks). Only anecdotal evidence and from what I have observed too, but it would seem that more people are using St Albans Abbey or going via Stanmore/Edgware, on the GN side people are using the Uno 644 too, desperate times...
I understand what you say and agree that it makes sense.
When I wrote "This route always seem to be a poorer relation compared to the long distance Bedford <--> central London route.", I was just lamenting the fact that St. Albans <--> central London seems to always take lower priority compared to Bedford <--> central London, not actually wanting this to be changed (except consider stopping some Bedford <--> central London trains between Radlett and Cricklewood if no St. Albans <--> central London trains for a while).
Last Thursday evening was another typical example.
The northbound Luton service arriving ELS at 23:49 was curtailed to start from West Hampstead Thameslink instead of through central London. For those who start their journeys from Central London, there was a Bedford service which was delayed by a few minutes, and would arrive at West Hampstead Thameslink at about the same time as the curtailed service was about to leave.
For a moment, it was touch and go whether the Bedford train would make the connection in time, but eventually it turned out that there would be a 1-2 minute time window. When the Bedford train arrived, I saw people running desperately up the stairs to the other platform to catch the curtailed service. Some made it, but I also saw some who did not (and had a nearly 40 minute wait for the next stopper service), and the door promptly closed 20 seconds before the scheduled departure time.
Why not hold the train for another minute or so, so that everyone makes the connection? There were loads of cancellations that night and there was nothing following the Luton service anyway.