You can to some extent manage this by diagramming - if you have fixed formation 8s and 12s like Thameslink, say, you can diagram things so the 8s are all already on their way out of London and the 12s on their way in at the high peak. VTWC are very good at diagramming the 9 and 11-car Pendolinos to meet demand.
There is also the point (as you say) that there is something extremely pleasurable about travelling on an unnecessary long train off-peak with a bay each. That said, it's probably not great for the environment.
But back to 442s, aren't they required to run in pairs for reliability reasons?
I've noticed that the fashion for running long or short trains off-peak has varied a fair bit through the years.
In the early-mid 80s when I first travelled the trains the hourly Portsmouith fasts (81) were all 12CIG-BEP-CIG (or CIG-BIG-CIG initially) with only one train in the main daytime period, the 0748 out of Waterloo, being eight cars. Likewise, the majority of the semi-fasts (82), with only perhaps two exceptions each way, were eight car.
Off-peak length reduction seemed to come in around 1986 or so; at this time, more services split off peak so you had more eight-cars on the fasts during the middle of the day and more four-cars on the semi-fast. An efficient way of doing this was often to split a 12-car fast at either Waterloo or Portsmouth, with the front 8 forming the next fast back and the rear 4 the next semi-fast. This meant that only one shunt to the sidings was necessary, the incoming (usually 8 car) semi-fast that would be replaced by the rear 4 from the fast.
This pattern continued through the NSE years into the early SWT years and was still the tendency as late as 1997 when most off-peaks out of Waterloo were either 4CIG/VEP or 5WES. The sole exception IIRC were the Portsmouth semi-fasts which ran as 4CIG/4BEP formations at this time. I also remember that the Direct had more 5WES off peak than peak - these would be split from and combined with other WES units on the Weymouth line in the peaks.
Another interesting feature of this time, notably the 1997 SWT timetable, was the tendency to put a lot of CIGs on such lines as Alton and Basingstoke slow off-peak, using units needed for fasts in the peaks. In the peak, these lines reverted to VEPs (typically 8VEP) for capacity reasons - but perhaps this off-peak usage of the higher-comfort CIGs also acted as a way to encourage off-peak travel on the Alton and Basingstoke lines?
Back to the long vs short thing, long off peaks once again started becoming the fashion perhaps around the year 2000, though there were still a good number of short ones. Late 2004 produced a very interesting and unpredictable mix on the direct, with 444s and 450s being introduced but some CIGs and VEPs still around - with all types having very varying lengths - and even things like VEPs on fasts. So you never knew just what you would get!