I don’t understand why they didn’t order fewer sets but made them 8-car. Surely the fact there are 33% fewer sets to staff, lease and maintain would be considerable.
DfT have decided what fleet should be, double units are painful to staff, but in theory can double up 5car to 10car for busy trains. The length being governed by relatively short (for mainline) St Pancras platforms.
But you are basically correct, the assumption that about half the trains only need half the capacity is bit blunt, GWR have struggled with it too, and too slow to couple and uncouple on route if only half the journey is busy.
There are trains where loads ideally need 7 or 8 cars which don't really fit this model. Stuck with 5 or 10 so overcrowded or wasting money, and staffing (ticket checking and catering) a second unit that only really needs to be extra 2-3 carriages is inefficient