do they sell tickets for specific trains though? If not, how does this work with flexible fares?
Also were there actually more people than the train could hold? It isn't mentioned that anyone was unable to board.
I'd imagine a refund would be available if anyone was unable to board or chose not to stand.
On most heritage railways, particularly when running a very limited service, it is a fairly safe bet that anyone turning up at the booking office wants to travel on the next train. The railway should stop selling tickets as soon as they have sold as many as the train can hold, or the train physically becomes full. In this case, they knew that one coach was reserved, they should be able to estimate from experience how many places are likely to be taken by those with flexible tickets, and making an allowance for the sprawl factor, gives the number of tickets that you should sell. It is not exactly rocket science.
As the OP pointed out, charging someone standing in a brake van the same fare as someone with a comfortable seat and a good view is not good, and it is likely to get the railway poor reviews on TripAdvisor and social media. Yes, passengers should be able to get a refund if they don't want to stand, but those passengers have already queued to buy a ticket and may have waited awhile for the train, and now have to waste further time to get a refund, so are not going to be happy.
Yes, the SR may have under-estimated demand. It can be very difficult to predict demand in advance. And even if you do predict the demand, it is not always practical or economic to match supply to demand. However, the critical thing is how you manage it. To get to the situation where people are having to stand in the guards van indicates that they lost control.