Do non-work travellers not count, then?
If you're having to choose who to disrupt, you're going to prioritise those who could defer their travel. Those who have to get to work, or else have to waste annual leave for the day off, are going to be more important than someone making a trip to go shopping or go to the seaside.
If you are organising alternative travel arrangements that will add a long time to journey times, or will have limited capacity in comparison, why would you do this on a weekday compared to a weekend or bank holiday?
Obviously there are people who work weekends too, plus shift workers that may suffer longer journey times, but you are massively reducing the impact on the majority of people.
Remember, you have to do the work so suggesting you don't impact anyone isn't an option. You have to make a decision, and the right decision is to do work at weekends, Sunday mornings, bank holidays and Christmas.