Dorking to Horsham has to be a very strong contender. That line is always extremely quiet. I have had at least ten occasions now over the years where i have had the entire train to myself for the whole journey. There is rarely more than ten people onboard. You might get twenty maximum on some rare occassions. Even during peak times i have had extremely quiet trains with just a couple others onboard. So even the peak trains are quiet. I use this line regularly and i think it has to be a contender for the least used line in the country. It has always baffled me why it gets such poor usage considering it gets an hourly service and links two major towns.
I use the line nearly daily, at varying times. It seems much busier than it was, say, winter 2023/24. Passenger numbers seem to be shifting more to the peaks from my own observations. If you have ever got the last train(s) on a Friday (23:25) or Saturday (17:41), you will also find this to be very well used (standing to Epsom, full to Dorking), albeit usually only 4 coaches. This makes me think a proper Saturday evening service would attract a lot more London day-trippers from the likes of Holmwood (Beare Green) and Ockley (Capel). I would certainly make good use of it.
Journey times from Horsham–Victoria (via Gatwick), and Dorking/Epsom–Waterloo are much better than via Sutton, which probably doesn't help with loadings, even on the stretch where there is a 7-day service.
A Sunday service would also be very helpful, for obvious reasons. There are something like 4-5 buses per day on Sunday. Not everyone in Surrey drives, despite what the council would have you think! As for the peaks, you wouldn't want the train to fill up between Horsham and Dorking, otherwise the good people of Sutton–Mitcham wouldn't have a chance!
Waffle over: I agree, it would probably contend for least used, particularly if taken in the context of its location and service level (hourly 5.5 days per week is something).