I mean, they'd still have had to cross the river at some point.
Not necessarily. The river loops. If the line had gone west of the loop it would never have had to cross the Thames near Abingdon at all (although it would have had to cross the Ock, though that's usually tiny). As it is it crosses the Thames twice.
If you look at old maps it's pretty clear Abingdon was built up hard against the river and even before the railway was a sizeable town for the time, so there was no obvious place to put a station that would be easy enough to build without prejudicing the mainline traffic between Birmingham and the GWR.
The relationship between town and river has been complex since before the Norman conquest. The town had a very large Saxon abbey whose monks arranged for the river's course to be altered more than once, both to improve navigation and to power their mill.
On a point of detail, Abingdon did have a railway station between 1856 and 1963 (for passengers) and 1984 (for goods). It was a few hundred metres from the river, and close to the town centre. It was the terminus of a branch line though.
I confess I've joined the forum largely to keep in touch with this very informative thread. I hope that's OK. I try to get out for a 20 mile walk once a week and one of my routes follows a footpath which runs alongside the railway south of the Nuneham viaduct and then, at the river, turns west and heads to Abingdon. By way of thanks for your useful info I will see, if the weather's kind and the path's still open, if I can get there towards the end of this week and post a few pictures of the works as seen from their side of the river. I should say though that the ground immediately south of the river and west of the railway embankment is most of the time completely waterlogged (up to a foot of standing surface water in places). I can negotiate that, but if it turns out to be any deeper now then my access will be pretty limited.