That's the maximum penalty that could lawfully be applied - and even then, it would strictly speaking depend on whether you had been given adequate notice of the restriction on breaking your journey when buying your ticket for that to become an incorporated term.
Some TOCs, particularly Northern, seem to be in the habit of trying to issue Penalty Fares for starting late or finishing short but this is not legally permitted and any such Penalty Fare should be appealed (we can no doubt assist with this).