But that's assuming that all those who voted to remain were happy with FoM. Some of them may not have been, but felt that it was worth enduring FoM to retain the rest of the package.
Anybody who voted to leave but expected FoM to be retained could not have quite gathered the mood in the room. Similarly those who believed "leaving" would see the retention of the UK's membership of the Single Market and/or the Customs Union were equally deluded (particularly with regard to the Single Market, which requires FoM). I don't believe there was any serious suggestion before the referendum that the UK would retain any of these features of EU membership. In fact, the government pamphlet published before the referendum makes it quite clear that the Single Market and FoM are inextricably linked and that leaving the EU would (at least) jeopardise membership of the Single Market and more probably see an end to it:
Remaining inside the EU guarantees our full access to its Single Market. By contrast, leaving creates uncertainty and risk.
No other country has managed to secure significant access to the Single Market, without having to:
• follow EU rules over which they have no real say
• pay into the EU
• accept EU citizens living and working in their country
Some argue that leaving the EU would give us more freedom to limit immigration. But in return for the economic benefits of access to the
EU’s Single Market, non-EU countries – such as Norway – have had to accept the right of all EU citizens to live and work in their country.
Quite honestly, if the UK had left the EU but retained membership of either the CU or (particularly) the SM all that would have happened is that our name would have been crossed off the list of members. It would have been "BrINO" (Brexit In Name Only). The second of those paragraphs above explains that perfectly. There were certainly suggestions after the referendum that this was a possible outcome (though quite a lot happened in those four chaotic years which were, shall we be kind and say, not particularly helpful). But nobody who voted to leave should have done so in the belief that we would retain FoM.