Have you forgotten, the inability to control levels of immigration from lots of European countries, and all the impacts that had on lowering wages, putting pressure on housing and infrastructure, making it harder for people to get somewhere to live, etc.?
Blame the good old capitalist system primarily, not immigration.
I'm
not saying we should abandon free-market capitalism by the way. Just pointing out, in a dispassionate way, that the profit motive and the free market is the ultimate cause of all of this. There are other, admittedly more left-wing, ways of attempting to tackle these problems. For example, rather than prohibiting immigration, what about prohibiting lowering wages or prohibiting increasing rents way above inflation? Radical perhaps. But really more radical than taking away 50-year-old rights and making the country the "odd one out" in Western and Central Europe?
Plus, other European countries have significant anti-immigrationist movements (Le Pen, Meloni, Wilders, AfD, etc). Why, then, are said countries not leaving Schengen and the EU and closing their borders, in order to pacify the anti-immigrationists?
Perhaps because, despite there being a good number of people who dislike immigration, the governments believe that open borders produce more good than harm? Even Le Pen and co do not seem to want to leave the EU.
So, what's different about us in this respect? (And no, it's not that we're an island - see Ireland, Iceland, Malta, Cyprus for example).
Rhetorical question of course: Brexit was a Tory vanity project, end of, designed to win elections by forming an alliance of social and economic conservatives that was enough to get them a majority. And it worked. Would we have still had a Tory majority in the early 2020s without Brexit? Perhaps not. Would we ever have had Brexit if the ERG and Farage hadn't been constantly agitating and backing Cameron into a corner? Probably not.
At the end of the day, I think most people in politics (including many Tories) realise Brexit was a silly idea but don't really know the way out of it - but will not admit that publicly. Instead they big it up because they know that many people like confident politicians, not indecisive ones.