In all honesty I’d be pretty pissed off, especially if I were a leave voter. However I agree that technically we elect the person not the party - although of course the party name does appear on the ballot paper.
"Crossing the floor" has been quite common in Welsh Assembly politics recently: Caroline Jones and Mandy Jones left UKIP to be Independents, Mark Reckless (yes, him) left the UKIP group to sit with the Conservatives, Dafydd Ellis Thomas left Plaid Cymru to be an Independent and join Labour in government. Mohamed Ashgar was first elected as a Plaid Cymru candidate, but then joined the Conservatives, and has stood (successfully) for them in subsequent elections. Some were elected "by name" in individual constituencies, but others are regional AMs elected from a party list. Even those from individual constituencies are considered to be representatives of a party, for the calculation of which under-represented parties get seats from the regional lists. But none of these have felt the need to stand down and trigger a by-election, as fas as I recall. (If a regional member stood down, they'd just be replaced with the next candidate of the list from the previous election.)
In the case of Mandy Jones, she got her place in 2017 after Nathan Gill stood down (a year after the election) because she was next on UKIP's regional list of candidates. But due to disagreements with the UKIP group in the Assembly, she effectively sits as an independent.
And returning to Westminster, Winston Churchill famously crossed the floor several times during his career, switching loyalty between the Conservative and Liberal parties (and being Independent when it suited).