This is my usual approach too. Sadly one time on a Voyager, I found quite an obnoxious lady sitting in my window table seat (which I had reserved so that I could plug my laptop in and work during the journey!).
The woman insisted I was mixing up the seat numbers of the window seat and the aisle seat.
Unfortunately when I persisted, the person sitting next to her offered (well, insisted) to move instead and went away, which didn't help at all because it just armed the lady in my actual seat with further ammunition of well just sit there, then!. And my insistence to have my actual seat got me some very strange looks from the people opposite who probably felt I'd done enough damage by causing somebody else to move (which I had no control over!)