I wouldn't, I'd tell (not ask) them to shift. Mainly because now VTWC has a seat selector, I normally reserve a window seat in a priority row to ensure legroom.
The problem is, you are on a loser, even if technically you are in the right. What do you do if the person stubbornly refuses to move? You have no power over them, unless you want to risk an assault charge (or a beating) by physically moving them against their will. This is one reason why people behave like bloody minded tossers, because in some situations they gain from it, get away with it, and no-one is or can do anything about it.