Nice try, but not really arguable. Such would give the word annoy a meaning broader than it can reasonably bear.There probably is an argument that the staff member's statement was accurate, the Railway Byelaws state:
Now the title including the word 'dangerous' perhaps has had some readers skip over this section, however:
So we can disregard the word 'dangerous'.
It is then not that far of a jump to say that an authorised person had the opinion that said item (the Virgin Trains coat) may annoy any person (by causing confusion), and that as they had asked the OP to removed it and should the OP have failed to do so immediately that it may be removed by or under the direction of an authorised person (although the fate of said item isn't mentioned, confiscation does not seem to be an unreasonable choice of words for these actions).
In the OP it isn't exactly clear what language the 'dispatcher' used however I think its reasonable to suggest their statement went up to and including 'we' (the railway) and that this terminology would include the BTP, who in any case would in all probability have been called upon should the OP not have followed the instruction to remove the coat.
Railway byelaws
www.gov.uk
The Byelaw is clearly aimed at, for example, ghetto-blasters, laser pens, open containers of bleach/paint and so on.
The ability to remove in para. 2 must also be read by reference to para. 1 which relates to an article being brought onto or remaining on the railway.
I see no authority for an article of clothing that is not in itself threatening or annoying to be forcibly removed from a person’s body, but not then removed from the railway. There is also no provision for outright confiscation.