Only as a gross generalisation. It's true for Average man vs Average woman, but there is plenty of overlap.
Yes, the average women versus the average man. I'm 5'8", and I accept that a 6'5" female basketballer is going to be more proficient at the sport than I am. I don't agree there is that significant an overlap between male and female sports. Give me a list of all the record holders by gender in the Olympics, in events both genders compete in, and we will see how many of them are women.
By the same token, it's not fair that tall men should play basketball against short men, etc. etc.
That's a false analogy, because people, like me, who are shorter will self-selectively not play basketball and there is not an equal rights movement to level the playing field so shortarses like me, cruel victims of our biology, can play.
There are plenty of people born male who are below average-sized for women. There are also plenty who had hormone treatment early whose bodies never developed like a male body.
Why should they not be allowed to play?
Men who are small should not play in female sports teams. They should play male sports. Sports are there to actually cherish and exemplify the differences between people, not to equalise everything. As an example, Kenyans and East Africans dominate distance running because of their superior biology in that given discipline which gives them a significant advantage. This is a difference to be wondered at and celebrated. There are an enormous number of different demographic factors you could divide people up by in order to make things "fair", but that is not the point of sport. The equality of opportunity should be there - you can enter the race if you wish - but if you are just born short and dumpy don't expect to win the 100m, ever. Equality of outcome is always wrong.
I don't agree that people who have hormone treatment can play female sports, because I don't agree they are female in the same way. It is also very one-sided; women who underwent hormone treatment to become men would make far less of an impact than men who became women would on the women's games. I would feel much more supportive of transgender sports events, which is something the mainstream is totally missing. The Paralympics have managed to (as effectively as they can) empower disabled people and dispel myths, and Trans/Intersex games would have a similar impact, I feel.
Ultimately, I support the empowerment of individuals, but I don't accept that should be done by telling fibs to ourselves that men who become women are the same as women who were born women.