That is basically the same as the oft-quoted "You're a Muslim, you do something about the terrorists", and it's just as ridiculous as that is. It's not the fault of right-thinking men either. We do not assume some kind of collective guilt simply from being male, and I find it quite offensive when feminists imply that we do or should.
Sexual assault and the likes do need taking very seriously indeed, probably much more seriously than they are now, but I don't see how I as a random bloke who is not in support of other men acting like that is going to be able to do something about it beyond stepping in if I actually see it taking place. There isn't some random Facebook group with all men on it where I can just post something asking them to stop it.
And to add a bit of a note, a bloke is more of a physical threat to a woman partly because blokes are generally stronger (and being 6' 4" and built like a rugby player I can see why people might see me as a physical threat, though in actual fact I'm about the least violent person in the world and would only be a physical threat to someone who attempted to attack me, and even then I'd run away if it was an option) and because when the male and female organs were given out we were given the one that could be used to cause damage without consent, whereas women weren't. But equally, women do sometimes sexually harass men, sometimes even in a physical way e.g. backside-pinching, and that is not OK either (and I was actually a victim of minor sexual-harrassment-type bullying at school). Would you agree that all women should be held responsible for the small number of them who do that? I don't think they should, even though it does need to stop[1], but that goes both ways.
[1] In essence, physically touching or harrassing someone else deliberately without consent isn't OK -whether they are male or female, straight or gay, adult or child or whatever.