People who harass the gender-odd often claim the privilege to do so because trans folks *remind* them of a threat. But not only are the trans folks not a threat, the harassor's behaviour doesn't match their claim of feeling threatened.
Perhaps I am unusual in this, but if I perceive someone as a threat, I do not immediately harass, block, stinkeye, taunt, poke or grope them. Yet this happens to trans folks all the time. And yet the exclusion of trans folks is often justified because we (be we gender variant, MtF, FtM, intersexed, crossdressed, genderqueer, etc...) "remind" cis people of threatening males. But, in my experience, most (sensible) people do not respond to genuinely dangerous men by verbally taunting them. Given this, their reaction to trans folks looks more like simple bullying, thinly justified.
Think of all the other times people rationalize injustice by claiming entitlement to a clearly irrational feeling of safety: people who won't fly with female pilots or drive with female bus drivers; calling the cops on the homeless in upscale neighbourhoods; drug laws; racial profiling at airports; veteran draftees being spat on; sex-worker evictions; refusing out queers work as priests, youth workers and teachers; numerous historical examples involving the words "protecting our women from...;" don't ask don't tell (you're in a *war*, and you're afraid of *gays*?); speaking of which, how many wars have started because one side wanted to protect itself from a fabricated threat?
The claim that transphobic gender policing and systems that support it are justified because cis people mistake us for threats is just one more example of socially-sanctioned bullying.