I don't think it's fair to say gender identity is "a mental phenomenon rather than a physical one." At least not entirely. I do think that gender identity is largely socialized, but it also is often rooted in physical reality, and in some cases, that reality can be messy, and hard to fit in a clean dual-gender system, such as athletics. Consider the South African athlete Caster Semenya, for example. She is physically female, but has XY chromosomes. Her condition is one of many rare conditions that are sometimes called "intersex," because they don't fit neatly within a fully male or fully female definition. Because her body is female, she was raised that way, and always identified as such. But concerns have nevertheless been raised that her particular genetic makeup gives her some of the same advantages that an XY male would have competing against females.
Anyway, I'm not bringing all this up to advocate for a certain position on this issue. I'm simply reminding everyone that biology is not neat and tidy, and so this issue probably can't be waved away with a simple "Compete with what you are biologically."