Zemen125 is right - it makes no sense to kill all males with the exception of one. If they hated men that much, they would kill every male born into their tribe. Also, they wouldn't let a man rule over them, either. Judging by that, maybe they hate all men with the exception of their own (and Link, of course, since he earned their respect).
Your reasoning behind this is interesting, and there are a few facts to back it up. Majora's Mask could well have originated in Hyrule, since the Happy Mask Salesman has it, and the Gerudos could have come from Termina through the hidden passage to Hyrule and then sealed it away (if they created the mask in Termina).
But here's what busts this theory - at this point in the timeline, the Gerudos were still around, and their presence was known in Hyrule. They had not disappeared yet. The HMS said in MM that the ancient tribe had vanished, but he would have known of the Gerudos, since he comes from Hyrule.