I've been suspicious about the Yiga for a while, and more so since I encountered Kohga.
Unlike the other members of the Yiga clan, he's flamboyant and clumsy... which doesn't really make sense if he is a Sheikah master, as they specialise in stealth.
Plus, his boulder-based attacks appear to be more akin to magic, rather than the science/illusions associated with the Sheikah in BotW.
If you look closely, it also appears that unlike the other members of the Yiga clan (who keep covered up while not in disguise), his skin is actually visible, and is quite dark, unlike the typically light skin of the Sheikah.
It's also implied that although formed by Sheikah defectors, not all of them are Sheikah, as they seem interested in recruiting anyone who will join... so what if Kohga himself isn't actually Sheikah? Do they actually say he's Sheikah, or is that just assumed?
Dark skin, magic user, flamboyant, lives in the desert, is after a Gerudo heirloom (which would otherwise seem of little use to the Yiga), and has sworn allegiance to Ganon... it just screams male Gerudo to me.
They're supposed to be born once every 100 years, so it seems strange than Ganondorf is the only once we've ever seen... especially since BotW gives us a 100 year time span since the calamity.
I've also noticed before now how the colours worn by the Yiga are actually fairly similar to those worn by Ganondorf in past games, which could also be a subtle hint.