Play Twilight Princess, you'll find plenty of evidence.
- Zant marks the Gerudo symbol.
- Once you enter the Gerudo Desert for the first time, Midna gives you a major hint on what happened to the Gerudo.
- Ganondorf (whom is Gerudo) possesses the power of the Twili, plus it is said he is the true King of the Twilight.
- Ganondorf was also banished to the Twilight Realm.
- The Gerudo is the only missing race in TP. (The Sheikahs still exist, go to Hidden Village)
- Ganondorf being a Gerudo tried to conquer Hyrule with his followers, he was later banished (most likely during Young Links timeline)
- The Twili possess skills in the black arts, as do the Gerudo.
- The fact the Mirror of Twilight is in the Gerudo Desert.
- The fact that the 'prison' (when Ganondorf was sentenced to death) is now in the Gerudo Desert.
Plenty more where that came from