Wow some good points, you're missing one valid point Mases, FSA is a SPINOFF. Even if we were to count it as part of the timeline and if your theory is correct, then ALttP must have happened before Twilight Princess as well (along with FS).
Plus..the fact that Ganondorf himself is using the power of the Twili...and the fact he is the first to be banished to the Twilight Realm.
Here's a most obvious reason why the Gerudo were banished...Ganondorf tried to take over Hyrule! By now you should know he is in fact a Gerudo...Now how can the people of Hyrule prevent another sick magic wielding gerudo from taking over Hyrule?...hmm banishment? I think so. Yes, the Gerudo race are in fact wielders of the 'black magic'. They have always wielded this magic...It even says in ALttP that Magdrag Ganon (Ganondorf Dragmire) is exceptional at the black arts...
Ok, now we're saying the dark Mirror is the same as the Mirror of Twilight? Ok, we might as well say the Magic Mirror from ALttP is as well..since it too gives Link access to a 'darkworld' oh wait, its just the Golden Land. Hell, we might as well say the Magic Book from LoZ is the Book of Mudora...and the Triforce of Courage..is also the Triforce of Power!!! LOL anyways...We wont truly know until TP's sequal releases...oops, I've said too much...
Also, one more thing..the Gerudo being banished would make total sense (kinda like a loophole for Nintendo) since well...Gerudo has not appeared in the games following TP in the timeline, that be including...ALttP, LA, LoZ, AoL, Oracles...