The Temple of Time in SS is indeed not the one on the other games. You see, Rauru built a new Temple of Time after the Sealed Temple crumbled and died, there he made the new Temple of Time. IN Ocarina of Time the Temple is in Castle Town, Yes, but perhaps during the long years after Link's departure back to his childhood, the Master sword is still in the temple of time. castle Town could've been rebuilt somewhere else while the Sacred grove stayed there and rotted, the sacred grove in TP used to be the Temple of time too. the Lost Woods could've been moved too, maybe the old lost woods were torn down and a new lost woods found it's reputation someplace else. In TWW the Master Sword ended up sealed in Ganon's head forever, stating that the Master Sword may never again be seen again in that timeline.
The simplest answer is that there were unnamed heroes too, they could've used the Master Sword as well and found it a resting place somewhere else, that's why the Master Sword was in Hyrule Castle in TWW. That last sentence is my simple answer to your question.