This particular theory has a long history. It used to make more sense between ALttP/OoT until TP came along and seemingly messed with the geography and context.
Some people think the ruins around the ToT in TP is that of Castle Town from OoT. It's a safe assumption but I don't see any particular evidence of that. TP clearly attempts to make a deliberate connection to ALttP, but the interior of the ToT in TP and the surrounding grounds, as well as the major discrepancies between TP and OoT's geography differ from that of OoT's Temple.
One theory is that TP's is a different, newer Temple built in the forest. One theory I have is that the Temple of Time is able to move through time and space, due to its connection to the Temple of Light in the Sacred Realm.