I'm going to introduce a new element to this, THE LOST WOODS, just by the OoT map you can say they make no sense at all, look at this:

Notice the entry point (the bridge between the field and the woods) appears two times in two different locations, it makes no sense! Now note something funny about the forest meadow in this image

YAY OVERLAPING
This thing is funny, I say that nothing related to the lost woods can be taken as the maps say, the lost woods aren't a normal wood, it's a dimension shifting wood, with warping capabilities, think this, the woods can take you from zora's domain to goron city, any path may take you to kokiri forest, it can take you to effin' Termina, the thing is these woods are just downright imposible to cartograph, notice that the in game OoT map just says "lost woods" and puts some trees and random holes in it, that's the best you can do with it, if you are right next to goron city and walk a few steps you sudenly are next to zora domain, and still you are a few steps away from kokiri forest, even the rose of winds is screwed in here, how can you tell if the meadow starts northwest and saria is southeast? How can you know if cardinal points even
apply here? theres not even a map of it (that is, the kind you normally see at the bottom-left corner of the screen).
You may be asking where am I trying to get, let's look at TP, the lost woods are present here, but where? where is this misterious wood? where is it that skull kids live? where there isn't a map? and gates are pitch black until you enter them?
The lost woods are the sacred groove now, and I think it's safe to assume it also has the same warping capacities, it is posible that the woods overrun the ToT and now it is dimension shifted (so it is imposible to know it's true location), that would explain why it is seemingly in the middle of nowhere in the map, this explanation also holds likely from a nintendo standpoint, even if locations tent to shift, Nintendo tends to keep the location of a few places similar (such as death mountain), so it could be safe to asume the designers said: "Hey, we have to put the ToT next to the citadel" "But that would make the conection too forced" "there has to be way around that" "That temple there doesn't fit with this plotline, it has to be a late dungeon, and the MS has to be received way before" "just put it in the lost woods, the woods overrun the temple or something" "good idea"
Also I want to note I believe the Hyrule Castle was rebuilt, there are ruins in the outside of the ToT in TP that resemble the castle town of OoT, and there is other evidence too
As a last note, it is not only plausible that the lost woods overrun the ToT, but it is also implied they are able to do so in another game,
FSA (scroll down to "Four swords adventures")