Hyrule Historia page 117: The Temple of Time, home to the Master Sword's pedestal, has rotted away and lies in ruins within the forest.
And to be honest, it seems that the ruins you see are actually the old Hyrule Castle town's remains.
If you look at the map (I had to mirror it because it was the Wii version) and in-game you'd see what seems the remains of Hyrule Castle Town and the route that went to the front door of the Temple of Time. The ( ) bit seems to be what's left of the fountain that stood in the middle of it.
At least, it seems that this is what Nintendo intended. There's also castle walls to be seen. So with this information, it seems that a battle took place here and after it was destroyed and deserted, nature took it back and transformed it into a forest. Assumably, this happening is related to Ganondorf's capture. Afterwards, the Castle and Market were relocated.
But yeah so far none of the geography changes from OOT to TP make any sense. It's just that the Hero's Shade doesn't look like he was buried because he's covered in vines, but if he died in the forest he'd become a Stalfos. And since he's not a Stalfos, my bet is that he died right there on the battlefield and the forest took over afterwards. But of course, that is all just speculation.