Thought so. I personally believe they were (for the most part) palaces for the evil demons, monsters, etc. that lived there. Or to house the weapon that it contained. Possibly it was some relic? Like the Goron hammer in OoT. Or, as mentioned by others, it was to lock up the boss it contained.
In the first Zelda, it was called a labyrinth. Then in the second, it was palace. I'm not sure what it was in aLttP, but I think the calling it temple started in OoT.