I always figured that the temples in some cases USED to be places of worship (forrest temple) or are currently places of worship, but you never see anyone there due to the enemies that have taken it over (spirit temple for Gerudo or Fire Temple for Gorons). I think in some cases there is even...