Yes, the mansion feeling is something unique to the Forest Temple. It feels more like a normal building that really any other dungeon of the game, and it becomes more disturbing because of that. I would however argue that the Shadow Temple only managed to be roughly as scary as the Forest...