Perhaps on the first playthrough when the puzzles aren't as familiar. I guess three dungeons with minor nuisances can stack up to one really frustrating temple.
On the other hand, there are some parts later in the game that can be pretty hectic on the first time round: Ironknuckles, the...