For those saying "Why isn't (x) here?", there
is an "Other" option
I vote for Other: Eagle's Tower from Link's Awakening (original). Main reason? The game-breaking glitch with the wrecking ball. The boss being able to throw you out is also obnoxious.
On a similar note, the second worst for me is the Wind Fish's Egg, because it turns out the library can glitch and not let you read the route.
For my money, it's hard to beat "OOPS YOUR ENTIRE PLAYTHROUGH JUST GOT INVALIDATED BY A GLITCH" for annoying.
Regarding the Water Temple... I never found that as bad as some people make it out to be. I never found the water level raising tedious, not did I find switching boots in the equipment menu tedious (and I actually
prefer that to making the boots a button item, so that change is one thing that puts me off wanting the remake). I also didn't have any trouble with the keys. The only spot in the Water Temple that ever really bugged me is the stupid freaking ramp up to the boss chamber.