The Fire Temple my first time around was the hardest for me. It was so big and I couldn't figure out where to go and getting lost and all the other things you go through when playing a temple for the first time. Water Temple was tough but I originally struggled more with the Fire Temple
I voted for the Forest temple because I'm sick of complaining about the Water Temple and its infamous key, which is really the only reason it was ever difficult in the first place. Otherwise it's just sort of tedious raising and lowering water levels and switching boots constantly.
The Forest temple was hard for me because it was a little like transitioning from elementary school (The first three temples) to middle school (the last 6). The first three temples are more linear and you have a lot more guidance, in addition to just being easier, as they should be. You don't even have to get boss keys.
Then you go to the Forest temple, your first day of middle school, you get lost in the hallways, trying to figure out which hallway you have to go down to get a key. You have to get a boss key for the first time, you have your first timed block puzzle, the layout isn't linear, the puzzles in general are harder, and the boss is harder than anything you faced as young Link.
The Forest temple was hard simply because it steps up the difficulty from what you experienced as a kid. Please excuse my cheesy grade school metaphor.
For the first time, the water temple was the most difficult. I still have to back track to find a key that I missed and the square room with a door in each direction makes it confusing. The forest temple can also be very confusing. In OoT3D this temple gave me the most troubles. I still vote for the water temple though, since it manages to fool me every time.
The water temple gets my vote more for annoyance than difficulty. I didn't find any one temple noticeably harder than the rest, but the constant changing of the water levels made me quite annoyed the first time around.
the water temple fo sho. it isnt hard but very annoying, constantly changing the water levels, ughhh!!!!! plus when u fight urself sumtimes i can kill my bad self without ever being hit, and other times i die 4-5 times... boss is easy tho
I find the shadow temple extremely difficult. First of all, I went in there with no magic left so the lens of truth was useless at first. Also, I just found the settup of the whole dungeon rather confusing. The main reason I don't like it is because it took at least 5x longer for me to complete that dungeon than any of the others. The boss was easy enough though.
I have to say that the hardest temple and this is only for one reason but the hardest temple for me was the forest temple. I just was hard with all the things you had to do perfect. If I had to say another hard temple it would be the water temple for its puzzles.
Shadow. The Water Temple can be frustrating as in not knowing where to go, but at I HATE DED HANDS WITH A PASSION, as well as REDEADS. So the Shadow Temple is torture for me. I also hate the part with the invisible Floor Masters and the guillotines and the Hover Boots. I hate feeling out of control and the Hover Boots are torture for me.