I would have to say that the game is, indeed, quite challenging. I started with Ocarina of Time, so that could easily explain why. I started with Ocarina of Time, then Wind Waker, then Phantom Hourglass, then Legend of Zelda. It was with that I grew up with the 3D titles (well, I wouldn't say grow up, since I started playing Zelda when I was 13) that I wasn't adapted to the 3D titles. A Link to the Past seemed a bit easier for me since I could go diagonal to go evasive, but I suppose the thing that made Legend of Zelda difficult was the lack of evasiveness. Even Adventure of Link offered more evasiveness with the ability to jump and move to dodge projectiles, but Legend of Zelda didn't have that option and limited you to only four directions with medium speed.
Plus, some of the enemies were cruel. Sure, the bosses were easy, but it was having a room with 10 Blue Wizrobes that does not allow you to live very long. Plus, there were quite a few Like-Likes out there that would eat your hard-earned shield. Earning 180 rupees in that game was not an easy feat, for rupees only came in 1s and 5s, with the exception of gambling, which could make you lose it all.