AoL is, indeed, one of my top 3 Zelda titles, but then maybe this is because for the longest time it was the only Zelda title I had to play back on the NES. (A friend owned LoZ, so to my parents, it didn't warrant being purchased as my friend and I were always hanging out.)
I think what amuses me about how people dislike the game so much is the game is exactly what people want out of a Zelda title these days. The main two points that people keep asking for in new Zelda titles are right there - It's loose in terms of your next goal, the difficulty is more than that of modern Zelda titles. Even lesser elements asked for by fans for upcoming titles are there in more RPG-like systems, more towns to visit, and the use of magic. So, it makes me shake my head when the same people asking for a combination of these elements in new titles dislike the game or don't even give it a try. I think they truly don't know what they want from their Zelda when I read these conflicting feelings.
It displeases me to see fans these days say it was always the black sheep. Back in my day (there, I said it, I'm old), AoL was the best of the NES. At the time of its release, no one cared it was different form the first game. It was well recieved, topped many Top 10 lists and was not considered bad by any means. The idea that it's the black sheep came with time, as the series changed direction back to the original formula. With each new release, it's pushed into it's own category, but when there were only two Zelda titles, even 3 and 4, it was still loved.
It also displeases me to see how many newer Zelda fans (and yes, I include "as of Ocarina of Time" in this category, since it was 12 years after the first, and therefore 12 years after I was a fan), won't even give the game a shot. It's a mere $5 on the Virtual Console these days. The only excuses not to try it are that you don't have a Wii, or $5. And I'm betting if you have the first, then the odds are you have $5.