It does seem that most people prefer the first game that they played, doesn't it? It's that way with me for a lot of games in other series. I think it's just a thing where the one that got you into it all is just automatically the best because it was the first that gave the impression of being awesome.
For me, Ocarina of Time is my favorite game and was also my first. That doesn't mean that I don't have anything to back it up, though. I don't like to jump on the bandwagon, but it seems like I am because people claim that the reasons for me liking it so much aren't that good, but it's just that kind of game. I suppose there's some factor with the first game you have ever played contributing greatly to the amount that you favor the game over others.