I believe that Majora's Mask was of average difficulty. If you're good with puzzles and enjoy them as much as I do, then the game could have seemed pretty easy in some parts. There were quite a few puzzles in Majora's Mask when I think about it. Of course that's nothing compared to the amount of puzzles in Oracle of Ages, but that's for another thread.
Majora's Mask gained most of its difficulty from the time limit. The time limit made players feel a little more pressured when they were doing something important, and sometimes you had to be a bit more careful. You couldn't do and do a sidequest then with one day left decide you're going to go do this or that when it takes too much time. The fact that some items disappeared when you went back also added a bit of difficulty to the game, seeing as how you may need those items and have to go out and recollect them, which takes a little bit of time. Players had to think a little bit more throughout the entire thing so as to not let the time limit slip their minds. This may be what some players talk about when they refer to Majora's Mask as hard.
As far as the other aspects of the game go, I don't see too much of it as hard. The only thing that I remember being any bit of difficult was the Stone Tower Temple. It just seemed a little more challenging with its puzzles than some of the other dungeons in the game did. The bosses were of average difficulty, so I didn't find too much trouble with that. If you're really good at games or you seem to get the hang of Majora's Mask a bit easier than some games, then you'll have no trouble with it. Majora's Mask was by no means a hard game, but it wasn't a disappointingly easy game as we've been seeing recently.