Definitley catching the rabbits in ST. Some are easy but the others are really hard. The part that makes them frustrating is that the ones you don't catch, they're really far away from a station or an entrance into another realm. So if you miss them, it takes a while to try again.
Another frustrating part was the Temple of the Ocean King in PH. While I think that now it's really fun, at times, I'm really annoyed about visiting solved places over and over again. Some parts were fun I guess which made them OK. Others were a lot harder and you had to be really careful to be able to make it.
I can't really think of anymore annoying parts right now, but I might type it up when it comes to me.