Most annoying is probably the sailing in Phantom Hourglass. I rather enjoyed sailing in the Wind Waker, if felt free and open - even if it could be boring at times.
However, sailing in Phantom Hourglass felt like the antithesis of the Wind Waker, it was constricted, limited, the process itself was irritating (I detested having to open the sea chart, draw a line, then sail there. Why couldn't I just.. sail there? It felt like such a hassle.)
Perhaps almost as annoying, was in Twilight Princess. Where every damn time you get a blue or higher rupee, Link has to present it to you like it's something he's never seen before, and act surprised every time - even if you've found that denomination a hundred times already. This wasted so much time, and just overall annoyed me.
The low-hearts indicator didn't bother me too much.. Except for a few times when I got down to one or two hearts, and had to run across the entire field or something like that just to get it to stop.