Out of the Mini-Bosses, Dark Link naturally. A guy with very limited ways to hit him, lots of counter-attack moves, of course he's going to be a problem.
Out of regular bosses, Bongo Bongo. I think the last time I played through OoT, I got to the end of the Shadow Temple and died trying to beat him. I came back, and died again. I went out, got some fairies and potions, went back, and died again. And this process just repeated itself over and over and over, and no matter what I could not beat him, and I had no idea why. Eventually I finally beat him, but it still stands that he's one of the most annoying bosses I've ever faced.