On my first play-through, Phantom Ganon was definitely the boss I had the hardest time with. Even after I figured out you were supposed to shoot his horse with arrows as he flew by, I had trouble actually hitting him enough times before I ran out of arrows (sadly). And then, when I finally got that part down, I had a really tough time getting the timing right for hitting his energy-balls back at him. Of course, it didn't help that my TV had glares from the windows so bad that I could hardly see anything in that room. Overall, I must have died upward of 10 times. Probably a decent bit more than that.
I had a pretty easy time with all of the Young Link bosses, and then, after Phantom Ganon, I had the biggoron sword, and the rest of the Adult Link bosses were pretty much cake. On my second play-through, however, when I did the three-heart challenge and didn't get the biggoron sword, the most difficult one was Morpha. I'd never realized how much less damage the master sword does as compared to the biggoron sword until then.