Well, Volvagia can be difficult or really easy. Easy if you shoot him in the face with arrows to prevent him from attacking and climbing onto the side when rocks start to fall. Without backup Health (Potions or Fairies) and these two methods, he can be difficult, as he does quite a bit of damage.
The second half of Phantom Ganon's fight is tough because sometimes the edges of the arena (the ropes) will get in the way out your view and unlike the fight with Ganondorf, sometimes when you hit a magic ball it will just spazz out and go in some random direction.
Ganondorf can be a little difficult too, mostly because of Magic Ball-Sword flukes and shooting him with a Light Arrow fast enough. Sometimes in my haste I will miss him by a thread and he'll be able to get back up.
Queen Gohma, uber easy, King Dodongo, uber easy, Barinade, pretty easy, Phantom Ganon, a bit difficult, Volvagia, difficult without tricks and/or backup health, Morpha, easy, Bongo Bong, easy, Twinrova, easy, Ganondorf, difficult, Ganon, easy.
It's conditional in my opinion. If you don't have the Defense Upgrade from the Great Fairy, or just suck at Dead Man's Volley, Ganondorf. If you do not have backup health, Volvagia, because his attacks can be hard to avoid and he does quite a bit of damage. Sure, you can just shoot him in the face with an arrow to stop his flying around and fire-breathing, and climb onto the sides of the arena to avoid his rock-dropping, but what's the fun in that?
So:
1. Ganondorf
1. Volvagia (It's a tie)
2. Phantom Ganon
I think those three are the only difficult bosses of Ocarina of Time. Everything else is really, really easy in my opinion.
