I remember my first time playing OoT and he was a real tough boss in my opinion. I had just like the same, i needed a real long time to beat him into the grounds. I remember i had 2 or 3 bottles with fairies and still lost to him. I have to say the first half with the Arrows wasn't that difficult. It was really annoying seeing Phantom Ganon in 2 or 3 ways. If you looked at the fake one you must hurry for his lightning strike. Be hit by this attack is a lose of 3 hearths i thought. I didn't have many hearths in that time so 3 attacks were fatal for Link or just me.
The second half was more annoying. I had problems with returning the Licht Balls, if it are lightballs, to Phantom Ganon. The problems are starting if he flies a little bit above the ground, the Licht Balls aren't returning well and goes to a real wrong way instead of hitting him. Well sometimes if you are lucky you can hit him but it's better when he flies higher, If he is in the higher level the Balls are returning very well and nothing goed wrong. So since i figure that out i waited till he was flying higher.
I think i have challenged him also like 20 times or less or more, don't remember it. I only remember i couldn't beat him even with fairies. But all the bosses from OoT were difficult and rough in the begining.
It's not you El Bagu, you aren't the loser, if it was your first time it's very normal to take 20 times beating it. If you didn't use fairier you are even better than me, i lost with them too.
Anyway, if i gonna play OoT ever again i could defeat him without losing a life, you have to pick up each boss. And the more you play it the easier the bosses will be. I still love the boss Phantom Ganon, and it's nice to remember a hard time beating him.