I think the Imprisoned is definitely the most annoying boss with having to refight him multiple times with minor changes, but I feel the worst is either Aquamentus from the original TLoZ or Blaaz from Phantom Hourglass. First bosses are supposed to be easy, but they are too simple.