I don't actually find the flower lady annoying, probably because it's easy to walk to the shrine without stepping on the flowers. The one I hate is some guy who scams Link at one of the stables; he wants you to buy bread or something? Some ugly young guy that is a shame you can't cut down...