I understand where you're coming from. But just because Link says "let's go right away", doesn't mean, in the game, you HAVE to go right away, now does it?
No, but it means that he has expressed an opinion with which I disagree, destroying the "Link" between me and the game and turning it into me controlling a character.
Example: I can do whatever I want in Legend of Zelda, and it's what Link does. If I am serious and go straight to my goal, so is and so does Link. If I like to play around and go on a lot of side quests, so does Link.
In, say, a Star Wars game where I am Anakin Skywalker, no matter how I feel, he's turning to the darkside (according to autocorrect, carmaker), therefore, he is the character and I am not, which takes away what makes Zelda great, the fact that I can explore and do what I want, because I am Link.