If, in reference to what I said, I didn't say before they created Termina, after. Or, as put by Zelda Wiki:
In the past, the Skull Kid was befriended by the Four Giants. After a time, however, the Giants left Termina and the Skull Kid was alone, motivating him to seek attention by causing...