To add to things:
Stone Tower was built by the people or Ikana/Termina/etc with the intent to reach the heavens, which is what the tower of Babel was for.
However, after shooting the gem with light arrows, the tower is flipped, and in turn it leads downwards, perhaps to the underworld as opposed to the heavens. A fitting punishment.
There are statues in the game that show the triforce, and the statues show some kind of rude/blasphemous imagery where the triforce is being licked or something like that, which is a sort of mockery of the golden gods or some such.
The tower's use and history is not 100% clear, though.
The land of Ikana was cursed when the doors of the tower were thrust open, which is strange since the tower doesn't have any doors in the game, but probably means when someone invaded the temple or something similar.
We don't know who did it.
The tribe that made Majora's Mask could have gone into the temple to seal the mask in the pit where Twinmold is (and in order to get to it you need to flip the tower so that it's leading downwards to the underworld or something, which may be a point of interest).
Then, much later, someone else, maybe the Garo or the interlopers or some unknown being or group invaded the tower and took the mask, and so the curse began and Ikana was doomed.
However, in the 3DS version, it's shown that the area bosses are connected to Majora, evident by imagery of Majora's Mask showing up in that purple haze briefly after defeating them. This would therefore mean that Twinmold is the source of Ikana's curse, and the boss was created, or perhaps used in some way by Majora. The bosses could be there as a seal to keep the Giants locked up or something like that, and defeating them undoes the seal.
But the point is, when Twinmold is defeated (like the other bosses/areas), the curse placed on Ikana is lifted, so Twinmold is the source, and it was created by Majora.
Now, one question is when this all happens.
If we take the other areas into consideration, then all four areas were cursed recently. Maybe it happened hours, days, or weeks before the game's main story begins. This in turn would mean that Ikana has only been cursed for a short time. If the kingdom fell into decline as a result of this curse, then the kingdom fell to ruin only a few days or weeks or so before the game actually begins, which is an interesting point to make.
The King of Ikana states that the kingdom fell, or the curse appeared when someone thrust open the doors to the tower. This someone could have been Skull Kid, who went to the temple to trap the Giant, make Twinmold etc, and this could have been hours or days or weeks before the game began, so the kingdom's ruin was very recent.
Perhaps based on what the game tells us, the player would think that the Garo or someone else did all this, and the kingdom was lost some time ago. So it would be interesting to think that Skull Kid actually did it shortly before Link arrives in Termina, which would make sense seeing as Twinmold is the source of the curse, or at least its destruction removes the curse, and the boss may have been created directly by Majora.
Would therefore mean that the curse only began as a result of Majora, and not because of anything that happened prior.
Just a thought.