I've heard speculation before that the tribe which worshiped Majora was connected to the Interlopers. And as it stand in the game, the Happy Mask Salesman was the foremost authority on said tribe and the power of the mask itself. He also seems to know a lot of magic, helping Link return to his...
While it may have been formed from a wayward branch of Sheikah, it's unlikely the Yiga specifically were the interlopers. We actually get some backstory for the Yiga clan from "Creating a Champion". 10,000 years before the Great Calamity that put Link into his 100 year nap, the King of Hyrule...