I think he spends his free time in an internal monologue. Plotting, planning, whatever it may be. I don't see him doing anything with his henchmen or subordinates. Unless they have "company" outings like a barbecue and "office" games. Hah.
Or he works in his "Gloffice" (Goblin/Office). A reference to the wonderfully hilarious web series, Legend of Neil. Inside this terrifiying Gloffice, he plans his next attack on Hyrule and how to get rid of that pesky fairy boy while holding that snotty princess of destiny in his dungeon!