A game can be made difficult for me by a number of circumstances. The genre of the game usually dictates what those circumstances are, so I will categorise my thoughts
FPS
Enemies have an unusually high amount of health and insanely good/powerful weapons while I get jack and squat.
Side-scrollers
An overabundance of enemies usually does the trick. In the case of shooters, such as Contra, the common case of "bullet hell" is always a pain.
Stealth
More often than not, I am my own worst enemy in stealth games. Whenever something happens to me in the game, it's almost always directly my fault. Not something that the programmers put in, but my own inability to be the ninja that I think I am when I play these games.
Adventure
Too little health, no recovery items, too many foes, the list goes on and on for these. However, much like the stealth games, it's usually just me overestimating my ability or my occasional incompetence.
Survival/Horror
Turning on the game is the most difficult part for me. I do not like horror games. I mean, movies are one thing, but having to be the person that gets mauled by zombies or attacked by demons is another. Even Metroid Prime scared the pants off of me when I was younger, and it still gives me the occasional scare. In any case, once I actually pick up the controller and, by extension, the cheap little pistol (and you know it's always a cheap little pistol), the only problem I have is ammunition (and trying not to accidentally throw my controller across the room when things jump out at me). I might as well be trying to fight everything off with a rock.