I've got an idea for a horror game where your character isn't limited by clunky tank controls and awful combat, because I find that limits the player's ability to take the game seriously enough to be scared. Once you start yelling at the screen to stop being such a pansy, all the thrills are kinda gone. Rather, the horror will come from you being stupidly outmatched, because no matter how athletic one human being may be, we're still squishy and breakable.

The problem is, I'm not 100% sure how to balance this so that it remains scary. There won't be any crazy Devil May Cry combos, no easy Zelda-style defenses, but you will be able to sprint, jump, strafe and pick up melee weapons. What can I do to stop that from becoming Ico?

I liked Ico, thank you very much. That aside, I did do quite a bit of reading on making game content scary and it all comes back to psychology. You have to strip away the things that make the player feel secure and by doing so you will create an atmosphere of horror.

If the player feels powerful, then the game will not come across as particularly frightening except for the occasional jump scare. So the first step is to remove the players sense of power. Don't go overboard with this, however, because making the player feel too helpless will only serve to become frustrating. A good balance is not letting the player kill enemies, but allowing him to temporarily drive them off. Imagine unloading a pistol into some shambling hulk, worrying to yourself as it shambles nearer and nearer despite your hail of fire. At the last moment, it decides you're not currently worth the trouble, glares at you for an unsettlingly long moment with bemused annoyance, and shambles off towards some unknown goal. Psychologically, this suggests that your efforts weren't meaningless, but that you didn't really win the fight since your enemy can always come back at you.

Permanence is another security blanket. Eternal darkness did a good job with breaking your sense of permanence. Doors would sometimes go to the wrong places, or objects in rooms would change enough for your mind to notice but not for your conciousness. Monsters would frequent a hallway often enough for you to expect them there, only to suddenly spawn in a different, unexpected room. When you remove permanence, you remove predictability, and humans fear the unpredictable.

Connected with predictability is comprehension. Make your enemies incomprehensible. I can't remember the title, but one game had several enemies all hunting the hero for absolutely insane reasons (one wanted a toy, one wanted to torture her, one wanted to experiment on her, and the robot wanted her uterus to become a "real woman"). These are fairly cliche, but it makes the characters unrelatable and therefore alien. Monsterous enemies should be even more incomprehensible, like a monster that will stop trying to eat you to stare intently at a picture... then rush you again. Or a monster that only attacks you if it sees you in the mirror.

A final security blanket has to do with conditioning. I still jump when I hear that old doom imp growl, or the sound of being spotted. When something bad happens, make a jarring sound to condition the player to expect negative consequences. Once this has been well established, you can spook the player by playing the sound, or make them think about bad situations by playing sounds that remind them of the alert sound. A dangerous monster that rushes you from corners might make a low rumbling noise. Teach the player that when they hear that noise, they have a short while to find it and scare it off before it charges them (and kills them). Later, include rooms that occasionally play the sound, but have no monster in them.

If ICO had those elements, it would have been a lot more horror oriented, I think. But it didn't need them. Because ICO rocks

XD I loved Ico, but it's a totally different genre. Thanks for the detailed response though!

I found that weapon conservation in Silent Hill 3 really added to the tension too, what with the constant sense of "God there's so many damn hell hounds here! If I could kill just one--! But wait, I might need that ammo later...unless they maul me before I can get any further. Crap..." I'd considered using a clean interface for my game, so if I add weapon durability without a convenient meter to measure it by, would that have the same effect? That way, you can still tell that your trusty kitchen knife is about to fail you, but it's all you've got so you hang onto it and try to get through with as few fights as possible, because you can't really be 100% certain when it's going to snap.

Good call with the permanence thing though -- I hadn't thought about that. It would really highlight another theme of the game, where the only thing you do get to feel familiar toward is a bird carcass you're carrying around in your pocket. It makes so much sense: if I keep everything else in a constant state of flux, then that would make the dead bird even more of an emotional anchor for the player, huh?

As far as alien enemies, the first monsters I've thought of are clothes draped in the form of a person. They're hollow, but if you damage them then they bleed profusely. If they spot you, they shriek furiously and run hell-for-leather toward you, beating you 100 times a minute while ranting incoherently--
and then suddenly stop when you're within an inch of your life. They forget you exist, fall quiet, and stroll around listlessly. Left with your wounds, you have to hobble away.

Good points from Zouflain. You want the player to experience a loss of control, but without a loss of agency. Amnesia did this well - the player has an objective, but no weapons. The character is very fragile, and can even lose sanity. I gather that most failure conditions result in passing out and finding yourself in a different location with things moved and monsters relocated, which can be much more disorientating and frightening than a death screen.

Personally for the clothes monsters I don't like random wandering. I like the idea of collapsing back into a pile of clothes. This would make all clothes piles in the game objects of suspicion. For drama they could just collapse, stalk away and collapse, or run full speed at a wall and collapse when they hit.

Why not take advantage of advances in genres outside of horror? Like stealth action. Usually you only feel tension in a stealth game when getting caught actually has consequences. Like maybe if you get caught, you'll be swarmed by guards who will almost definitely kill you; or the mission will become much harder. You suddenly want to play the game right, rather than run through with your guns up (like in MGS4, the least stealthy "stealth" game ever).

This relates to horror because it gives the enemies a true power over the player. You can take away this threat of mission failure or being captured, and make it into a fear of being brutally murdered. You can combine this with other ideas.

For example, analyze real-life fears. What is it that stops people from fighting in real life when outside consequences don't matter? Why is it that some people will get hit and still say "I don't want to fight you"? It's because we're afraid of what you've already said: Human beings are squishy and breakable. We've all heard a story or two about some guy that got punched in the face once and died on the spot. We're cognizant of the fact that we're fragile. And so we're terrified of things that might potentially break us. Use that to your advantage. Have the player be completely fragile. Remove all HUD, so there's no health bar, and make it so just a few hits could kill a player.

You could make it so enemies are just as fragile (maybe not all enemies). This will tempt the player into sometimes defending himself, because it means he does have a chance. But the fact that he too is fragile will make him doubtful. Doubt is good. Doubt is scary.

And then you could combine that with something someone else mentioned in the comments above: Remove predictability. Like I said, we've all heard of a guy who died from one punch. That is the unpredictable nature of any conflict, be it physical, verbal, or otherwise: You don't know if the thing that worked the last few times will work this time. And you don't know if the thing that didn't work against you last time will for some reason actually work this time. This is playing into doubt again. Make the player unsure of whether an enemy will actually kill him in one hit. This doubt might tempt him into going at some enemies, but make him cautious against others. I saw this a lot in Demon's Souls/Dark Souls and it was gut-wrenching.

But as you've probably already realized, a lot of this stuff goes against typical game design rules: What the player expects to happen should always happen, etc, etc. But I think horror games will have to break a lot of rules to actually ever scare anyone.