True. This works if the game world is small enough, or takes effort to travel in. Keeping track of people in-game and their reputation on a piece of paper quickly gets tiring once you've bumped into over 100 PKs, and spreading word of them is an effort as well.I can imagine bots would need a reputation system.. but not humans.
If someone steals from you ingame.. he loses reputation with you.. because you dislike him now and know hes a thief.
Now if characters have a territory of sorts, home ground where they hang out and rarely travel beyond you can familiarise yourself with the somewhat established players in that area and keep track of them. Same goes if the game server only hosts a few number of players, but that's not really an MMO then.
But if all the tens of thousand of players they all can teleport, gate, fly or somehow very easily navigate the game world you probably want an easy way to keep track of players.
I was primarily thinking of player-run merc-corps, and some kind of advertisement or contract system or channel where you can easily get protection.Mercs are available in all games.. just that most games have made it so its either not worth paying someone to guard/fight for u or that you cant get paid enough to make it worth fighting/guarding someone else. But theres always anti-pks that will do it for free.
This is true as well. Ideally no artificial punishment system would be needed. If it's done in spirit of your wild west it could work, just give players the tools they need, and they'll make their own laws, and band together.PKs should not be auto punished thru some killing penalty system for killing someone.. or else whats the point? just to grief? I thought you didnt want griefing.
I reckon games like this in itself would work, provided they make it easy for new players that don't know the laws or faction, to get into the these factions and become part of them.
I think the real problem is to attract players in the first place like you said.
Sheep friendly mmos like WoW will get the people that just want to have fun with friends in dungeons, while pvp-people will search out pvp-dedicated games.
It takes a certain personality and attitude to want to expose yourself to danger in a non-pvp focused game; I imagine it's the same people that would roll hard-core characters in diablo
And as you said it's rather niche.
So, closing though. While doable, it wouldn't be as profitable as an MMO could be. Or so I believe.

Find content
Not Telling