How to time bomb a beta?

Started by
79 comments, last by Mouser9169 10 years ago

So if you were the programmer of Ocarina of Time you would have fixed all these bugs before the game shipped?

absolutely! assuming i could find them.

That assumption, right there, is really not a safe one. No offense, but even the best designed code with zero warnings vs. -Wall is still going to run into unforeseen logic errors...and the more complex the code, the larger the code base, the higher those odds. I doubt that your code is all so trivial as an empty main function. smile.png

(But, if you really do have a full proof way...you're in the wrong line of work. Software Hardening and Static/Dynamic Taint Analysis are both very vibrant fields of research; they could use you. Not to mention how every single other software package in the wild right now experiences bugs for the first several years of their life time in consumer hands.)
Advertisement


So if you were the programmer of Ocarina of Time you would have fixed all these bugs before the game shipped?

absolutely! assuming i could find them.

In a perfect world, sure -- but in reality that's not the way it works. Games ship with unknown and known bugs all the time. Like you, the engineers who take pride in their work don't like it, but can't press back on the publisher when they come in and say "Hey guys, you can't let the schedule slip over 20 known bugs that only 1 in 10000 people will be adversely affected by." I've got no love for slave-driving publishers, and I think that often its their own policies and practices that end up putting themselves and their studios in a hard place, but you really can't miss that holiday launch window if you were relying on it -- people will lose their jobs if you do, usually starting with the people at the bottom.

In your situation, perhaps you aren't relying on that window and you don't have anyone but yourself to answer to or worry about affecting, but as soon as you have a publisher to please, a marketing budget, or an employee who's not a partner in the company, the game changes.

throw table_exception("(? ???)? ? ???");


Implement a random obfuscation virtualization environment.

already done.

there is one in the game exe that does both game related and DRM related tasks,. it uses code generated on the fly, self modifying code, and encrypted code. all programs are written in its proprietary machine language. you write one line of assembly as a comment, followed by usually three lines of machine code to implement the one line of assembly.

there is a second separate stand alone vm that decrypts and runs the game exe, as well as running its own proprietary vm programs in general. it has its own proprietary architecture, instruction set, .programming language, compiler, p-code language, and vm runtime. programs to encrypt a game exe and decrypt and run a game exe are just two example programs written for the stand alone VM. the VM was made BEFORE the game, just for copy protection purposes. no sense making a game for profit if you can't protect it. this vm is much more sophisticated. i've actually written direct3d fullscreen test apps with it that run at 62.5 fps.

i look forward to the day when PC's are fast enough to just write a vm, then write the game in the vm's language - eliminating the dreaded release build delay from the development iteration loop. until then, i use a macro processor to generate c++ code from a high level, low keystroke scripting language. saves on typing, like a custom vm language can, but doesn't help with release build times.

however, as pointed out by ApochPiQ, VMs are not impervious to attacks:

from:

http://www.gamedev.net/topic/654995-checksum-crc-md5-hash-check-game-code-in-ram/


The VM concept may seem like a solid approach, but you have a critical flaw in your architecture.

An observant reverse engineer will notice within seconds of viewing your program's import table that you use a select few "interesting" API functions. Encrypted code is not a surprise to anyone in the reversing field; even "innocuous" things like packers have been around for ages, and most reversers know how to get past them.

If I were, hypothetically, to take a look at your program, this is how my thought process as an experienced reverse engineer would look:
Hmm, weird set of APIs. Also if I break the game execution at random times I end up in this weird call stack.
Let's see what this code does... oh, it's a VM.
Hey look! It's running encrypted code!
Let's find all the places that decrypt input code and hook them...
... aaaand shunt over to a dummy VM implementation in an injected DLL (or monkey-patched binary if I feel brave)
Lookit that, I can dump all the raw VM bytecode now
It's a trivial VM, so figuring out what each bytecode package does is just another reversing exercise
Wow, some of these look like they are game-critical...
But these ones don't! I found your DRM!
Surely there is some recognizable pattern to what opcodes or general code pattern is followed by DRM code versus game logic...
Little analysis later, some shunting back of the VM decrypt process, and poof...
And now I have a crack for your game.

A good reverser could pull this off in a long day of work; a mediocre one could do it in a week.


Unless you expect that buying yourself 24 hours of "safety" will be the difference between commercial success and commercial failure, you're wasting your time.

software seems to need two types of protection:

1. copy protection to prevent illegal distribution having a negative effect on legit sales.

2. anti-crack protection for the copy protection.

also, any game that has artificial built in limits that can be cracked (such as only play for free to level 20) needs anti-crack protection.

i've gotten exe checksum with no checksum in the exe and ram checksums working for anti-crack, and expiration date not based on system clock for DRM working.

i've been able to hide string constants, and pepper the code with tons of system calls to drastically increase the number of references to suspicious systems calls such as GetSystemTime.

one big thing i've done relates to the following quote:

"they can't hack what ain't there!"

so to the greatest extent possible i've used #ifdef DEMO and dead code elimination to completely remove full version code from the demo. in some cases i've had to write custom code for the demo, such as a generate_demo_map() routine.

but getting back to the ET phone home....

so the server sends some VM p-code to the client. the client runs it. the client doesn't store it on disk. the server must send vm code every time the game wants to run.

the packets could be intercepted and the p-code gotten at that way. a dummy client could be written that gets the p-code and saves it to disk. a ram spy tool could get the p-code from ram. and the cracker has the client exe , so they have the vm code. once they reverse engineer the VM, and get the p-code somehow, its just matter of time. needless to say, all of this is a bit easier said than done. but still do-able.

all these types of attacks would need anti-crack protection.

doing a google on your three suggestions turned up a bunch of useful stuff for a search on "random obfuscation virtualization environment".

i'll be posting all the links i found in my gamedev info journal here as soon as i get a chance to sift through it all, and at least provide a couple words for each link about what the info is about.

so it a two front war:

1. don't copy me illegally - copy protection.

2. don't crack the code that provides copy protection - anti-crack protection.

note that sometimes its not illegal copying, its time limits, cheating in multiplayer online games, etc. that you want to make tamper resistant.

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php


Runtime functions - Use VirtualProtect (or similar for your OS) and implement it. Is a lot of fun and you'll learn a lot.

so you DL some code to the client, and use:

BOOL WINAPI VirtualProtect() to set PAGE_EXECUTE ?

but they could still get a memory dump, couldn't they? then make their own code?

That really sums up the point we were getting at earlier: There is no perfect solution that can be isolated on the users machine -- you need a a trusted machine, or in a distributed system a quorum of neutral third parties, to have any semblance of security. A dedicated attacker will always be able to go one step further on their own turf. And lets not forget that their job is far, far easier than yours in this arena -- you must unfailingly block every attack -- they just have to wait for you to fail to do so one time. In the example above, lets say that you could prevent them from dumping memory -- sounds good, but a truly dedicated attacker could cool the physical memory, pull it from the system, and then probe it to read out your code. This is a known attack vector in high-end reverse-engineering efforts. Is it likely that a cracker will go to this extent to make your $20 cult hit free to the masses? Not likely, but it illustrates just the same that no matter how far you can go, it just isn't far enough to be perfect.

This is the reality you have to accept if you can't enforce security in a connected fashion. You can make a cracker work harder, but you can never make it impossible for them to win in the end. Many small teams have had success by marketing their lack of DRM as a feature, others have found creative ways to cajole pirates into paying up, others have found new monetization models that aren't deflated by piracy. No one has found a fool-proof way to stop pirates dead in their tracks.

On a further note -- if all this concern, at least early on, is about people being able to pirate and distribute your Beta -- what would be so bad about requiring your beta participants to connect to a server to validate? You wouldn't need to carry that policy forward for paying customers. Requiring an internet connection might possibly skew your beta feedback, but that's probably an acceptable tradeoff unless your target market is the rural farming set.

throw table_exception("(? ???)? ? ???");


That assumption, right there, is really not a safe one. No offense, but even the best designed code with zero warnings vs. -Wall is still going to run into unforeseen logic errors...and the more complex the code, the larger the code base, the higher those odds. I doubt that your code is all so trivial as an empty main function.

i totally agree. thats why i said "assuming i could find them". odds are in that title, they had a base engine, then kept adding more custom stuff to it. eventually almost every item becomes a special case with its own behavior and code. such "evolution" of the code is ok, but can lead to glitches down the road as things become bigger and more complex. its better to try to figure out a framework or system for the different things first, so undesirable interactions are easier to predict.


(But, if you really do have a full proof way...

unfortunately no. rigorous attention to detail and testing has allowed be to release games in the past with very few errors, and never with a major one. then again, apparently i'm the "judge / perfectionist" personality type when i'm on the clock (and the "happy hedonist" personality type when off the clock! <g>). typos in string constants are another thing though, there's no spell cheker in my IDE! <g>.

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php


In your situation, perhaps you aren't relying on that window and you don't have anyone but yourself to answer to or worry about affecting, but as soon as you have a publisher to please, a marketing budget, or an employee who's not a partner in the company, the game changes.

yes, i guess i'm blessed in the fact that i'm the developer, publisher, and financial backer. so all three are of like mind and one goal.

as a fan of the game, i know what needs to be done - hopefully - can't think of everything. this is where other playtesters help.

as the dev, hopefully i know how to do it, or can figure it out.

as a pub, i have the good sense to listen to the devs and not push half finished crap out the door. my job is marketing and sale fulfillment, not design, development, or project management. as a pub, i never forget i'm really nothing but a glorified car salesman - i don't build them, i just try to get you to buy them.

and as backer i have the wisdom to sit back and let my people do their thing, after all that's why i bet on them in the first place.

i've never really timed my releases around the holidays etc. i release when its ready. but i have found that releases made in summer and after the holidays do quit well. probably because they are released when there's little in the was of new AAA titles to compete for attention. summer releases get the user's dollars before the xmas AAA onslaught, and after xmas releases get the after xmas sales when users buy what they didn't get for xmas, or after they've finished the games they got for xmas.

. i did release caveman v1.x during september/october 2000. so it was hitting the radar about xmas time. NBC evening news in Washington DC picked it up as a last minute xmas gift idea. got so many hits, it crashed by website. took a week to get back up again, as i was going through a local mom and pop webhosting reseller who was closed for xmas.

Caveman will be released when ready, perhaps a couple months from now. but there's still so much to do, i may have to release a version 3.0 and immediately begin on version 3.1 to get it all in there - stuff like the "bring on the dinos!" option. <g>.

cavemen with javelins and knives vs wolley mammoths is already a sight to see, i can only imagine what it would be like vs t-rex! actually i don't have to. t-rex is one of the animals implemented in the game already but not used. but i can trigger a t-rex encounter from the playtest menu. cool! i'm going to have to try that! and post screen shots! <g>. a little preview of the "bring on the dinos!" option.

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php


On a further note -- if all this concern, at least early on, is about people being able to pirate and distribute your Beta

actually, i got all that drm and anti crack stuff working, then tuned it off in the demo and beta! <g>.

you can email me right now and get a link to the beta demo with nothing more than watermarks, an unprotected expiration date, and an unprotected 30 gameday limit.

even cracking all that, they'd only get a game with a 100x100 mile world, with no load or save, and many other high end features missing. only the watermarks, nag screens, 30 gameday limit, and beta expiration date are crack-able. nothing else can be cracked - "cause it ain't there!". thank god for #fidef DEMO, and especially dead code elimination and link time code generation (that's the one that only links code actually called, right?).

i've even considered releasing the game in a "Caveman III, version 0.x" format for free. this approach is popular with indie games under development. they give away what they have while under development, eventually evolving their code base into a stable release version 1.0, which they then sell. sort of a long term open beta.

but caveman may be too close to release for this.

as it stands now, the complete version makes for quite a game that would take a LONG time to play.

unlike most rpgs, you can have more than one player character (band member) under your control - like household members in the sims. you can tab between band members at any time. you can have up to 10 band members in your band. you can recruit new band members at any time you have less than 10 band members. and the game doesn't end until the last band member dies. so the original band member, the character you created when you started a new game, might die, but the game continues with the remaining band members, who can then recruit a new band member to replace the dead founding band member. so the game could theoretically go on forever - despite the fact that back ground radiation modeling (IE old age) will eventually kill off individual members of the band from time to time. band members may die, but the band lives on.

so its a pretty big game already, despite the fact that i have yet to finish the quest generator, raiding and inter-band rivalry, mating and offspring, modern animal types, aquatic animal types, "bring on the dinos!" etc.

 
 
 
fn v demo_intro_screen
i y dy
= y 100
= dy 25
c Zbeginscene
c Zclearscreen 0 0 0
c Zbeginsprite
c tx2 400  y "This is a playable demo version of Caveman 3.0."
 
c tx2 400 y+dy*2  "Limitations of this demo:"
c tx2 400 y+dy*3  "1. A 100x100 mile game world."
c tx2 400 y+dy*4  "2. The same hard coded game world every time you play."
c tx2 400 y+dy*5  "3. Only 4 types of animals."
c tx2 400 y+dy*6  "4. Encounter types chosen totally at random."
c tx2 400 y+dy*7  "5. No quest generator or quests."
c tx2 400 y+dy*8  "6. No tools, editors, or testing cheats like teleport, kill all "
c tx2 400 y+dy*9  "   hostiles, recharge all stats, or view unexplored map."
c tx2 400 y+dy*10 "7. No load game, save game, or autosave."
c tx2 400 y+dy*11 "8. You can only play for 30 game days."
c tx2 400 y+dy*12 "9. No hiring warriors to fight for you."
c tx2 400 y+dy*13 "10. No friendly NPCs as traveling companions."
c tx2 400 y+dy*14 "11. No getting your travelling companions to join your band."
c tx2 400 y+dy*15 "12. No God artifacts. - Hey, whats an rpg without artifacts, right?"
c tx2 400 y+dy*16 "13. No high quality objects. - the Caveman equivalent of magic items"
c tx2 400 y+dy*17 "14. No Caverns: dungeon adventuring - caveman style!"
c tx2 400 y+dy*18 "For more information on the full version, visit:"
c tx2 400 y+dy*19 "https://sites.google.com/site/rocklandsoftware/"
 
c tx2 400 y+dy*21 "Click to continue..."
c Zendsprite
c showscene
nb 
p
nb
 
 
 
c Zbeginscene
c Zclearscreen 0 0 0
c Zbeginsprite
c tx2 400  y "This is a playable demo version of Caveman 3.0."
 
c tx2 400 y+dy*2  "Features only available in the full version:"
c tx2 400 y+dy*3  "1. A 2500x2500 mile game world."
c tx2 400 y+dy*4  "2. A randomly generated true world with islands, mountain chains,"
c tx2 400 y+dy*5  "   forested areas, deserts, jungles, tropical savannas, etc. All"
c tx2 400 y+dy*6  "   terrain types are included."
c tx2 400 y+dy*7  "3. A completely new randomly generated world every time you start"
c tx2 400 y+dy*8  "   a new game."
c tx2 400 y+dy*9  "4. Over 50 types of actual extinct Mega-fauna to hunt or be hunted by!"
c tx2 400 y+dy*10 "5. Encounter types based on terrain type and animal frequency."
c tx2 400 y+dy*11 "6. Quest generator and quests."
c tx2 400 y+dy*12 "7. Tools, editors, and testing cheats like teleport, kill all "
c tx2 400 y+dy*13 "   hostiles, recharge all stats, and view unexplored map."
c tx2 400 y+dy*14 "8. Load games, save games, and autosave."
c tx2 400 y+dy*15 "9. No time limit. The game ends when your last bandmember dies."
c tx2 400 y+dy*16 "10. Hire warriors to fight for you."
c tx2 400 y+dy*17 "11. Friendly NPCs as traveling companions."
c tx2 400 y+dy*18 "12. Get your travelling companions to join your band."
c tx2 400 y+dy*19 "13. God artifacts. - Hey, whats an rpg without artifacts, right?"
c tx2 400 y+dy*20 "14. High quality objects. - the Caveman equivalent of magic items."
c tx2 400 y+dy*21 "15. Caverns: dungeon adventuring - caveman style!"
c tx2 400 y+dy*22 "16. No demo nag screens, 'feature unavailable in demo' messages,"
c tx2 400 y+dy*23 "    or demo watermarks."
c tx2 400 y+dy*24 "For more information on the full version, visit:"
c tx2 400 y+dy*25 "https://sites.google.com/site/rocklandsoftware/"
 
c tx2 400 y+dy*27 "Click to continue..."
c Zendsprite
c showscene
nb 
p
nb
.
 
 
 

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php


lets say that you could prevent them from dumping memory -- sounds good, but a truly dedicated attacker could cool the physical memory, pull it from the system, and then probe it to read out your code. This is a known attack vector in high-end reverse-engineering efforts. Is it likely that a cracker will go to this extent to make your $20 cult hit free to the masses? Not likely, but it illustrates just the same that no matter how far you can go, it just isn't far enough to be perfect.

drm anti-crack protection is not a battle to be won. the objective of stopping cracking entirely is almost impossible - with trusted machine (and user) perhaps the only way - like the special PCs we had at wright pat air force base for classified data.

instead, drm is a delaying tactic. the objective is to make it more hassle than its worth.

and as you say its unlikely a cracker will go to extraordinary lengths to crack my $20 cult hit. i'm sort of counting on it in fact. the old version that got cracked has zero anti-crack protection, and was a textbook case of the simplest code to crack - find string constant, then modify nearby jmp instruction, done!

so the plan is to only put drm and anti-crack in the full version. no need to give them an advance look at the code in a demo or beta. they'll have to buy or get the game from a legit user to crack the full version.

and the full version will strive to make the ratio of "glory to be gained by the cracking" vs "hassle involved in the cracking" so unfavorable that no one goes to the effort.

if i were a big AAA game, the glory would be much greater. sometimes its nice to be the little guy.

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php

the game world contains 60,000 caves, which can be unoccupied, or occupied by animals or cavemen.

it contains 5000 rockshelters - which are similar to caves, but only found along cliffs.

it contains 18,000 huts, which are occupied by cavemen.

it contains 20,000 caverns, which are the equivalent of a dungeon level, or a doom level, or a cave level in skyrim. caverns are generated when you enter them the first time, and saved when you leave. when you return, they are reloaded. caverns repopulate over time.

a new world is generated each time you start a new game - so its random, not hard coded content like skyrim. the entire world is saved when you save a game, so its persistent. and you can build and demolish structures, etc, so its modifiable.

TBH, this sounds to me like a game world around a hundred times bigger than it needs to be. How many players are going to go through 20,000 caverns without starting a new game? If that's the measure of the size of game, then Torchlight (and FATE) are even bigger since they have 'infinite' dungeon levels (or close enough anyway). Saying your game is 'as big as Skyrim' is a poor comparison, since a game like Skyrim is measured in terms of content, not how far you can run before you hit the edge of the world.

I hope you're right and your game isn't cracked - but I also hope that your game is fun to play (some games aren't cracked because no one wants to play them...).

Btw, it also sound like a game where the performance is going to keep dropping the longer you play. In the beginning nothing is generated, so the system resources needed are scant. After a few thousand caverns have been formed though, will the game still be able to run as fast and smooth? Something for your beta testers to look into - maybe include a 'generate caverns' cheat, so they can test the game with more and more caverns and caves and shelters being stored.

"The multitudes see death as tragic. If this were true, so then would be birth"

- Pisha, Vampire the Maquerade: Bloodlines


Saying your game is 'as big as Skyrim' is a poor comparison, since a game like Skyrim is measured in terms of content, not how far you can run before you hit the edge of the world.

then its not as big as skyrim, athough measured in content it might well be as well. dunno, don't really care. i pretty much used up skyrim in about 2 months of heavy play. i didn't even get close to using up caveman v1.x in 3 years of long term playtesting. but then again its designed that way, to be a continuous paleo-world simulation that will run forever.

frankly, i'm not interested in whether to do it, or who does it and who doesn't, or how many copies they sold without it. i'm interested in technical info on howto. THAT is the topic at hand. other posts strike me as being off topic, though some may see "whether to or not" as a more fundamental question to be answered first. if you have a suggestion for a drm free way of selling the title in question that will work pretty much for sure, given a title with a proven track record, please share.

you seem to know a lot about torchlight, is there a cracked version ? (make that freely downloadable - it doesn't have drm). they might make a good case study of how to do it drm free. $15 price point, diablo clone, steam and direct web sales, correct? did they do a pre v1.0 release or large public beta to build a fan base first?

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php

This topic is closed to new replies.

Advertisement