Jump to content

  • Log In with Google      Sign In   
  • Create Account


detect corrupt savegame file - best practices


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
11 replies to this topic

#1 Norman Barrows   Crossbones+   -  Reputation: 1973

Like
1Likes
Like

Posted 10 March 2014 - 09:13 AM

So whats a good way to detect a corrupt savegame file?

 

I ran into a corrupt savegame file this morning while playtesting Caveman.

 

http://www.gamedev.net/blog/1730/entry-2258672-caveman-v30-general-desciption/

 

thanks to round robin save and round robin autosave, i had three other good recent saves to chose from.

 

but the only indication that the file was corrupt was overflow of hut and cave indexes after loading the CRH (cave / rockshelter / hut) database.

 

perhaps a checksum added to the end?

 

 


Norm Barrows

Rockland Software Productions

"Building PC games since 1988"

 

rocklandsoftware.net

 


Sponsor:

#2 aregee   Members   -  Reputation: 925

Like
0Likes
Like

Posted 10 March 2014 - 12:01 PM

Checksums are certainly a way to check for corrupt save games.  Checking for impossible conditions or illegal values are of course another.  Checking that sections of your save game is the size they should be, or even if the size given would go beyond the size of the save game file itself.  If you have sections with identifiers, you could also check if a sections reported size would pass the beginning of the next section.  If you have sizes for sections, you could also check if a section is following immediately after where it is supposed to follow.  In all cases, you will want to decide wether you want to try to recover the save game to a sane or reasonable state, if possible, or if you want to reject the save game altogether to be safe.  I tend to be paranoid about checking conditions to be sure a save game won't put the game in a weird state.

 

If you have ever parsed an Amiga Sound Tracker module song, you know what I mean.  That is the most horrible "format" ever made, or lack of format, due to differences between versions, and organic expansion during the years it was common to use that format.  There were maybe 50 different "Trackers" made, and some were using the same codes for different things, WITH NO VERSIONING or identifier to tell which "Tracker" you made your song, even with the format being the same between versions that does different things.  A song will play in almost any tracker, but if you want it to play correctly, you have to do some qualified 'guessing' by seeing different hints stored in the save file.  One of these hints is: Is song name or instrument names written in ALL CAPITAL LETTERS?  Then you can assume it was a really old version.  Of course, it might be just the author actually writing the names in capital letters on a newer version...



#3 nobodynews   Crossbones+   -  Reputation: 1802

Like
1Likes
Like

Posted 10 March 2014 - 12:40 PM

Yeah go with a checksum. If you want something a little more robust than a simple "add up all the bytes" method, consider a CRC checksum. It shouldn't be too hard to find a free implementation, but there's one I found here along with an explanation: http://www.barrgroup.com/Embedded-Systems/How-To/CRC-Calculation-C-Code

C++: A Dialog | C++0x Features: Part1 (lambdas, auto, static_assert) , Part 2 (rvalue references) , Part 3 (decltype) | Write Games | Fix Your Timestep!


#4 jHaskell   Members   -  Reputation: 977

Like
3Likes
Like

Posted 10 March 2014 - 02:56 PM

Checksums will provide some protection to external corruption, but if your game is the source of the corruption, and that corruption gets computed into the checksum, then the checksum accomplishes nothing.



#5 frob   Moderators   -  Reputation: 19786

Like
4Likes
Like

Posted 10 March 2014 - 03:52 PM

CRC is common, cryptographic hash is frequent but less common. Also the individual components being loaded should validate their data, and when they detect an out-of-range value some proper response should be taken.

For example, I spent the last two years working on The Sims where DLC could be installed, removed, upgraded, or downgraded between saves. If the file was detected as corrupt we would notify the user and not load the file. When an object is loaded but has out-of-bounds values we don't load the customized data and reset to defaults instead. When something in use was uninstalled between play sessions, such as furniture, we would replace it will a default fallback and provide notice to the user that objects were being replaced. Sometimes when there was no good option for replacement (because nothing like it exists in base game) we would go with party balloons for a replacement object.

Most consoles and handhelds specify minimum standards for detecting and dealing with the files, including guidelines for recovery or mandatory deletion of invalid files.

As for solutions, one I have seen used occasionally is a two-of-three storage system. It is robust and occasionally used for cartridge-based memory, but few studios used it because it slightly raised the cost of goods. Basically you have three slots and two versions. You start with an old version (denoted by lowercase) and replace with a new version (denoted by uppercase) writing out [Abc] then [ABc] and then [ABC]. If there is a catastrophic error, such as a read error or a cartridge pulled when writing, you always have at least one complete file that is valid. During the first write you can recover the original save. During the second write you can recover either the newer or older save, and during the final write you can recover either of the new saves. You can then use one slot for a "quicksave" system if you are fine with less redundancy.

Edited by frob, 10 March 2014 - 04:00 PM.

Check out my personal indie blog at bryanwagstaff.com.

#6 aregee   Members   -  Reputation: 925

Like
0Likes
Like

Posted 10 March 2014 - 04:00 PM

Checksums will provide some protection to external corruption, but if your game is the source of the corruption, and that corruption gets computed into the checksum, then the checksum accomplishes nothing.

 

Just out of genuine curiosity, how can a checksum provide protection to external corruption?  If I sound critical, I am not, just genuinely curious as I always want to learn more.  + for mentioning the case where your game may be the cause of corruption, baking it into the checksum.

 

 

Yeah go with a checksum. If you want something a little more robust than a simple "add up all the bytes" method, consider a CRC checksum. It shouldn't be too hard to find a free implementation, but there's one I found here along with an explanation: http://www.barrgroup.com/Embedded-Systems/How-To/CRC-Calculation-C-Code

 

Just curious, what implications would there be using a MD5 hash instead of CRC checksum as a checksum?



#7 Norman Barrows   Crossbones+   -  Reputation: 1973

Like
1Likes
Like

Posted 10 March 2014 - 04:58 PM


Sometimes when there was no good option for replacement (because nothing like it exists in base game) we would go with party balloons for a replacement object.

 

LOL!  Oh, i love that one!  very SIMs! <g>


Norm Barrows

Rockland Software Productions

"Building PC games since 1988"

 

rocklandsoftware.net

 


#8 Norman Barrows   Crossbones+   -  Reputation: 1973

Like
1Likes
Like

Posted 10 March 2014 - 05:09 PM


Just out of genuine curiosity, how can a checksum provide protection to external corruption?

 

it provides detection, not protection.

 

write out the entire savegame file, adding up the bytes as you go, or write it out then read in and add up the bytes. this gives you a checksum. the sum of all the bytes in the file added together. you then store the checksum someplace like the end of the file. when you read the file in, you add up the bytes again, and check the result vs the stored checksum. if they don't match, you know the file has been corrupted.

 

crc checks are similar, but reduce the chance that two erroneous bytes might cancel each other out and leave a simple checksum thinking everything was ok.

 

there are also methods of error correction coding where all stored values differ by at least 3 bits. so an off by one bit error can be error corrected to the closest legit value. as i recall, EBDIC is an example of this. such codes are commonly used in telecom and networking protocalls, where transmission errors are a concern.


Edited by Norman Barrows, 10 March 2014 - 05:11 PM.

Norm Barrows

Rockland Software Productions

"Building PC games since 1988"

 

rocklandsoftware.net

 


#9 Norman Barrows   Crossbones+   -  Reputation: 1973

Like
0Likes
Like

Posted 10 March 2014 - 05:35 PM

hmm,

 

power outages are the cause, not program errors - i double checked to makes sure the reads and writes match up. and there was a power outage soon after the file was saved, so it was already suspect.

 

real data validation would be a lot of work. however it was a basic form of data validation (data structure overflow error message) that first clued me in to something not being right with the file.  otherwise i might have continued happily playtesting away with a corrupt game loaded. THAT would have sucked!

 

crc would be more reliable than checksum.

 

anything beyond crc is probably overkill.

 

the game uses round robin saving and autosaving, with 2 slots each.  so if one goes bad, i have three good ones to work with.

 

in the past i've relied on the built-in game editor and playtest controls to fix corrupt / obsolete save games.

 

trying to automatically recover a saved game would be a lot of work.

 

probably best to go with crc for detection, and load previous save as the recovery method, discarding the corrupt file. that's what i did when it happened yesterday. but i had to do it manually (rename files to force it to load the older savegame). with crc check, it could do it automatically if a corrupt file was detected.


Norm Barrows

Rockland Software Productions

"Building PC games since 1988"

 

rocklandsoftware.net

 


#10 Bacterius   Crossbones+   -  Reputation: 8315

Like
2Likes
Like

Posted 11 March 2014 - 12:37 AM


Just curious, what implications would there be using a MD5 hash instead of CRC checksum as a checksum?

 

There are none, really. MD5 basically guarantees you will detect accidental corruption with essentially 100% probability, while a CRC checksum will detect accidental corruption with about as good probability (not nearly as good, since it's not cryptographic, but well above 99.9% since it's after all a checksum). The CRC checksum will completely fail at detecting intentional corruption, MD5 will handle that with no issues (MD5's collision resistance is broken, but in this scenario only preimage resistance matters and MD5 is still strong with respect to that, though you could do better). Computing a CRC checksum is slightly faster than computing an MD5 hash, but not by much (this is because while MD5 does a lot of work, most CRC's operate on a per-byte basis and do a complex finite field polynomial multiplication which is rather expensive)  An MD5 fingerprint is four times as long as a CRC checksum. Take your pick.


The slowsort algorithm is a perfect illustration of the multiply and surrender paradigm, which is perhaps the single most important paradigm in the development of reluctant algorithms. The basic multiply and surrender strategy consists in replacing the problem at hand by two or more subproblems, each slightly simpler than the original, and continue multiplying subproblems and subsubproblems recursively in this fashion as long as possible. At some point the subproblems will all become so simple that their solution can no longer be postponed, and we will have to surrender. Experience shows that, in most cases, by the time this point is reached the total work will be substantially higher than what could have been wasted by a more direct approach.

 

- Pessimal Algorithms and Simplexity Analysis


#11 aregee   Members   -  Reputation: 925

Like
0Likes
Like

Posted 11 March 2014 - 06:57 AM

trying to automatically recover a saved game would be a lot of work.

 

I know I suggested to consider to automatically recover a saved game, and you are of course right.  Just had a big laugh imagining trying to automatically recover a save game in bigger games, like for instance Skyrim...  Picturing a LOT of code, and possibly a very disappointed player in the end, and it might not even be feasible.  I think redundancy is the cheapest and easiest way to go, when I think more closely about it.



#12 Norman Barrows   Crossbones+   -  Reputation: 1973

Like
1Likes
Like

Posted 11 March 2014 - 08:36 PM


Just had a big laugh imagining trying to automatically recover a save game in bigger games, like for instance Skyrim..

 

that's about the size of the game i'm working with. a core dump of the entire game state is about 67 meg. its a persistent open world, 2500 miles x 2500 miles in size, with 60,000 caves, 5000 rock shelters, 18,000 huts, and untold (maybe 2500?) procedurally generated caverns (dungeons).


Norm Barrows

Rockland Software Productions

"Building PC games since 1988"

 

rocklandsoftware.net

 





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS