• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
SteveDeFacto

Files can always be recovered no matter what you do...

34 posts in this topic

At least that is what my NOS-110 teacher said. This automatically got me thinking about ways to permanently delete a file without possible recovery. It seems simple enough to me. Write over the bytes of the file with random values and use a seed value from a unrecoverable source. For example the computer clock, CPU temperature or internet connection speed. Make sure not to write over bytes which have to do with the file system since they are in known patterns and could be used to uncover the seed value by checking every single possible value for the seed using the algorithm which was used to generate the random number. This should naturally be avoidable by using a standard file open command to write the data which I believe should avoid writing to any known structures in the file system. Unless however the file type is know and includes a standard header. However one could also make the permanent deletion algorithm avoid file type headers.

So anyway I told my teacher that I was pretty sure it was possible to permanently delete a file using an unknowable seed value to overwrite the file with random values. This made the teacher go into a long rant about how I think I know everything and telling me all of his qualifications. Then he went on to explain that no values on a computer are random and once you know the algorithm you can hack any encryption. I tried to explain to him that you may know the algorithm but an unknowable seed value is easily obtained. I'm almost certain I'm right but he humiliated me in front of the whole class... Anyway, what do you guys think? Am I right? If not then why?
0

Share this post


Link to post
Share on other sites
[url="http://en.wikipedia.org/wiki/Degaussing#Degaussing_magnetic_data_storage_media"]http://en.wikipedia....a_storage_media[/url]

Also see:

http://en.wikipedia.org/wiki/Data_remanence
0

Share this post


Link to post
Share on other sites
I'm fairly sure you can just write to your disk until you run out of memory to make erased data unrecoverable in most cases...

There's also the [url="http://en.wikipedia.org/wiki/Gutmann_method"]Gutmann Method[/url], though I know little about it's necessity.
1

Share this post


Link to post
Share on other sites
Even if you write over a file with (unknowable, truly random) random bits, forensic analysis can still detect the original values of the bits in some cases. Think of it like, how when you erase a page of pencil-writings, and then scribble over it, there will still be a faint impression of the original text on the paper. Same thing happens with magnetic disks, so you need to scribble really hard to make sure no-one can decipher those imprints.

The standard practice that I know of for permanently deleting a file is to write over it will all zeros, then write over it with all ones, then write over it a few times with random bits, [i]and then to smash the hard-drive enclosure with a hammer and discard of it in protected land-fill.[/i]

As for truly random numbers -- electronic slot machine regulations in some jurisdictions actually require truly random numbers, not just pseudo-random numbers. These are generated from a real RNG device ([i]which measure entropy from radiation, decay, thermal noise, etc[/i]), not a PRNG algorithm, and it is truly impossible to predict the output numbers, even if you know how it works.
0

Share this post


Link to post
Share on other sites
The issue isn't with overwriting with randomness, it's that hard-drives are analog beasts in a digital world. Even when you write '1' to the bit, it's not completely totally a one. It's just more than .5 so the hard-drive says it's a one. A dedicated attacker can crack open the spindles and take an electron microscope to it to get a better picture of what the bits used to be.

In software, the best you can do is write to the drive random bits 13-25ish times. Otherwise, wiping the drive with industrial magnets or causing the platters physical/chemical damage is your best chance of obscuring the data on the drive. Blossum's links are good.

So you're both kind of right; 'just' deleting a file won't do it. Even overwriting garbage won't do it, but you can eventually get to a state where the information is sufficiently obscured to be considered 'lost'.
0

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1316742131' post='4864968']
Even if you write over a file with (unknowing) random bits, forensic analysis can still detect the original values of the bits in some cases.

The standard practice that I know of for permanently deleting a file is to write over it will all zeros, then write over it with all ones, then write over it a few times with random bits, [i]and then to smash the hard-drive enclosure with a hammer and discard of it in protected land-fill.[/i]

As for truly random numbers -- electronic slot machine regulations in some jurisdictions actually require truly random numbers, not just pseudo-random numbers. These are generated from a real RNG device ([i]which measures entropy from radiation, decay, thermal noise, etc[/i]), not a PRNG algorithm.
[/quote]

Even that would be less effective than what I proposed since you are effectively reducing the magnetic field, increasing the field, and lastly using a PRNG algorithm to scramble it. To recover the data one would need hyper sensitive equipment to read off the data then reduce the values down to get the background noise. Then you simply read the bits that make up the header of an NTFS or FAT file system and run them through an algorithm to reverse PRNG randomization with every single possible seed value. You then amplify the background noise yet again and check to see if the data matches the standard header for that file system.

But if you avoid knowable headers you can just overwrite the data once with random values and result in almost absolutely irreversible deletion.
0

Share this post


Link to post
Share on other sites
[quote name='SteveDeFacto' timestamp='1316743379' post='4864978']you can just overwrite the data once with random values and result in almost absolutely irreversible deletion.[/quote]No. If I write '4' on a sheet of paper, then you erase it and write '7' in it's place, I can still see the imprint of the 4 on the paper.
You've got to [i]repeat this process many times[/i] until there's too many imprints there to make out any specific one....


And then you've got to burn it to be sure.
0

Share this post


Link to post
Share on other sites
When I have to be certain something is erased and not recoverable... I use this tool:

[img]http://static.howstuffworks.com/gif/power-drill-1.jpg[/img]
2

Share this post


Link to post
Share on other sites
You can totally erase something, but it's a bigger pain in the ass than it's worth. It pretty much boils down to writing random data a couple times, writing very specific data to the data you want erased a bunch of times, then writing some more random data again. The random data is moreso just to add some more jitter and randomness to it more than anything.

Not positive how it works, but I'd presume it estimates what the original file would look like after being overwritten a bunch of times, then produces a bunch of data that is expected to have the inverse of what the original file would have looked like resulting in neither set of data appearing to have ever existed. THIS EXPLANATION IS PURELY CONJECTURE ON MY PART, but the first paragraph stuff I did skim a paper on, and that appeared to be what they set up.

edit: [quote][color=#1C2837][size=2]When I have to be certain something is erased and not recoverable... I use this tool:[/quote][/size][/color]
[color=#1C2837][size=2]I use alcohol 8-)[/size][/color]
0

Share this post


Link to post
Share on other sites
Pretty simple system. If you have a blank HD and you write a file and want to remove that file write the XOR of it over the original data. It's even wear on the HD. Reading doesn't change the value for a magnetic drive so you've essentially just set all bits evenly.

Speaking of destroying HDs I put one on an industrial magnet before. Once you flick that switch anything on those platters are gone. (High frequency AC ftw?)
0

Share this post


Link to post
Share on other sites
[quote name='Sirisian' timestamp='1316750457' post='4865010']
Pretty simple system. If you have a blank HD and you write a file and want to remove that file write the XOR of it over the original data. It's even wear on the HD. Reading doesn't change the value for a magnetic drive so you've essentially just set all bits evenly.

Speaking of destroying HDs I put one on an industrial magnet before. Once you flick that switch anything on those platters are gone. (High frequency AC ftw?)
[/quote]

Simply using XOR with same data is probably not a good idea.

I am assuming you are thinking that the net result will consist of all zeroes, hence my two points to ignore below.

Let's ignore that we have...:

1. Rotating platter (can't predict exactly where on one sector data will be written)
2. Encoding schema (data represented in memory != data represented on disk)

This allows us to assume the unlikely event that you will manage to write each bit of information on the exact same spot as the original data and that the data is formatted the same in memory as on disk. Even being one microsecond off would leave revealing peaks that would make recovery easy.

Visualized:
EDIT: my visualization was faulty so I removed it, and I have a strange feeling this is just some hypothetical rambling, but I am tired from being up all night. See my last line that really sums this up. ;)

Reality is different. You can't predict where within a sector data will be written. In fact all data inside one sector has to be rewritten even though you just want to update a part of it.

How can you manage to hit exactly the one bit of information you want to change on a rotating platter? Answer: You read the whole sector with related and unrelated data, change the bit of information you want to change and write the whole sector again. (With sync mark to identify the start of the sector and a gap so you don't overwrite the start of the sector you just wrote.)

Encoding schemas may also complicate things further. By XOR-ing the original data you WON'T get a XOR-ed result written to the disk. (Have a look at MFM encoding for instance, although it is not relevant today, I remember the good old Amiga and trackmo's.)

I would suppose just writing a few passes of pseudorandom data would suffice for most users here, and for those the rest there are probably already existing policies on how to handle sensitive data.

NOT TO mention that reading the data from the disk, then XOR-ing it with the same data, and then write it back would in effect be just writing all zeroes in the first place...
0

Share this post


Link to post
Share on other sites
I was living in a perfect world where a bit is only written when the bit was not already set to show that such a system would be possible. Just not on a conventional drive. :mellow:
0

Share this post


Link to post
Share on other sites
The problem has to do with what Telastyn said, that the hard drives are actually analogue. The area that is considered a "bit" is composed of tiny granules. If you were to go through and write a 0 over every bit on the hard drive only once, some of the granules in the bit area might not get converted, and a computer forensics specialist has equipment that looks for this. As was mentioned before, writing 0's to the disk multiple times helps to make sure that the granules have all been converted.
0

Share this post


Link to post
Share on other sites
There are algorithmic erasure programs which can mostly kill data from casual inspection, and even some more sophisticated approaches. Basically they write 0s, then 1s, then alternating patterns of 1s and 0s, a few more patterns, a few passes of random data, then back through each stage in reverse, ending with 0s. This is probably good enough for the average user -- its still possible for sophisticated forensic analysis to detect the data though -- they will sometimes go as far as removing the platters and reading them in a very precise mechanism that can read the 'edges' of the bits which may still contain old data, due to greater mechanical tolerances causing the head to drift minutely over time.

Basically, short of physical destruction by shattering the platter, you can't absolutely guarantee data destruction, you just have to do a good enough job that its more trouble than its worth, as sophisticated analysis becomes expensive quite quickly.

SSD drives throw another wrench in the works, because wear-leveling algorithms mean that the same part of the logical file is almost never written to the same physical flash cell -- meaning that, even if you over-write it a thousand times, its still very likely that the original data is sitting in some flash cell untouched. By reading directly from the flash chips themselves, researchers have been able to easily recover data from SSDs. The drive interface/firmware of these drives do not expose any way to address a specific flash cell, so you simply can't overwrite data at will.

For data integrity, encrypted volumes are probably a better bet, and if you need to work with the data as decrypted files, create and decrypt them to a RAM disk to make sure they never tough persistent storage. While you're at it, make sure the pagefile and hibernation files are disabled too... Those files are another back door to reading your data, if something sensitive happened to be in ram when when it was paged out, or when you put your computer into deep sleep.
0

Share this post


Link to post
Share on other sites
Something being "digital" is just a concept, or should I say, an interpretation of an analogue signal with specific constraints. Therefore, digital circuits are inherently analogue, except they operate within a minimum tolerance that allows us to classify the signal in discrete states. This is one of the reasons why a digital signal is so immune to noise, one can clearly reconstruct the bit states from a degraded signal. You'd be surprised how malformed some of the supposedly square waveforms are in a high speed bus on your motherboard, or even in RAM, which is nothing but an array of capacitors that butcher the rise and fall time of your bit states.

Anyway, with magnetic media, once you magnetise the surface, it's difficult to restore that surface in its original state. Magnetic domains under the platter surface could be still aligned well enough to extract meaningful data. Your best bet is to overwrite the same area with alternating bit states, then random data, then more alternating bit states... etc, until the original data has eroded enough to fall under the noise floor of the scanning probe analysis equipment. This procedure is a bit like degaussing the magnetised surface and then swamping it with noise.
0

Share this post


Link to post
Share on other sites
I think it's important to know the difference between a theoretical and a practical application.

There are very few people in this world that have to actually worry about someone renting an electron microscope to retrieve erased data. I would guess that no one on this website is doing something worthy of that kind of expense. For practical purposes, just filling the drive with random garbage several times would be enough that no one would be willing to go through the trouble of retrieving it. Especially when you could do cheaper things like just bugging your keyboard, since the data they're after is probably something you're still trying to use.

However, theoretically, your data is there and will always be there, short of physical destruction. It's an analog world, and everything you touch you leave a part of you behind. It's just a matter of finding the right tool to find it.
0

Share this post


Link to post
Share on other sites
[quote name='SeraphLance' timestamp='1316741675' post='4864965']
There's also the [url="http://en.wikipedia.org/wiki/Gutmann_method"]Gutmann Method[/url], though I know little about it's necessity.
[/quote]

Which is nice if you happen to stumble upon a deLorean and travel back to 70s and need to recover 8 inch flopies or an odd HDD that happened to use MFM.

Drives today tend to use PRML or EPRML, which is several generations and ages beyond that. It's not even reliable, PR is partial response, so even at best the signal is unreliable.

Regarding electron microsopes - MFM peak disk capacity was 10MB on 5 inch drive and several platters. Tracks on those could be viewed with almost regular microscope. Today, such drive would house several TB. MFM also suffered from mechanical issues, when a track was rewritten the mechanics were not accurate enough to it slightly deviated and left old track partially recoverable.

[quote] an electron microscope[/quote]Electron microscopes don't really work that way. They need the sample to be treated and even then, it has never been published, not even for MFMs which were shown to be recoverable.

One could point out that drives use robust recovery methods due to that (Reed Solomon codes) which one could use to reconstruct the data. But, if data X with CRC 7 is rewritten, the result will be Y with CRC 3. So both pieces change and the CRC would only correct for Y.


I have not heard or seen any published document that would demonstrate recovery of deleted data on post MFM drives. A theory appears here and there, but not a single proof, not even for a handful of bits. All recovery companies only deal with recovering data from physical damage. Another detail is that what you write is almost completely irrelevant. Whether random, XOR or zeros, it's all the same, it destroys old data. The pattern written on media is also not 1:1, so writing 0 will not result on 8 values of 0 on platter.

It's simply an urban myth. It sounds romantic that there is some deus ex machina possibility, but it has never been done.


One thing I have experienced and will vouch for: dd. As demonstrated by a sysadmin who ran it on my production machine and switched the in/out parameters while doing backup. It's super effective.
0

Share this post


Link to post
Share on other sites
[quote name='Antheus' timestamp='1316785312' post='4865143']
It's simply an urban myth. It sounds romantic that there is some deus ex machina possibility, but it has never been done.
[/quote]
That.

It may be possible for very well funded governmental institutions or multinational corporations, but these would probably have much easier and much more cost effective ways to gain the relevant information. The weakest link in the information chain is always a human.

[quote]
an electron microscope
[/quote]
An electron microscope can't be used for that. You'd need a magnetic force microscope (ironically also called MFM), which is even more expensive.
0

Share this post


Link to post
Share on other sites
How do you know the history of bits though?

---- = blank space

----[FILE 1]----------
(delete file 1)
-------[FILE 2]-------

0000111222222000000 (how much history this memory has)


So they look at see no history for a few, then they see a history of 1 for the very first part of FILE 1, then they hit some bits that have a history of 2. And they just go back in time over bits that have different amounts of history? How much history is there anyway? Say I did that diagram 32 times deleting and adding file1 and file 2 in those same spaces with images of the same exact size. So if you dont know the history since the life of the HD, you cant know which history to grab, because you have no boundaries to know where a specific file had history. Right?
0

Share this post


Link to post
Share on other sites
[quote name='Yann L' timestamp='1316814293' post='4865311']
It may be possible for very well funded governmental institutions or multinational corporations[/quote]
Money can't buy what isn't there.

In the 90s the "Gibson" was a romantic take on what big companies and money had. But the reality was much more mundane. Sure, the Cray was stylish, the kid who brought O2 to the party was the chick magnet, but at the end of the day, it was just POSIX, same as 30 years ago and today, running ls and chmod.

As mentioned, I don't know of a single published case where an erased drive, even in perfect condition, were recovered, excluding the possibility of MFM (10MB or less drives).


Unfortunately, education should show the latest greatest, not old-wives lore of yesterday. Some 8 years ago I read a publication on scientists who managed to decode wire signal from LED on back of network card using a telescope (peek from across the street, read network traffic). Limited in application due to responses and network rates going higher, but still, a valid experiment. Recently, researchers managed to correlate smart meter readings with TV channel selection based on dark/bright image which cause power consumption fluctuations in TVs. Again of limited use, but an old and proven concept. A slightly different technique but also much more reliable was developed a decade or two ago and in some cases fell under privacy regulation, since meter could deduce individual appliance usage based on current phase changes completely accurately.

Yet there is not a single mention of erased HDD information ever successfully read, not even a single bit.

Ergo, it has never been done.

And as always, if something needs going away - industrial shredder. Cheap, reliable, fast. Will also dispose of witnesses.

The ultimate irony is discussing this in the age of Facebook, Google, Twitter, ad agencies and ISPs tracking every action you take (methods mentioned above can and are used by those to determine what you're doing).

[quote]how much history this memory has[/quote]
It would depend on method. Since you know that head follows a circular path you could extrapolate to which track the sample belongs. You would then use CRC present in same track to correct for missing bits or other errors. But this is about the same as asking what kind of batteries light saber uses. One cannot ponder the details of something that isn't.
0

Share this post


Link to post
Share on other sites
Use the DoD standard for wiping hard drives just for that the bits that contain the file. Then no one would be able to recover the actual file.
0

Share this post


Link to post
Share on other sites
[quote name='Antheus' timestamp='1316815352' post='4865317']
Money can't buy what isn't there.
[/quote]
It is extremely unlikely that there is [i]nothing[/i] there. In fact, given the analog nature of the storage medium, it is almost guaranteed that residual magnetization is left to a certain degree. Reconstructing the data is extremely difficult and/or uneconomical with current publicly available technology. This does not mean that it is [i]impossible[/i] given enough resources and/or time. A wiped disk may be recoverable 20 years from now, but the information it contains can still be highly relevant.

DoD mandates physical destruction or degaussing before disposal of harddrives with sensitive information. Overwriting is specifically not considered sufficient.

[quote name='Antheus' timestamp='1316815352' post='4865317']
As mentioned, I don't know of a single published case where an erased drive, even in perfect condition, were recovered, excluding the possibility of MFM (10MB or less drives).
[/quote]
Such cases would most likely be classified or involve illegal activities (such as high profile industrial espionage). Wait 50 years and some may fall under the FOIA.
0

Share this post


Link to post
Share on other sites
One thing I want to point out is that "physical destruction," short of burying the disk in a volcano, isn't secure at all: while there are no documented cases of data being recovered after a disk is wiped, there [i]are[/i] cases of data being recovered from a disk that's shredded: you can just scan the pieces with an electron microscope, the reconstruct them with a computer (the data tracks follow a predictable curve so it's not at all hard to figure out where even a small piece belongs).

Also, the "just because there haven't been any published cases of data being recovered doesn't mean it's not happening! The government just keeps it secret" argument isn't compelling to me. To me it doesn't sound any different than the notion that the government uses people with psychic powers to solve crimes, but they don't acknowledge it because then people will be likely to hurt the psychics: In both cases, it seems like the government would have more to gain by publishing their ability to gain knowledge (e.g. as a deterrent; you'll stop a lot more people from committing computer crimes by making it clear that you have the technology to catch them than you will by actually trying to hunt them down after the fact) than by keeping it a secret.
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0