At least that is what my NOS-110 teacher said. This automatically got me thinking about ways to permanently delete a file without possible recovery. It seems simple enough to me. Write over the bytes of the file with random values and use a seed value from a unrecoverable source. For example the computer clock, CPU temperature or internet connection speed. Make sure not to write over bytes which have to do with the file system since they are in known patterns and could be used to uncover the seed value by checking every single possible value for the seed using the algorithm which was used to generate the random number. This should naturally be avoidable by using a standard file open command to write the data which I believe should avoid writing to any known structures in the file system. Unless however the file type is know and includes a standard header. However one could also make the permanent deletion algorithm avoid file type headers.
So anyway I told my teacher that I was pretty sure it was possible to permanently delete a file using an unknowable seed value to overwrite the file with random values. This made the teacher go into a long rant about how I think I know everything and telling me all of his qualifications. Then he went on to explain that no values on a computer are random and once you know the algorithm you can hack any encryption. I tried to explain to him that you may know the algorithm but an unknowable seed value is easily obtained. I'm almost certain I'm right but he humiliated me in front of the whole class... Anyway, what do you guys think? Am I right? If not then why?
1
Files can always be recovered no matter what you do...
Started by SteveDeFacto, Sep 22 2011 07:25 PM
34 replies to this topic
Sponsor:
#3 Members - Reputation: 437
Posted 22 September 2011 - 07:34 PM
I'm fairly sure you can just write to your disk until you run out of memory to make erased data unrecoverable in most cases...
There's also the Gutmann Method, though I know little about it's necessity.
There's also the Gutmann Method, though I know little about it's necessity.
#4 Moderators - Reputation: 14300
Posted 22 September 2011 - 07:42 PM
Even if you write over a file with (unknowable, truly random) random bits, forensic analysis can still detect the original values of the bits in some cases. Think of it like, how when you erase a page of pencil-writings, and then scribble over it, there will still be a faint impression of the original text on the paper. Same thing happens with magnetic disks, so you need to scribble really hard to make sure no-one can decipher those imprints.
The standard practice that I know of for permanently deleting a file is to write over it will all zeros, then write over it with all ones, then write over it a few times with random bits, and then to smash the hard-drive enclosure with a hammer and discard of it in protected land-fill.
As for truly random numbers -- electronic slot machine regulations in some jurisdictions actually require truly random numbers, not just pseudo-random numbers. These are generated from a real RNG device (which measure entropy from radiation, decay, thermal noise, etc), not a PRNG algorithm, and it is truly impossible to predict the output numbers, even if you know how it works.
The standard practice that I know of for permanently deleting a file is to write over it will all zeros, then write over it with all ones, then write over it a few times with random bits, and then to smash the hard-drive enclosure with a hammer and discard of it in protected land-fill.
As for truly random numbers -- electronic slot machine regulations in some jurisdictions actually require truly random numbers, not just pseudo-random numbers. These are generated from a real RNG device (which measure entropy from radiation, decay, thermal noise, etc), not a PRNG algorithm, and it is truly impossible to predict the output numbers, even if you know how it works.
#5 Members - Reputation: 1414
Posted 22 September 2011 - 07:43 PM
Actually, you can just erase it so well that it never existed in the first place.
-~-The Cow of Darkness-~-
#6 Members - Reputation: 3371
Posted 22 September 2011 - 07:50 PM
The issue isn't with overwriting with randomness, it's that hard-drives are analog beasts in a digital world. Even when you write '1' to the bit, it's not completely totally a one. It's just more than .5 so the hard-drive says it's a one. A dedicated attacker can crack open the spindles and take an electron microscope to it to get a better picture of what the bits used to be.
In software, the best you can do is write to the drive random bits 13-25ish times. Otherwise, wiping the drive with industrial magnets or causing the platters physical/chemical damage is your best chance of obscuring the data on the drive. Blossum's links are good.
So you're both kind of right; 'just' deleting a file won't do it. Even overwriting garbage won't do it, but you can eventually get to a state where the information is sufficiently obscured to be considered 'lost'.
In software, the best you can do is write to the drive random bits 13-25ish times. Otherwise, wiping the drive with industrial magnets or causing the platters physical/chemical damage is your best chance of obscuring the data on the drive. Blossum's links are good.
So you're both kind of right; 'just' deleting a file won't do it. Even overwriting garbage won't do it, but you can eventually get to a state where the information is sufficiently obscured to be considered 'lost'.
#7 Banned - Reputation: 103
Posted 22 September 2011 - 08:02 PM
Even if you write over a file with (unknowing) random bits, forensic analysis can still detect the original values of the bits in some cases.
The standard practice that I know of for permanently deleting a file is to write over it will all zeros, then write over it with all ones, then write over it a few times with random bits, and then to smash the hard-drive enclosure with a hammer and discard of it in protected land-fill.
As for truly random numbers -- electronic slot machine regulations in some jurisdictions actually require truly random numbers, not just pseudo-random numbers. These are generated from a real RNG device (which measures entropy from radiation, decay, thermal noise, etc), not a PRNG algorithm.
Even that would be less effective than what I proposed since you are effectively reducing the magnetic field, increasing the field, and lastly using a PRNG algorithm to scramble it. To recover the data one would need hyper sensitive equipment to read off the data then reduce the values down to get the background noise. Then you simply read the bits that make up the header of an NTFS or FAT file system and run them through an algorithm to reverse PRNG randomization with every single possible seed value. You then amplify the background noise yet again and check to see if the data matches the standard header for that file system.
But if you avoid knowable headers you can just overwrite the data once with random values and result in almost absolutely irreversible deletion.
#8 Moderators - Reputation: 14300
Posted 22 September 2011 - 08:08 PM
No. If I write '4' on a sheet of paper, then you erase it and write '7' in it's place, I can still see the imprint of the 4 on the paper.you can just overwrite the data once with random values and result in almost absolutely irreversible deletion.
You've got to repeat this process many times until there's too many imprints there to make out any specific one....
And then you've got to burn it to be sure.
#10 Members - Reputation: 658
Posted 22 September 2011 - 09:29 PM
You can totally erase something, but it's a bigger pain in the ass than it's worth. It pretty much boils down to writing random data a couple times, writing very specific data to the data you want erased a bunch of times, then writing some more random data again. The random data is moreso just to add some more jitter and randomness to it more than anything.
Not positive how it works, but I'd presume it estimates what the original file would look like after being overwritten a bunch of times, then produces a bunch of data that is expected to have the inverse of what the original file would have looked like resulting in neither set of data appearing to have ever existed. THIS EXPLANATION IS PURELY CONJECTURE ON MY PART, but the first paragraph stuff I did skim a paper on, and that appeared to be what they set up.
edit:
I use alcohol 8-)
Not positive how it works, but I'd presume it estimates what the original file would look like after being overwritten a bunch of times, then produces a bunch of data that is expected to have the inverse of what the original file would have looked like resulting in neither set of data appearing to have ever existed. THIS EXPLANATION IS PURELY CONJECTURE ON MY PART, but the first paragraph stuff I did skim a paper on, and that appeared to be what they set up.
edit:
When I have to be certain something is erased and not recoverable... I use this tool:
I use alcohol 8-)
#11 Members - Reputation: 1286
Posted 22 September 2011 - 10:00 PM
Pretty simple system. If you have a blank HD and you write a file and want to remove that file write the XOR of it over the original data. It's even wear on the HD. Reading doesn't change the value for a magnetic drive so you've essentially just set all bits evenly.
Speaking of destroying HDs I put one on an industrial magnet before. Once you flick that switch anything on those platters are gone. (High frequency AC ftw?)
Speaking of destroying HDs I put one on an industrial magnet before. Once you flick that switch anything on those platters are gone. (High frequency AC ftw?)
#12 Members - Reputation: 134
Posted 22 September 2011 - 10:55 PM
Pretty simple system. If you have a blank HD and you write a file and want to remove that file write the XOR of it over the original data. It's even wear on the HD. Reading doesn't change the value for a magnetic drive so you've essentially just set all bits evenly.
Speaking of destroying HDs I put one on an industrial magnet before. Once you flick that switch anything on those platters are gone. (High frequency AC ftw?)
Simply using XOR with same data is probably not a good idea.
I am assuming you are thinking that the net result will consist of all zeroes, hence my two points to ignore below.
Let's ignore that we have...:
1. Rotating platter (can't predict exactly where on one sector data will be written)
2. Encoding schema (data represented in memory != data represented on disk)
This allows us to assume the unlikely event that you will manage to write each bit of information on the exact same spot as the original data and that the data is formatted the same in memory as on disk. Even being one microsecond off would leave revealing peaks that would make recovery easy.
Visualized:
EDIT: my visualization was faulty so I removed it, and I have a strange feeling this is just some hypothetical rambling, but I am tired from being up all night. See my last line that really sums this up. ;)
Reality is different. You can't predict where within a sector data will be written. In fact all data inside one sector has to be rewritten even though you just want to update a part of it.
How can you manage to hit exactly the one bit of information you want to change on a rotating platter? Answer: You read the whole sector with related and unrelated data, change the bit of information you want to change and write the whole sector again. (With sync mark to identify the start of the sector and a gap so you don't overwrite the start of the sector you just wrote.)
Encoding schemas may also complicate things further. By XOR-ing the original data you WON'T get a XOR-ed result written to the disk. (Have a look at MFM encoding for instance, although it is not relevant today, I remember the good old Amiga and trackmo's.)
I would suppose just writing a few passes of pseudorandom data would suffice for most users here, and for those the rest there are probably already existing policies on how to handle sensitive data.
NOT TO mention that reading the data from the disk, then XOR-ing it with the same data, and then write it back would in effect be just writing all zeroes in the first place...
#14 Members - Reputation: 100
Posted 23 September 2011 - 01:27 AM
The problem has to do with what Telastyn said, that the hard drives are actually analogue. The area that is considered a "bit" is composed of tiny granules. If you were to go through and write a 0 over every bit on the hard drive only once, some of the granules in the bit area might not get converted, and a computer forensics specialist has equipment that looks for this. As was mentioned before, writing 0's to the disk multiple times helps to make sure that the granules have all been converted.
#15 Members - Reputation: 2912
Posted 23 September 2011 - 02:03 AM
There are algorithmic erasure programs which can mostly kill data from casual inspection, and even some more sophisticated approaches. Basically they write 0s, then 1s, then alternating patterns of 1s and 0s, a few more patterns, a few passes of random data, then back through each stage in reverse, ending with 0s. This is probably good enough for the average user -- its still possible for sophisticated forensic analysis to detect the data though -- they will sometimes go as far as removing the platters and reading them in a very precise mechanism that can read the 'edges' of the bits which may still contain old data, due to greater mechanical tolerances causing the head to drift minutely over time.
Basically, short of physical destruction by shattering the platter, you can't absolutely guarantee data destruction, you just have to do a good enough job that its more trouble than its worth, as sophisticated analysis becomes expensive quite quickly.
SSD drives throw another wrench in the works, because wear-leveling algorithms mean that the same part of the logical file is almost never written to the same physical flash cell -- meaning that, even if you over-write it a thousand times, its still very likely that the original data is sitting in some flash cell untouched. By reading directly from the flash chips themselves, researchers have been able to easily recover data from SSDs. The drive interface/firmware of these drives do not expose any way to address a specific flash cell, so you simply can't overwrite data at will.
For data integrity, encrypted volumes are probably a better bet, and if you need to work with the data as decrypted files, create and decrypt them to a RAM disk to make sure they never tough persistent storage. While you're at it, make sure the pagefile and hibernation files are disabled too... Those files are another back door to reading your data, if something sensitive happened to be in ram when when it was paged out, or when you put your computer into deep sleep.
Basically, short of physical destruction by shattering the platter, you can't absolutely guarantee data destruction, you just have to do a good enough job that its more trouble than its worth, as sophisticated analysis becomes expensive quite quickly.
SSD drives throw another wrench in the works, because wear-leveling algorithms mean that the same part of the logical file is almost never written to the same physical flash cell -- meaning that, even if you over-write it a thousand times, its still very likely that the original data is sitting in some flash cell untouched. By reading directly from the flash chips themselves, researchers have been able to easily recover data from SSDs. The drive interface/firmware of these drives do not expose any way to address a specific flash cell, so you simply can't overwrite data at will.
For data integrity, encrypted volumes are probably a better bet, and if you need to work with the data as decrypted files, create and decrypt them to a RAM disk to make sure they never tough persistent storage. While you're at it, make sure the pagefile and hibernation files are disabled too... Those files are another back door to reading your data, if something sensitive happened to be in ram when when it was paged out, or when you put your computer into deep sleep.
#16 Members - Reputation: 544
Posted 23 September 2011 - 03:20 AM
Something being "digital" is just a concept, or should I say, an interpretation of an analogue signal with specific constraints. Therefore, digital circuits are inherently analogue, except they operate within a minimum tolerance that allows us to classify the signal in discrete states. This is one of the reasons why a digital signal is so immune to noise, one can clearly reconstruct the bit states from a degraded signal. You'd be surprised how malformed some of the supposedly square waveforms are in a high speed bus on your motherboard, or even in RAM, which is nothing but an array of capacitors that butcher the rise and fall time of your bit states.
Anyway, with magnetic media, once you magnetise the surface, it's difficult to restore that surface in its original state. Magnetic domains under the platter surface could be still aligned well enough to extract meaningful data. Your best bet is to overwrite the same area with alternating bit states, then random data, then more alternating bit states... etc, until the original data has eroded enough to fall under the noise floor of the scanning probe analysis equipment. This procedure is a bit like degaussing the magnetised surface and then swamping it with noise.
Anyway, with magnetic media, once you magnetise the surface, it's difficult to restore that surface in its original state. Magnetic domains under the platter surface could be still aligned well enough to extract meaningful data. Your best bet is to overwrite the same area with alternating bit states, then random data, then more alternating bit states... etc, until the original data has eroded enough to fall under the noise floor of the scanning probe analysis equipment. This procedure is a bit like degaussing the magnetised surface and then swamping it with noise.
Latest project: Sideways Racing on the iPad
#17 Members - Reputation: 538
Posted 23 September 2011 - 04:47 AM
I think it's important to know the difference between a theoretical and a practical application.
There are very few people in this world that have to actually worry about someone renting an electron microscope to retrieve erased data. I would guess that no one on this website is doing something worthy of that kind of expense. For practical purposes, just filling the drive with random garbage several times would be enough that no one would be willing to go through the trouble of retrieving it. Especially when you could do cheaper things like just bugging your keyboard, since the data they're after is probably something you're still trying to use.
However, theoretically, your data is there and will always be there, short of physical destruction. It's an analog world, and everything you touch you leave a part of you behind. It's just a matter of finding the right tool to find it.
There are very few people in this world that have to actually worry about someone renting an electron microscope to retrieve erased data. I would guess that no one on this website is doing something worthy of that kind of expense. For practical purposes, just filling the drive with random garbage several times would be enough that no one would be willing to go through the trouble of retrieving it. Especially when you could do cheaper things like just bugging your keyboard, since the data they're after is probably something you're still trying to use.
However, theoretically, your data is there and will always be there, short of physical destruction. It's an analog world, and everything you touch you leave a part of you behind. It's just a matter of finding the right tool to find it.
#18 Members - Reputation: 2369
Posted 23 September 2011 - 07:41 AM
There's also the Gutmann Method, though I know little about it's necessity.
Which is nice if you happen to stumble upon a deLorean and travel back to 70s and need to recover 8 inch flopies or an odd HDD that happened to use MFM.
Drives today tend to use PRML or EPRML, which is several generations and ages beyond that. It's not even reliable, PR is partial response, so even at best the signal is unreliable.
Regarding electron microsopes - MFM peak disk capacity was 10MB on 5 inch drive and several platters. Tracks on those could be viewed with almost regular microscope. Today, such drive would house several TB. MFM also suffered from mechanical issues, when a track was rewritten the mechanics were not accurate enough to it slightly deviated and left old track partially recoverable.
Electron microscopes don't really work that way. They need the sample to be treated and even then, it has never been published, not even for MFMs which were shown to be recoverable.an electron microscope
One could point out that drives use robust recovery methods due to that (Reed Solomon codes) which one could use to reconstruct the data. But, if data X with CRC 7 is rewritten, the result will be Y with CRC 3. So both pieces change and the CRC would only correct for Y.
I have not heard or seen any published document that would demonstrate recovery of deleted data on post MFM drives. A theory appears here and there, but not a single proof, not even for a handful of bits. All recovery companies only deal with recovering data from physical damage. Another detail is that what you write is almost completely irrelevant. Whether random, XOR or zeros, it's all the same, it destroys old data. The pattern written on media is also not 1:1, so writing 0 will not result on 8 values of 0 on platter.
It's simply an urban myth. It sounds romantic that there is some deus ex machina possibility, but it has never been done.
One thing I have experienced and will vouch for: dd. As demonstrated by a sysadmin who ran it on my production machine and switched the in/out parameters while doing backup. It's super effective.
#20 Moderators - Reputation: 1754
Posted 23 September 2011 - 03:44 PM
That.It's simply an urban myth. It sounds romantic that there is some deus ex machina possibility, but it has never been done.
It may be possible for very well funded governmental institutions or multinational corporations, but these would probably have much easier and much more cost effective ways to gain the relevant information. The weakest link in the information chain is always a human.
An electron microscope can't be used for that. You'd need a magnetic force microscope (ironically also called MFM), which is even more expensive.an electron microscope
