Securely overwrite files and free space

[wraithdu]

Update: 11/14/2012

- Fixed bug in Freespace erase - it was using sector size by mistake (did not cause failure, but slowed the wipe)

- Replaced dir move/remove functions with long file name compatible API functions (thanks KaFu)

- Added long path and UNC support to the _FileMapping UDF (this change affected portions of the UDF only used to wipe sparse / encrypted / compressed files on NTFS)

- Wiping of these special files is now restricted to local volumes - you cannot get the volume handle required from remote volumes

- NOTE: When running under UAC full admin permissions are required to use the defrag API to wipe these special files

- Various updates to included _FileEx UDF

Update: 10/1/12

- _SecureDirectoryDelete is now an iterative function, instead of recursive.

- Added ability to overwrite with random data. To activate this feature, you must use the user array format with the array element corresponding to the random pass set to -1. Note this *will* incur a performance penalty since the buffer is filled with new random data each time it is used.

- Registered _SD_FreeBuffers() as an on-exit function.

- Changed the way buffer recreation is handled - new buffers are automatically created each time you pass a different value or array for $aPatterns.

- Some updates to the included _FileEx UDF.

Update: 9/11/12

- _SecureDirectoryDelete returns an array of errors for all errors encountered during recursion. This includes the error number, path at which the error was encountered, and the type of path item (file or folder).

Update: 9/10/12

- More optimizations, read new header for important info

• Optimized buffer creation - done once on first call to any function, survives for life of script.
• $aPatterns parameter now accepts a single integer 0 <= i <= 255 to indicate a single pass using that byte pattern (-1 for default 3-pass DoD_E)
• Changed final file / dir removal to restore original file / dir name in case of removal failure.

- SCRIPT BREAKING - Changed the order of the $fDelete and $aPatterns parameters, moving the more commonly used $aPatterns parameter to a higher priority.

Update: 9/7/12

- Code cleanup and some optimizations

- Fixed a bug that could slow down wiping of compressed / encrypted / sparse files if using more than one pass (this bug did not affect the actual wiping of these files)

- Added callback functions to monitor the progress of the wipe

• File: Callback receives bytes written and total bytes.
• Dir: Callback is passed to file delete and receives bytes written and total bytes per file. Total directory size and bytes is not calculated, as the penalty would be too high for these basic functions. If you need total directory size, do this calculation yourself before calling this function, and keep track of the data written.
• Freespace: Callback is passed bytes written and total free bytes. Under certain circumstances, total bytes written may not equal total free bytes on the last call.

Update: 9/5/12

- Major rewrite

• Rewrote delete function, includes a special method to securely erase NTFS compressed, sparse, and EFS encrypted files.
• Simplified the parameters. Only need to specify if the file should be deleted after being overwritten, and can specify custom delete patterns.
• Added a directory deletion function to recursively wipe directories
• Added a freespace wipe function, based on the Sysinternals SDelete utility
• Reparse points: symlinks and junctions are now handled, they are simply deleted, NOT followed or overwritten.

- The archive includes my _FileMapping UDF and _FileEx UDF for support functions.

I tested this A LOT, which also accounts for the delay in releasing. I've tested on NTFS and FAT32 drives, and it functions as it should. That said, I take no responsibility for data loss.

If there are testers familiar with data recovery, I'd love tests done on the thoroughness of the wiping, especially the free space wipe. NTFS has some real gotchas that normal apps can't deal with safely, so some stuff may slip through. If so, I'd like to know about it.

I plan on working on some sort of cluster tip / slack space wipe to add to the freespace function, but I have to figure out how best to handle that.

Update: 8/20/10

- Uses unbuffered I/O to overwrite the file

- Added fFileTime and fDelete parameters

- Code cleanup

EDIT: New version. The buffer creation has been optimized. This one optionally renames the file 10 times to a random name, resets the file timestamps to Jan. 1, 1980, 12:01am, and optionally takes a user array of integer values from 0 to 255 (decimal or hex) to use as the erasing patterns. This way the user can control the patterns used to erase the data and the number of passes. If no array is passed, then the default DoD short method (DoD_E) is used. The last parameter lets the user specify the block size to be used, default is 32768 bytes (32 Kb).

Just something I felt like whipping up. This script does a secure erase of any file. It uses the DoD short method as described in the script. Currently it writes in 32Kb blocks which seems to give pretty good performance. Depending on the file size, it may increase the erased file's size by up to 32 Kb. You can play with the size if you feel like it. On my computer, it erased a 5.5 MB file (16.5 MB in total written data) in 0.08 seconds. I verified the results with a Hex editor to be sure. Then it will rename the file 10 times using a randomly generated name. This *should* make the file names unrecoverable as well as the data.

Enjoy!

Example

$r = _SecureFileDelete(@ScriptDir & "test.bin", -1, False) ; use default patterns, do not delete
ConsoleWrite($r & " : " & @error & @CRLF)

Local $array[3] = [0x00,0xFF,0x1C]
$r = _SecureFileDelete(@ScriptDir & "test.bin", $array, False) ; use custom patterns, no delete
ConsoleWrite($r & " : " & @error & @CRLF)

Dim $array[2] = [0,-1]
$r = _SecureFileDelete(@ScriptDir & "test.bin", $array, False) ; two passes, last pass random data, no delete
ConsoleWrite($r & " : " & @error & @CRLF)

SecureDelete.zip

Edited November 14, 2012 by wraithdu

[WeMartiansAreFriendly]

Nice to see this, You should add more methods

Would randomize the creation/modified/accessed dates add any security benefits?

[trancexx]

It's good to know how it's done.

Nice

[wraithdu]

I suppose it could. Probably just easier to set it to something static like 19800101000001.

Re: methods...there's really no advantage to any method over another, and I think most experts agree that on modern drives one random pass is good enough. 3 to 7 passes is probably overkill, while Gutmann is just a waste of HDD cycles.

I have some thoughts about user options. Let me see what I like and I'll post an update.

[wraithdu]

See first post Edited October 20, 2008 by wraithdu

[Creator]

I've used DoD a thousand times before. Its safe to say that anything destroyed by that method is gone for good.

So cool to see the internals of that beast! Thanks for that!

[wraithdu]

Updated the first post. I tried to optimize the buffer creation. This was the part that was taking time, not the actual data writing. Ironically, converting the patterns to strings then filling the buffer all in one shot was way faster than keeping all the data binary and filling the buffer by byte index -

For $i = 0 to 2
    For $j = 1 to 32768
        DllStructSetData($buff, 1, $aPatterns[$i], $j)
    Next
Next

Edited October 20, 2008 by wraithdu

[Creator]

ah..beta

Edited October 24, 2008 by Creator

[wraithdu]

Wha, beta??

[wraithdu]

Update, see first post.

[cobra11]

...

Edited December 6, 2017 by cobra11

[KaFu]

"Studies have shown that most of today’s media can be effectively cleared by one overwrite..."

http://en.wikipedia.org/wiki/Data_remanence

[wraithdu]

Updated, see OP.

[KaFu]

Definitely looks great, 5 stars from me ... will try to implement it into the next release of SMF ...

[wraithdu]

One more update, hopefully the last for a while unless bugs are found.

[wraithdu]

Updated. See OP and read UDF header for important info. WARNING - this version has a script breaking change!!

[guinness]

This UDF has come on leaps and bounds. Thanks wraithdu.

[wraithdu]

Thanks. I usually dislike frequent updates, and especially script breaking ones, so I'm breaking my own rules. But I've been working on a script, the basis of which is this UDF, so I keep tweaking and refining based on the greater body of work. A lot of the optimization has grown out of the recursive directory function.... which is another broken rule. I usually don't write recursive functions for my UDFs, as I believe it should be left up to implementation. But this one has enough nuance that I thought it was worth including.

[elektrongr]

Hello,

i just tried your script but cannot make it to work.

Here is the test .au , i used on a file test.bin inside the script directory.

<code>

#include <_SecureDelete.au3>

$r = _SecureFileDelete(@ScriptDir & "test.bin", True) ; use default patterns, do not delete

ConsoleWrite($r & " : " & @error & @CRLF)

Local $array[3] = [0x00,0xFF,0x1C]

$r = _SecureFileDelete(@ScriptDir & "test.bin", False, $array) ; use custom patterns, no delete

ConsoleWrite($r & " : " & @error & @CRLF)

</code>

I get

0 : 3

0 : 3

both times ,

file is not destroyed (monitored via a hex-editor)

i even tried True for deleting this file , neither deletes the file.

Also tried the same for "D:test.bin" with the corresponding changes on the .au script , neither saw anything changes.

Any ideas?

[BrewManNH]

Reread the information in the first post, he has modified the _SecureFileDelete function so that the parameters are not in the same place any longer. The example script in the first post has not been updated to reflect the changes made. You need to put the patterns array first, and then the option to delete the file or not.