NTFS Data Recovery

[joakim]

Developmen of this project has recently been re-activated.

DDan at forensicfocus made huge contribution!

It supports recovery of all files off an NTFS volume, when the $MFT record entry is still intact. That means regular deleted files, resident, non-resident, compressed, sparse, alternate data streams etc.

It supports images files (dd style) of both disk and partition. For disk images, both MBR and GPT style is supported.

Reparse points is now solved internally without depending on external tools.

It has a simpel gui and is not that hard to figure out how to use.

Anyways, here's the code, and I'm sure some people find it interesting.

It has been tested ok on XP SP3 x86 and x64, Vista x86, Windows 7 x86 and x64, and Windows 8 x64.

Documentation: http://code.google.com/p/mft2csv/wiki/NTFS_File_Extracter

Latest version can usually be found at; http://code.google.com/p/mft2csv/downloads/list

Todo:

• Add native support for PhysicalDrive to remove the requirement that the volume must be mounted (though the code is already there in the support for image files).

Edited March 14, 2013 by joakim

[joakim]

We continued development recently and now there is a much more stable and mature version available. The crash on Windows 7 x64 has been fixed, just to name one. From the readme:

This tool's main purpose is to extract files off NTFS formatted volumes. In this version it is not possible to specify specific files to extract. It will extract all. When starting up the program, you will be asked to select an output directory where to dump all extracted files. This can be reconfigured later anyway. On program startup, it will scan attached disks for any mounted NTFS volumes. Found volumes are put into the upper dropdown box. Select target volume and click button "Extract All Files".

It is also possible to select a disk image file to extract from. Found NTFS volumes will be put into lower dropdown box. Select target volume and click button "Extract from image".

Extracted files will be put into specified output directory with the original directory structure. $MFT records with files marked as deleted, will get a prefix in the output similar to [DEL+refnumber]originalname.ext. It supports extraction of all files with its MFT record intact.

A logfile is generated for each extraction, where issues and the relevant record are written.

Has been tested OK on XP 32-bit and 64-bit, Vista 32-bit, Windows 7 32-bit and 64-bit.

.

NTFSDataRecovery_v2000.zip

[llewxam]

Niiiiiiiiiiice, makes me motivated to complete the forensic imaging software I started forever ago! Are there any plans at this point to parse the bitmap file in order to allow selection of specific files for extraction?

Thanks for the continued efforts!

Ian

Edited November 9, 2012 by llewxam

[llewxam]

Quick first impressions:

Disk Access error until I included #RequireAdmin (Win7x64)

Line 63, "output" written as "utput", only shows when you change the source after reaching the GUI

This looks like another script where using an AdlibRegister to update the GUI during extraction phase could increase performance a lot

Also possible improvement would be progress bar for overall progress and per-file for when it hits a big one

Possibly consider having the GUI not stay on top

AND IT IS TOTALLY AWESOME!!!!!!! I am highly impressed!

Ian

[JFX]

That's really great

To speedup use case sensitivity for hex-string comparison

Also dllopen should help a bit.

[joakim]

@Ilewxam

I thought about different things regarding $bitmap. For the $MFT attribute bitmap, it is possible to parse it too, but since we scan the complete $MFT anyway and generate an array of the filesystem, it is probably easier to expand on that like populating a second array for deleted files. Regarding the $bitmap metafile, that's an other story, but could be used to dump unallocated. Also related is "slack" in all its forms and definitions, which I've thought about as an option to handle. However it will not happen anytime soon. Regarding the AdlibRegister we only update the gui every 10th record so I'm not sure the gains are that much. However if it turns out anything makes it run faster, it is still worth implementing.

@JFX

Did you mean setting something like

Global $hKernel32dll = DllOpen("kernel32.dll")

and modifying all winapi functions accordingly to use the global handle?

Just tried both those fixes (if I got them right) and it runs a little faster..

[llewxam]

Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write. However, that makes it impossible to have a smooth progressbar on transferring those large files. I started working on breaking it up in to smaller bits but my eyes can't handle any more coding right now so reverted in order to post this hack, which is my idea of the AdlibRegister.

Again though, I am really loving what I am seeing!

Ian

NTFSDataRecovery.zip

[JFX]

@joakim

Yes, passing the dllopen handle to the dllcalls should avoid loading and closing the dll on every call.

Speed gain is very small, just 3 seconds here, but maybe useful for a drive with very much files.

[joakim]

So a new version is up with the fixes suggested so far. Also now is added an option to only extract deleted files.

All suggestions for improvements are much appreciated

NTFSDataRecovery.zip

[wraithdu]

Couple things from recent comments:

I'm not sure how exactly you're using ReadFile / WriteFile, but check out my _LargeFileCopy UDF for buffered copying with callbacks for progress.

DllCalls to system loaded DLLs like kernel32 have miniscule (at best) speedups from using DllOpen / DllClose and passing the handle to the call. Those DLLs are always loaded so there is no load / unload overhead. And the lookup from passing the string name "kernel32.dll" versus a handle makes virtually no difference. So I wouldn't bother wasting time converting code or losing sleep over it.

Edited November 11, 2012 by wraithdu

[Ddan]

Hi,

I'm new to the forum, so if I'm doing something wrong please tell me kindly.

Joakim and I are working independently on this project. My main focus is getting the MFT analysis working properly. At this stage, I'm less concerned with minor speed-ups or smoothing progress bars as I think that can all come later. I'd love to hear any comments in regard to incorrectly extracted files. There is one bug fix that wasn't included in the latest version. If you search for "$Type > 128" (without the inverted commas) and change the value to 256, those No Data Attribute toss-outs should stop.

I'd like to take up some of the comments which have been posted and which don't seem to have been addressed. So in no particular order:

@llewxam (and later wraithdu)

<Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write.>

I'm not sure where you get this from, the files are handled in 16 cluster chunks. You'll both probably recognise that 16 clusters is a compression unit. A little experiment will show you why we went down this track. For simplicity try this on a freshly formatted usb stick. Copy a single file of 500+ mb to the stick and then look at the mft record and in particular the data run. What you will see is a simple data run of x clusters at location y. Now mark the file for compression and look at the new mft record. It will now have a non-resident attribute list. If you follow the list to the mft records that now contain the data run, you will see the longest data run you have ever seen all neatly broken up into compression units. The problem with handling the run is that you need to look at each compression unit to determine whether or not any compression has been done.

@llewxam

< Are there any plans at this point to parse the bitmap file in order to allow selection of specific files for extraction?>

No plans currently for single file extraction. Should be very easy to implement though as all the required data is in the FileTree array. It's held in mft record number order and contains all files and folders resolved back to the root folder (as far as possible anyway). Just really needs to be sorted, collected and presented for choosing. I'm tempted to add "Go for it!".

@llewxam

< This looks like another script where using an AdlibRegister to update the GUI during extraction phase could increase performance a lot>

Did you take any timings on this? I find running the same extraction multiple times often shows variations up to 2m (in 1h 50m). I guess it depends on what the system idling in the background consumes.

Ddan

Edited November 23, 2012 by Ddan

[llewxam]

Hi,

I'm new to the forum, so if I'm doing something wrong please tell me kindly.

HI! Nice to have you come by!

@llewxam (and later wraithdu)

<Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write.>

I'm not sure where you get this from, the files are handled in 16 cluster chunks.

I can only take your word for it, as nothing ever comes right out and says "for.... to 16" or "Do...Until X == 16", I have never tried to find the values of items such as $RUN_Clusters[$r] or tried to understand the workings of "If (($RUN_VCN[$r+1]=0) And ($i+$RUN_Clusters[$r+1]=16) And $IsCompressed) Then"

@llewxam

Did you take any timings on this? I find running the same extraction multiple times often shows variations up to 2m (in 1h 50m). I guess it depends on what the system idling in the background consumes.

I didn't get very scientific with this script, but when I put the same AdlibRegister method to work on Joakim's MFT2CSV performance jumped 20% for me, from 200 records per second top 240. And the way I figure the performance improvement could be even better for lower-end machines, as the GUI updating would take more time on a slower computer, so the less time you have to do it the more time can be spent on number crunching. But I could try to find the time to run a comparison to see what happens on this script.

Glad to see this is still a work in progress, and I wish I were able to contribute more to it than just some little GUI speed tweaks, but you guys are pretty far above my head as it is!

Thanks, and welcome.

Ian

[Ddan]

I can only take your word for it, as nothing ever comes right out and says "for.... to 16" or "Do...Until X == 16", I have never tried to find the values of items such as $RUN_Clusters[$r] or tried to understand the workings of "If (($RUN_VCN[$r+1]=0) And ($i+$RUN_Clusters[$r+1]=16) And $IsCompressed) Then"

True, but on the other hand:

_WinAPI_ReadFile1($hDisk, DllStructGetPtr($cBuffer), $BytesPerCluster * 16, $nBytes)

and

_WinAPI_WriteFile1($hFile, DllStructGetPtr($cBuffer), $BytesPerCluster * 16, $nBytes)

sort of says it all. (Haven't worked out the smiley bit yet, so assume a bigger grin than your's)

Ddan

[llewxam]

(Haven't worked out the smiley bit yet, so assume a bigger grin than your's)

HAHAHA!!!!

Ian

[llewxam]

Just a quick message that I am trying this tool in a real-world situation and it is showing good results. A client's hard drive crashed, I pulled the files I needed with R-Studio and got her back up, did not make an image of the drive and decided to try this tool on her failing drive to see how it would go. It has been running a very long time since the drive is in very bad shape, but the files it has pulled off look good.

Another hugely useful feature would be to select from the available files which you want it to grab, no doubt that will be in the works at some point.

So, A+ for effectiveness with the current build on a dying drive!

Ian

[Ddan]

NTFSDataRecovery.zip

Just a quick message that I am trying this tool in a real-world situation and it is showing good results. A client's hard drive crashed, I pulled the files I needed with R-Studio and got her back up, did not make an image of the drive and decided to try this tool on her failing drive to see how it would go. It has been running a very long time since the drive is in very bad shape, but the files it has pulled off look good.

Just a few quick comments.

First off, use this attached version as earlier one had a few bugs, which I think have now been corrected.

In regard to recovering a failing drive, always take an image first (I use WinHex). That way you don't need the drive anymore but more importantly, the access time is a million times quicker. Well...maybe not a million.

[llewxam]

hehe, agreed on imaging first, I usually do but just felt like giving it a good torture test!

I started coding a drive cloning program of my own which won't have nearly all of the features of WinHex which I use as well, but it will do the things I care about - clone to or from an image, disk-to-disk clone, sector integrity test, and zero-fill the drive (secure wipe). It mostly works but without having a way to gain exclusive access to the target drive when restoring an image it fails most of the time, and I have not found a solution to that.

I have sent the as-is version to a few people to check out, if you would like to look it over I would be happy to send it to you in a PM. Just let me know.

Ian

[joakim]

$Ddan

I will take a look at it when I'm done moving into the new house.

$llewxam

I bet it's nt.6x that's screwing you. Read up on this; http://msdn.microsoft.com/en-us/library/windows/desktop/aa364575(v=vs.85).aspx I wrote a tiny app a few years ago called LockVolume, which was posted as reboot.pro, but the site is currently down. There you will see an example of how I solved it. But I could be wrong, as it might be something different. Hard to say without having seen it.

[llewxam]

Thanks joakim, I will have a look. I found some code on the forum here yesterday that I thought would fix me up, it helped but not 100%, when writing an image to the disk it gets the first 1 or 2 8MB chunks written then has 5-5 fails in a row, then works again. Frustrating!

Thanks again, will check it out later tonight.

Ian

[joakim]

Then it's a different thing as I thought the issue where related to writing directly to filesystem sectors (for instance restoring image). If it was for image creation, ignore my comment completely.