MFT-Access - Reading & parsing the Master File Table on NTFS Filesystems

[wraithdu]

I've got a personal project I've been working on to update a secure erase UDF of mine. It accomplishes some of the same things as your UDF, but using the defrag API. Is that at all related to what you're doing and would you be interested in seeing it? It works in at least one direction to find virtual and physical clusters belonging to any file, including sparse and EFS encrypted files. It cannot decrypt these files however, and extraction only yields undecryptable data. But it is useful for my purposes.

Edited December 28, 2011 by wraithdu

[joakim]

@wraithdu

Yes it would be interesting to see if anything there could be used.

Attached is the latest sources I've used. Resident data extraction is the latest fix.

NTFS_271211.zip

[wraithdu]

Here's my current UDF and an example.

_FileMapping.au3

FileMapExample.au3

Edited December 28, 2011 by wraithdu

[KaFu]

Here's my current UDF and an example.

Now that looks really nice , I definitely have to dive deeper into this one. One quick finding I've got, replace the $GENERIC_READ in the _FM_GetDiskSectorSize() function call to _FM_CreateFile() with 0 otherwise on Vista+ elevated Access is required.

Edit: Oh, that's not enough. In the function _FM_ReadMapToFile() you access the disk directly again, and at that point the $GENERIC_READ is definitely required. When UAC is acitvated the script definitely needs elevated access, add this line to the top of the example and compile it.

#AutoIt3Wrapper_Res_requestedExecutionLevel=requireAdministrator

Maybe add a cross check if UAC is activated to the top of the include, and if so a second check if the script runs elevated.

But this looks really, really good !

Edited December 29, 2011 by KaFu

[joakim]

Here's my current UDF and an example.

Thanks for those. You have had many examples I've had great benefit from. I'm currently not sure if these ones can be used in "my" solution since it's purpose is to be completely FS api independant. But I'm very thankful for the sources anyway.

In the last version of the extracter (now at v1.6) I've added experimental support for compressed files. And thanks to trancexx for the examples of RtlDecompressBuffer usage that now came useful.

The biggest and most important challenge, the way I see it, now is solving file paths / B+ trees. When that's done a whole new world of interesting possibilities suddenly became real..

[wraithdu]

Just back from vacation

I'm glad you found it at least interesting, I know it's a different approach than your work. To comment on one thing, admin access has always been required for disk defragmentation. Since my UDF uses that same API it's no surprise the same access is needed.

[joakim]

Added an option to the file extracter for browsing and doing a low level file copy/extraction based on the file name. It was added for faster extraction and detection of regular files (ie non system files), and uses NtQueryInformationFile and FileInternalInformation to get to the index number. (wraithdu maybe you recognize parts of that code)

[joakim]

There has been som great improvements to these tools lately:

• A new console app to decode records for individual files has been introduced.
• Compressed and sparse files are now fully working with extraction.
• Extraction for extremely fragmented files are also now supported, as $ATTRIBUTE_LIST is resolved and implemented.
• Full extraction of all ADS's tied to a file.

Source: http://code.google.com/p/mft2csv/downloads/list

[llewxam]

Looking great! One suggestion though - too much GUI updating. I've found that throwing in an AdlibRegister for things like GUI updates can massively increase performance. I commented out the ProgressSet of your For/Next loop, placed AdlibRegister("_DisplayProgress") immediately above the For/Next loop, and added the function:

Func _DisplayProgress()
ProgressSet(Round((($i / $ExpectedRecords) * 100), 2), Round(($i / $ExpectedRecords) * 100, 2) & " % finished with exporting $MFT data to csv", Round($i / (TimerDiff($begin) / 1000), 1) & " Records per second.")
EndFunc ;==>_DisplayProgress

That increased performance from 200 records per second to 240!!

But it is looking pretty damn sweet man! I would assume that some correlation would need to be made with the $Bitmap file and some pretty handy data recovery software could be built from this!!!

Great stuff

Ian

EDIT - Sorry, should have specified that this change was made to MFT2CSV

Edited June 29, 2012 by llewxam

[joakim]

I think that's the ultimate goal, to create a data recovery solution for NTFS. But it's still a way to go from here to there.. Looking more promising though

Thank you very much for the tip

[DeltaRocked]

Hi Joakim,

any idea on how to read/modify $I30 ?

Regards

Deltarocked.

Edited July 27, 2012 by deltarocked

[joakim]

Reading is not a problem. Have a look at the newest code in MFTRCRD how $INDEX_ROOT and $INDEX_ALLOCATION is decoded. Writing is slightly more tricky though, and completely blocked on the systemdrive on nt6.x (unless you implement a kernel driver with it). If you look at SetMACE, a timestamp manipulating utility, writing is actually implemented in the latest version. Of course you need to know for sure what, where and how you are writing to disk, to prevent filesystem corruption. And actually you gave me an idea there, for how to implement timestamp modification in INDX records..

[DeltaRocked]

Hi joakim,

1: modifying INDX records do not require kernel level driver.

2: for past 2 days I have been searching all over the net for step 1 : modifying $I30 and ultimately decided to ask you - cause yours was the only utility which does 100% justice to reading/parsing MFT.

3: INDX records are used for Forensics - hence you will have to modify this inorder to make SetMACE a perfect utility to bypass even the best of the best Forensic softwares.

regards

DeltaRocked

[joakim]

The kernel driver requirement is only necessary when working on nt6.x's systemdrive (inside the boundaries of the volume/filesystem). All else works just fine without such a driver.

With the last code I added yesterday, $I30 is almost fully decoded. Implementing modification of it can be done with a little more work.

Regarding forensic implications, there's also other things to handle to make it perfect. The journal is one place I already have thought of. Another is the $LogFile, but that one is more tricky. When that's done, it is approaching perfection.

What exactly is your goal?

[DeltaRocked]

Goal : to add a file in MFT and add an INDX record. that is for anti-malware purpose.

As of this moment , I am creating a file through CreateFile, then opening up MFT reaching that location and reading 1024 bytes and then writing '0's . after that I am relocating the same file to a different record within MFT.

Due this INDX contains the file record (wyhich has been created by Windows ) but physically the file is located in some other location - hence another file with the same name cannot be created. I cannot say more but its for anti-malware.

regards

Deltarocked.

[uPDATE]

I am unable to print $INDEX_ROOT when using MFTRCRD.

MFTRCRD.exe g:autorun.inf -d attriblist=on indxdump=on

Starting MFTRCRD by Joakim Schicht

Version 1.0.0.12

Target is a File

Filesystem on g: is NTFS

NtQueryInformationFile: Success

File IndexNumber: 27

NtQueryInformationFile: Success

BytesPerSector: 512

SectorsPerCluster: 8

ReservedSectors: 0

SectorsPerTrack: 63

NumberOfHeads: 255

HiddenSectors: 8064

TotalSectors: 7827583

LogicalClusterNumberforthefileMFT: 262144

LogicalClusterNumberforthefileMFTMirr: 489223

Target 1B000000 found

Found attributes:

$STANDARD_INFORMATION (1)

$FILE_NAME (1)

$OBJECT_ID (1)

$DATA (1)

Record header info:

Offst to update sequence number: 48

Update sequence array size (words): 3

$LogFile sequence number (LSN): 0000000000801A64

Sequence number: 1

Hard link count: 1

Offset to first Attribute: 115

Flags: FILE

Real size of the FILE record: 344

Allocated size of the FILE record: 1024

File reference to the base FILE record: 0000000000000000

Next Attribute Id: 0005

Number of this MFT Record: 27

Update Sequence Number (a): 0400

Update Sequence Array (a): 00000000

$STANDARD_INFORMATION 1:

HEADER_Flags:

CreationTime (CTime): 2012-07-27 10:49:11:370:5000

LastWriteTime (ATime): 2012-07-27 11:21:29:058:0000

ChangeTime (MTime): 2012-07-27 11:21:29:058:0000

LastAccessTime (RTime): 2012-07-27 10:49:11:370:5000

DOS File Permissions: archive

Max Versions: 0

Version Number: 0

Class ID: 0

Owner ID: 0

Security ID: 258

USN: 0000000000000000

$FILE_NAME 1:

ParentSequenceNo: 5

CreationTime (CTime): 2012-07-27 10:49:11:370:5000

LastWriteTime (ATime): 2012-07-27 10:49:11:370:5000

ChangeTime (MTime): 2012-07-27 10:49:11:370:5000

LastAccessTime (RTime): 2012-07-27 10:49:11:370:5000

AllocSize: 0

RealSize: 0

Flags: archive

NameLength: 11

NameType: POSIX

NameSpace: 10

FileName: autorun.inf

ParentReferenceNo: 5

$OBJECT_ID 1:

GUID Object Id: EFAC75C8-B6D7-E111-A7A3-002618C2EA56

GUID Birth Volume Id: NOT PRESENT

GUID Birth Object Id: NOT PRESENT

GUID Domain Id: NOT PRESENT

$DATA 1:

Length: 32

Non-resident flag: 00

Name length: 0

Offset to the Name: 24

Flags: 0000

Attribute Id: 0001

Resident - Length of the Attribute: 3

Resident - Offset to the Attribute: 24

Resident - Indexed flag: 0

Resident - Padding: 00

Non-Resident - Starting VCN:

Non-Resident - Last VCN:

Non-Resident - Offset to the Data Runs:

Non-Resident - Compression Unit Size:

Non-Resident - Padding:

Non-Resident - Allocated size of the attribute:

Non-Resident - Real size of the attribute:

Non-Resident - Initialized data size of the stream:

The Attribute's Name:

FALSE------------->

FINISHED!!

Edited July 27, 2012 by deltarocked

[joakim]

I think I need some more details in order to understand the excersise. Consider communication channel alternatives.

[joakim]

Are you sure that the file autorun.inf really has an $INDEX_ROOT attribute?

Try again with;

MFTRCRD G:5 -a attriblist=off indxdump=on

Note attriblist=off since I'm 100% a small text file never will be that fragmented.. (speeds up processing). The "G:5" will then dump INDX's for the root directory.

[joakim]

Also, you should use v1.0.0.13, the latest version which has fixes for a lot of the relevant things you care about

[DeltaRocked]

Hi joakim,

the G:5 has worked perfectly. Sorry, couldn't answer earlier (Spondylosis) kept me away from the comp.

Within next few hours chk your mail, I will be sending you an interesting log of $MFT and $I30.

You can replicate it and have fun.

Regards

Deltarocked.

[nullschritt]

I had wondered, if it would be possible to use this script to find all the data on the disk that is "available for writing" (files that have been deleted by the OS, but which remain on the HDD), and overwrite it with blank data (basically removing the ability to recover any files). [sorry if the answer is obvious, I am at work and cannot read through all of the code/comments yet]

It could be a good security/privacy tool, I tried something similar a few years back, but it was sooo slow.