Read Binary index.dat file

[Graywalker]

The goal here is to get the Internet History for endpoints and users in the enterprise to find what business use websites use Java.

When opening index.dat in Notepad, I can see all the data I need. Unfortunatly, trying to read that data in AutoIT is proving diffucult.

I've tried :

expandcollapse popup

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,2)
MsgBox(0,"String from Binary",$strIndexdat)
$FileArray = StringSplit($strIndexdat,@CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries

; Start reading from line 1
For $line In $FileArray
$content = StringStripWS($line,7)
  Select
   Case StringInStr($line,"REDR")
    ; this is the start of a record
    MsgBox(0,"REDR",$content)
   Case StringInStr($line,"URL")
    ; this may be the start of a record
    MsgBox(0,"URL",$content)
   Case StringInStr($line,"LEAK")
    ; this is the start of an error record
    ; I will likely ignore it.
    MsgBox(0,"LEAK",$content)
   Case StringInStr($line,"http://")
    ; this is the line with a couple entries
    MsgBox(0,"http",$content)
   Case StringInStr($line,"Content-Type:")
    ; this is an entry I want
    MsgBox(0,"Content-Type",$content)
   Case StringInStr($line,"X-Powered-By:")
    ; this is an entry I want
    MsgBox(0,"Powered-By",$content)
   Case StringInStr($line,"~U:")
    ; this is an entry I want and it marks the end of a record
    MsgBox(0,"~U",$content)
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc

That doesn't get the info... That code doesn't return anything.

Using

$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$strIndexdat)

the "string from binary" msg box shows a LOT of data... but URL and http case are both blank or gibberish. BinaryToString($indexdat,2) and ,3 return Nothing in the cases - 4 does the same as 1.

Using :

$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Shows that $indexdat and $strIndexdat are effectively the same.

Using :

; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
;$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($indexdatpath);$Bindexdat)
;$strIndexdat = BinaryToString($indexdat,4)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($indexdat,@CRLF)

It reads a lot more info... Content-Type is fine. ~U: is fine. URL is mostly blank.

Case http:// it will pop up the msg box, but $content is blank. So the data is there... I just can't figure out how to get it into a string.

Using :

$Bindexdat = FileOpen($indexdatpath,256)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Returns data, but URL and Http:// pop up as blank...

$Bindexdat = FileOpen($indexdatpath,48) - URL and Http are blank.

I've found a VBScript that is supposed to read the files... so far no luck on Win7 - it can't find index.dat file... lol! On remote XP machines, it can't find a history folder. So I don't know if it DOES read index.dat files.

I've attached it.

Any ideas or code help is greatly appreciated!!

IE_Network - Copy.txt

Edited January 23, 2012 by Graywalker

[Graywalker]

Looks like Vengy forgot to include a function in his IE_Network script. It can be found here :

http://www.motobit.com/tips/detpg_binarytostring/

[Graywalker]

I altered the vbscript to point directly to the index.dat file and it gets info, but returns all kinds of gibberish.

... that totally messes up the reply even when pasted as code!

So, I've attached it as a screen cap.

Edited January 23, 2012 by Graywalker

[Graywalker]

Okay, I've found out that the data IS there. I've just got to find a way to get TO it.

expandcollapse popup

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
;MsgBox(0, "Index dat", $indexdat)
$strIndexdat = BinaryToString($indexdat, 1)
$strIndexdat = StringStripWS($strIndexdat, 7)
;MsgBox(0, "String from Binary", $strIndexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat, @CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries
; Start reading from line 1
For $line In $FileArray
  $line = StringReplace($line, @CRLF, "")
  $line = StringReplace($line, @CR, "")
  $line = StringReplace($line, @LF, "")
  Select
   Case StringInStr($line, "http://")
    ; this is the line with a couple entries
    ;MsgBox(0, "http", $line)
    ; Check to see if URL is in the line
    If StringInStr($line, "URL") Then
     ; it is a REDR or LEAK, trim to URL
     $urlpos = StringInStr($line, "URL")
     $line = StringTrimLeft($line, $urlpos)
    EndIf
    $httppos = StringInStr($line, "http")
    $line = StringTrimLeft($line, $httppos - 1)
    $dotpos = StringInStr($line, ".", "", 3)
    If $dotpos > 10 Then
     $content = $line
     $linelen = StringLen($line)
     $trimfromright = $linelen - ($dotpos + 3)
     $line = StringTrimRight($line, $trimfromright)
     ;MsgBox(0,"Trim", "string lenght : " & $linelen & @CRLF & "dot position: " & $dotpos & @CRLF & "Trim from Right : " & _
     ;$trimfromright & @CRLF & $line)
    Else
     ; nothing
    EndIf
    $record = $line & ","
   Case StringInStr($line, "Content-Type:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "X-Powered-By:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "~U:")
    ; this is an entry I want and it marks the end of a record
    $line = StringReplace($line, "~U:", "")
    $line = StringStripWS($line, 7)
    $record = $record & $line
    FileWriteLine($logfile, $record)
    $record = ""
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc   ;==>_ParseIndexdat

I am getting fairly spotty results - leaving in lots of random characters before and after the URLs.

Could anyone come up with a way to use StringRegExp to pull the URLs - all of them - from a line?

I've tried several from : http://regexlib.com/Search.aspx?k=URL&AspxAutoDetectCookieSupport=1

with no consistent luck.