getting last between start and end string of a big file

[tom13]

this has some success for me...

#RequireAdmin
#include <WinAPI.au3>
Dim $nBytes
$sFile = @ScriptDir & "\test.txt"

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        ; read 100 bytes from end of file
        $tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")
        $hFile = _WinAPI_CreateFile($sFile, 2, 2)
        If $hFile And ($hFile <> 0xFFFFFFFF) Then
            _WinAPI_SetFilePointer($hFile, $prevsize)
            _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
            _WinAPI_CloseHandle($hFile)
            $sText = DLLStructGetData($tBuffer, 1)
            MsgBox(4096, "", $sText)
           $prevsize = $newsize
        EndIf
    EndIf
WEnd

but this will cycle crazy if it can't get the $hFile. Maybe an Else... Sleep(50) on the If $hFile .

Lar.

I tried your example (ofc changed the logfile path) and while it does notice when the file is changed, the messagebox is empty.. any idea why? I am using vista ultimate 64 bit if that is of any importance. I have tried to run it in both 32 bit and 64 bit autoit beta.

[tom13]

Hey tom sorry didn't check back on this. Alot of replies but looks like you might not have it the way you want yet.

Here's a working example of tail.exe I tested myself using a space in path.

#include <Process.au3>
$File = "C:\Documents and Settings\Administrator\Test.txt"

_RunDOS('Tail.exe 10 "' & $File & '">Tail.txt')

;In this exmaple tail.exe is in the script directory
;and output will dump into Tail.txt in script directory

Now that you have a much smaller file you could read it all into a string if you want:

#Include <Array.au3>
#Include <File.au3>
$FileRead = ""

_FileReadToArray("Test.txt", $FileRead)
$FileRead = _ArrayToString($FileRead)
$FileRead = StringReplace($FileRead, "|", "");If you want to get rid of the | delimeter.
MsgBox(4096,"$FileRead", $FileRead)

-Kenny

Hey, thanks but are you sure you have the same version as that link you gave me? Because when I do it the Tail.txt file is empty.. even though the log file is like 6MB big.

[ken82m]

I wondered the same thing. I downloaded the same version I sent you and the syntax was a a little different.

The line count used to come last this version it came first.

Give the new script a try and see what happens. Just change the $FILE variable to whatever your file is and see if it works.

I found the one I used in my old script it was written by Microsoft and is 6K.

The syntax is definitely different.

-Kenny

Edited October 12, 2008 by ken82m

[Zedna]

this has some success for me...

but this will cycle crazy if it can't get the $hFile. Maybe an Else... Sleep(50) on the If $hFile .

Lar.

It works for me too. It just gets text without 2 first chars.

[LarryDalooza]

Is this an ANSI file?

[tom13]

I wondered the same thing. I downloaded the same version I sent you and the syntax was a a little different.

The line count used to come last this version it came first.

Give the new script a try and see what happens. Just change the $FILE variable to whatever your file is and see if it works.

-Kenny

Hey like I said above: I tried it and it did not work because the result file (Tail.txt) was empty while the logfile is not empty at all.

@ Zedna,

That is so weird.. anything I can try? Is there a relation between this autoit script giving an empty string and the third party program of Kenny giving an empty string aswell maybe?

[tom13]

Is this an ANSI file?

Hmm, the harddisk is NTFS. Or isn't that what you mean?

Edit: oh you mean character encoding of course, it's just normal text like we are typing now. This is a part of it:

[08/10/12][14:12:37][zWB_POS]1415053.625x1454215[/zWB_POS]
[08/10/12][14:12:37][zWB_POS]1415036.875x1454193.25[/zWB_POS]
[08/10/12][14:12:37][zWB_POS]1414979.375x1454118.25[/zWB_POS]
[08/10/12][14:12:38][zWB_POS]1414931.375x1454055.75[/zWB_POS]
[08/10/12][14:12:38][zWB_POS]1414926.125x1454046.25[/zWB_POS]

Edited October 12, 2008 by tom13

[ken82m]

can you attach a small portion of the log file, say 10 or 20 lines. And the directory the file is in?

I can try to reproduce the problem here.

-Kenny

[tom13]

can you attach a small portion of the log file, say 10 or 20 lines. And the directory the file is in?

I can try to reproduce the problem here.

-Kenny

Hey, yeah I edited it to my previous post. Also, the path is C:\WAR\logs\testchat.log

[Zedna]

Is this an ANSI file?

@OP

If it's unicode then change

$tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")

to

$tBuffer = DLLStructCreate("wchar[" & ($newsize - $prevsize) & "]")

[tom13]

@OP

If it's unicode then change

$tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")

to

$tBuffer = DLLStructCreate("wchar[" & ($newsize - $prevsize) & "]")

That did the trick!

Very very awesome.

Much love to Zedna, Kenny, Jos, Bowmore and Larry!

Edited October 12, 2008 by tom13

[ken82m]

Okay I created that directory and the file portion you posted.

I ran this and it correctly returned what I asked for:

#include <Process.au3>
#Include <Array.au3>
#Include <File.au3>
$FileRead = ""
$File = "c:\war\logs\testchat.log"

_RunDOS('Tail.exe 11 "' & $File & '">Tail.txt')
_FileReadToArray("Tail.txt", $FileRead)
$Header = $FileRead[0]
$Header = StringLen($Header)
$FileRead = _ArrayToString($FileRead)
$FileRead = StringReplace($FileRead, "|", @CR);If you want to get rid of the | delimeter.
$FileRead = StringTrimLeft($FileRead, $Header)
MsgBox(4096,"$FileRead", $FileRead)

Exit

-Kenny

Hey, yeah I edited it to my previous post. Also, the path is C:\WAR\logs\testchat.log

[Zedna]

It may be sharing mode problem. Application writing to file must open file in mode for share reading.

You must open this file in share mode too.

Try

$hFile = _WinAPI_CreateFile($sFile, 2, 2+4)

instead of

$hFile = _WinAPI_CreateFile($sFile, 2, 2)

[ken82m]

Cool, I wish I knew C so I could make sense of that lol

@OP

If it's unicode then change

$tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")

to

$tBuffer = DLLStructCreate("wchar[" & ($newsize - $prevsize) & "]")

[tom13]

OK, so I thought I had fixed it however,

it only works when the application that is writing the log file is not active (or when log writing has been disabled).

But, when I start the application the handle will always fail (returns 0xFFFFFFFF).. this is my snippet code:

$newsize = FileGetSize($chatlog)
            if $newsize > $prevsize then
                $nrbytes = $newsize - $prevsize
                $tBuffer = DLLStructCreate("wchar[" & $nrbytes & "]")
                $hFile = _WinAPI_CreateFile($chatlog, 2, 2)
                If $hFile And ($hFile <> 0xFFFFFFFF) Then
                    _WinAPI_SetFilePointer($hFile, $prevsize)
                    _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $nrbytes, $nBytes)
                    _WinAPI_CloseHandle($hFile)
                    $sText = DLLStructGetData($tBuffer, 1)
                    $prevsize = $newsize
                    $pos = _StringBetween($sText, "[zWB_POS]", "[/zWB_POS]")
                Else
                    Sleep(5)
                EndIf
            EndIf

I have also tried several other handles but they did not succeed either:

$hFile = _WinAPI_CreateFile($chatlog, 2, 2+4)
$hFile = _WinAPI_CreateFile($chatlog, 2, 2, 2)

Can anyone help me out, please?

Edited October 12, 2008 by tom13

[Bowmore]

OK, so I thought I had fixed it however,

it only works when the application that is writing the log file is not active (or when log writing has been disabled).

But, when I start the application the handle will always fail (returns 0xFFFFFFFF).. this is my snippet code:

$newsize = FileGetSize($chatlog)
            if $newsize > $prevsize then
                $nrbytes = $newsize - $prevsize
                $tBuffer = DLLStructCreate("wchar[" & $nrbytes & "]")
                $hFile = _WinAPI_CreateFile($chatlog, 2, 2)
                If $hFile And ($hFile <> 0xFFFFFFFF) Then
                    _WinAPI_SetFilePointer($hFile, $prevsize)
                    _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $nrbytes, $nBytes)
                    _WinAPI_CloseHandle($hFile)
                    $sText = DLLStructGetData($tBuffer, 1)
                    $prevsize = $newsize
                    $pos = _StringBetween($sText, "[zWB_POS]", "[/zWB_POS]")
                Else
                    Sleep(5)
                EndIf
            EndIf

I have also tried several other handles but they did not succeed either:

$hFile = _WinAPI_CreateFile($chatlog, 2, 2+4)
$hFile = _WinAPI_CreateFile($chatlog, 2, 2, 2)

Can anyone help me out, please?

I think we may have hit an impasse. If the application that is writing the file has the file locked for writing and has not allowed.

FILE_SHARE_READ (0x00000001) access rights then _WinAPI_CreateFile() will never be able to get a valid handle to be able to read the file.

[tom13]

I think we may have hit an impasse. If the application that is writing the file has the file locked for writing and has not allowed.

FILE_SHARE_READ (0x00000001) access rights then _WinAPI_CreateFile() will never be able to get a valid handle to be able to read the file.

Why is this the case while my notepad etc. can still read the logfile? Why can I still use Fileread from the normal AutoIT functions (which loads the whole file into memory unfortunately)?

Did we just waist 2 days for something that is never going to happen?

Edit: I do not need to open the file for writing, I only need to read it. So why would I need shared writing rights? I only need to read (the end of the file), just like Fileread() and notepad.

Edit2: does it have to do anything with Chrisl's post?

Not sure if this will work or not but try replacing this function _FileOpenAPI($sFile) in the APITailRW.au3 file with this one below. According to MSDN the file is opened with exclusive access so it is likely failing becasue another process already has it open, I think this should enable it to share access

Func _FileOpenAPI($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
;$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", 0, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc ;==>_FileOpenAPI

Edited October 13, 2008 by tom13

[tom13]

Oh come on, this can't be the end right? There must be something to fix this? I tried the following but no success either..

$hFile = _get_file_handle($chatlog)

func _get_file_handle($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
endfunc

Anything I can do?

If this does waist all the time we had spent, is there any other way to read (the last lines of) a big text file within a couple of seconds? Note that I only need to read, just like notepad can still open the file even when the application is logging! Any ideas?

Edited October 13, 2008 by tom13

[PsaltyDS]

Oh come on, this can't be the end right? There must be something to fix this? I tried the following but no success either..

$hFile = _get_file_handle($chatlog)

func _get_file_handle($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
endfunc

Anything I can do?

If this does waist all the time we had spent, is there any other way to read (the last lines of) a big text file within a couple of seconds? Note that I only need to read, just like notepad can still open the file even when the application is logging! Any ideas?

Looks to me like you are still requesting WRITE access because of: BitOR($GENERIC_READ, $GENERIC_WRITE)

Shouldn't that just be $GENERIC_READ for the dwDesiredAccess parameter?

[tom13]

Looks to me like you are still requesting WRITE access because of: BitOR($GENERIC_READ, $GENERIC_WRITE)

Shouldn't that just be $GENERIC_READ for the dwDesiredAccess parameter?

Thanks it actually seems to be able to open the handle now:

$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",$GENERIC_READ, "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)oÝ÷ Øz0z÷«²¶)²Ø^­ën®{-®)àÓ~¬MìmǦ¦Ü½éí Z½æuçf¢·­{m¡éÝ¡ûazZ ~)^¬j|¨ë§uê쵩ÝÂ+"±«'^¶æÉÊy«­¢+ØÀÌØíÁ½Í}½±ô}Ñ}Á½Í}µ¹Õ° ¤($%5Í    ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½ÐíÁ½Ì½±èÅÕ½ÐìµÀìÀÌØíÁ½Í}½±¤($%}µ½Ùµ¹Ñ}ÍÑÉÐ ¤($$ÀÌØíÁ½ÌôÀí¥¹¥ÐÁ½Ì($$ÀÌØíÁ½ÌÈôÀí¥¹¥ÐÁ½ÌÈ($%¼($$$ÀÌØí¡¥±ô}Ñ}¥±}¡¹± ÀÌØí¡Ñ±½¤($$$ÀÌØí¹ÝÍ¥éô}]¥¹A%}Ñ¥±M¥éà ÀÌØí¡¥±¤($$%}]¥¹A%}
±½Í!¹± ÀÌØí¡¥±¤($$$ìÀÌØí¹ÝÍ¥éô¥±ÑM¥é ÀÌØí¡Ñ±½¤($$%5Í ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½Ðí½±Í¥éèÅÕ½ÐìµÀìÀÌØíÁÉÙÍ¥éµÀì
HµÀìÅÕ½Ðí¹ÜÍ¥éèÅÕ½ÐìµÀìÀÌØí¹ÝÍ¥é¤($$%¥ÀÌØí¹ÝÍ¥éÐìÀÌØíÁÉÙÍ¥éÑ¡¸($$$$ÀÌØí¹ÉåÑÌôÀÌØí¹ÝÍ¥é´ÀÌØíÁÉÙÍ¥é($$$$ÀÌØíÁÉÙÍ¥éôÀÌØí¹ÝÍ¥é($$$$ÀÌØíÑ    ÕÈô11MÑÉÕÑ
ÉÑ ÅÕ½ÐíÝ¡ÉlÅÕ½ÐìµÀìÀÌØí¹ÉåÑ̵ÀìÅÕ½ÐítÅÕ½Ðì¤($$$$ÀÌØí¡¥±ô}Ñ}¥±}¡¹± ÀÌØí¡Ñ±½¤($$$%5Í    ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½Ðí¡¹±èÅÕ½ÐìµÀìÀÌØí¡¥±¤($$$%%ÀÌØí¡¥±¹ ÀÌØí¡¥±±ÐìÐìÁá¤Q¡¸($$$$%}]¥¹A%}MÑ¥±A½¥¹ÑÈ ÀÌØí¡¥±°ÀÌØíÁÉÙÍ¥é¤($$$$%}]¥¹A%}I¥± ÀÌØí¡¥±°11MÑÉÕÑÑAÑÈ ÀÌØíÑ   ÕȤ°ÀÌØí¹ÉåÑÌ°ÀÌØí¹  åÑ̤($$$$%}]¥¹A%}
±½Í!¹± ÀÌØí¡¥±¤($$$$$ÀÌØíÍQáÐô11MÑÉÕÑÑÑ ÀÌØíÑ   ÕȰĤ($$$$%5Í ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½ÐíÑáСÍÑÉÑ¥¹ÉÉäÑÈÑ¡¥Ì¤èÅÕ½ÐìµÀìÀÌØíÍQáФ($$$$$ÀÌØíÁ½Ìô}MÑÉ¥¹    Ñݸ ÀÌØíÍQáаÅÕ½Ðímé]  }A=MtÅÕ½Ðì°ÅÕ½Ðíl½é]   }A=MtÅÕ½Ðì¤($$$$%¥¡¥ÍÉÉä ÀÌØíÁ½Ì¤¤¹ ÀÌØíÁ½Ì±ÐìÐìÀ¤Ñ¡¸($$$$$$ÀÌØíÁ½ÌÈôÀÌØíÁ½Íl¡U    ½Õ¹ ÀÌØíÁ½Ì¤´Ä¥t($$$$$%5Í    ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½ÐíèÁ½ÌÈèÅÕ½ÐìµÀìÀÌØíÁ½ÌȤ($$$$%±Í($$$$$$ÀÌØíÁ½ÌÈôÀ($$$$%¹¥($$$$%5Í ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½Ðí½¹Ý¥Ñ ÉÉäÅÕ½Ðì¤($$$%±Í($$$$%M±À Ô¤($$$%¹%($$%¹%($$%5Í   ½à À°ÅÕ½ÐìÅÕ½Ðì°ÅÕ½ÐíÁ½ÌÈèÅÕ½ÐìµÀìÀÌØíÁ½ÌȵÀì
HµÀìÅÕ½ÐíÁ½Ì½±èÅÕ½ÐìµÀìÀÌØíÁ½Í}½±¤($%չѥ° ÀÌØíÁ½ÌȱÐìÐìÀÌØíÁ½Í}½±¤¹ ÀÌØíÁ½ÌȱÐìÐìÀ¤()Õ¹}Ñ}¥±}¡¹± ÀÌØíÍ¥±¤(%1½°ÀÌØí9I%
}IôÁààÀÀÀÀÀÀÀ°ÀÌØí9I%
}]I%QôÁàÐÀÀÀÀÀÀÀ°ÀÌØí=A9}1]eLôаÀÌØí%1}QQI%  UQ}9=I50ôÁàÀÀÀÀÀÀàÀ°ÀÌØí%1}M!I}9=9ôÁàÀÀÀÀÀÀÀÀ°ÀÌØí%1}M!I}IôÁàÀÀÀÀÀÀÀÄ°ÀÌØí%1}M!I}]I%QôÁàÀÀÀÀÀÀÀÈ(1½°ÀÌØí=} ($ÀÌØí=} ô±±
±° ÅÕ½Ðí­É¹°Ìȹ±°ÅÕ½Ðì°ÅÕ½Ðí¡Ý¹ÅÕ½Ðì°ÅÕ½Ðí
ÉÑ¥±ÅÕ½Ðì°ÅÕ½ÐíÍÑÈÅÕ½Ðì°ÀÌØíÍ¥±°ÅÕ½Ðí±½¹ÅÕ½Ðì°ÀÌØí9I%
}I°ÅÕ½Ðí±½¹ÅÕ½Ðì°    ¥Ñ=È ÀÌØí%1}M!I}I°ÀÌØí%1}M!I}]I%Q¤°ÅÕ½ÐíÁÑÈÅÕ½Ðì°À°ÅÕ½Ðí±½¹ÅÕ½Ðì°ÀÌØí=A9}1]eL°ÅÕ½Ðí±½¹ÅÕ½Ðì°ÀÌØí%1}QQI%  UQ}9=I50°ÅÕ½Ðí±½¹ÅÕ½Ðì°À¤(IÑÕɸÀÌØí=}¡lÁt)¹Õ¹

I hope we can finally get this working now!