getting last between start and end string of a big file

[trancexx]

Yeah. And the file is like 200 MB. Makes sense right?

Needs to go within a few milliseconds.

Your first post was mentioning 20MB! I did read it couple of times.

If it's 200MB then you are right. It will crash anytime.

[tom13]

Your first post was mentioning 20MB! I did read it couple of times.

If it's 200MB then you are right. It will crash anytime.

Even with 20 MB it'd not be fast enough with your (30/40MB)/sec speed.

Anyhow, that is not the topic. I hope someone can help me with tail-ing.

[Zedna]

For your info: All file API functions are now (in AutoIt from 3.2.13.6 version) as standard UDFs in WinAPI include file:

Look here: http://www.autoitscript.com/forum/index.php?showtopic=76829

[tom13]

For your info: All file API functions are now (in AutoIt from 3.2.13.6 version) as standard UDFs in WinAPI include file:

Look here: http://www.autoitscript.com/forum/index.php?showtopic=76829

Thanks Zedna but is it just me or is it not possible to tail with these functions? If its not too much asked, could you maybe give a short example of how to get the last 5 lines of test.txt (or amount of changed bytes is good too)?

Edited October 12, 2008 by tom13

[Zedna]

Thanks Zedna but is it just me or is it not possible to tail with these functions? If its not too much asked, could you maybe give a short example of how to get the last 5 lines of test.txt (or amount of changed bytes is good too)?

These API functions are not for reading number of lines but for reading number of bytes!

You can read some amount of bytes from end of file and then parse it to lines by StringSplit.

Edited October 12, 2008 by Zedna

[tom13]

These API functions are not for reading number of lines but for reading number of bytes!

You can read some amount of bytes from end of file and then parse it to lines by StringSplit.

I understand, but I have no idea how to do that. The example scripts are writing to a new file and the functions like SetEndOfFile do not return the last X bytes, right?

Could you tell me or give an example of how to receive the X last byes of file.txt?

[Jos]

There is a read example too... just take the time to read it all.

example to read the last 200 bytes:

#include <WinAPI.au3>
Global $sFile, $hFile, $sText, $nBytes, $tBuffer
$sFile = 'your file name'
; read last 200 bytes
$tBuffer = DllStructCreate("byte[200]")
$hFile = _WinAPI_CreateFile($sFile, 2, 2)
_WinAPI_SetFilePointer($hFile, -200 ,2)
_WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), 200, $nBytes)
_WinAPI_CloseHandle($hFile)
$sText = BinaryToString(DllStructGetData($tBuffer, 1))
ConsoleWrite('last 200 bytes ' & $sText & @LF)

[tom13]

There is a read example too... just take the time to read it all.

example to read the last 200 bytes:

#include <WinAPI.au3>
Global $sFile, $hFile, $sText, $nBytes, $tBuffer
$sFile = 'your file name'
; read last 200 bytes
$tBuffer = DllStructCreate("byte[200]")
$hFile = _WinAPI_CreateFile($sFile, 2, 2)
_WinAPI_SetFilePointer($hFile, -200 ,2)
_WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), 200, $nBytes)
_WinAPI_CloseHandle($hFile)
$sText = BinaryToString(DllStructGetData($tBuffer, 1))
ConsoleWrite('last 200 bytes ' & $sText & @LF)

Thanks, I edited it to show the new bytes when the file gets updated:

#RequireAdmin
#include <WinAPI.au3>
Global $sFile, $hFile, $sText, $nBytes, $tBuffer
$sFile = "C:\WAR\logs\testchat.log"

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        ; read last $bytes bytes
        $bytes = $newsize - $prevsize
        $tBuffer = DllStructCreate("byte[" & $bytes & "]")
        $hFile = _WinAPI_CreateFile($sFile, 2, 2)
        _WinAPI_SetFilePointer($hFile, -$bytes, 2)
        _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), $bytes, $nBytes)
        _WinAPI_CloseHandle($hFile)
        $sText = BinaryToString(DllStructGetData($tBuffer, 1))
        $prevsize = $newsize
        MsgBox(0, "", $sText)
    EndIf
WEnd

However, it returns an empty string. Am I doing something wrong now? When I use your example I got 1 letter as return (f).

Edited October 12, 2008 by tom13

[Zedna]

I understand, but I have no idea how to do that. The example scripts are writing to a new file and the functions like SetEndOfFile do not return the last X bytes, right?

Could you tell me or give an example of how to receive the X last byes of file.txt?

#include <WinAPI.au3>

Dim $nBytes

$sFile = @ScriptDir & '\test.txt'
; read 100 bytes from end of file
$tBuffer = DLLStructCreate("byte[100]")
$hFile = _WinAPI_CreateFile($sFile, 2, 2)
$size = FileGetSize($sFile)
;~ $size = _WinAPI_GetFileSizeEx($hFile)
;ConsoleWrite('Size: ' & $size & @LF)
_WinAPI_SetFilePointer($hFile, $size - 100)
_WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), 100, $nBytes)
_WinAPI_CloseHandle($hFile)
$sText = BinaryToString(DLLStructGetData($tBuffer, 1))
ConsoleWrite('Text: ' & $sText & @LF)

[tom13]

#include <WinAPI.au3>

Dim $nBytes

$sFile = @ScriptDir & '\test.txt'
; read 100 bytes from end of file
$tBuffer = DLLStructCreate("byte[100]")
$hFile = _WinAPI_CreateFile($sFile, 2, 2)
$size = FileGetSize($sFile)
;~ $size = _WinAPI_GetFileSizeEx($hFile)
;ConsoleWrite('Size: ' & $size & @LF)
_WinAPI_SetFilePointer($hFile, $size - 100)
_WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), 100, $nBytes)
_WinAPI_CloseHandle($hFile)
$sText = BinaryToString(DLLStructGetData($tBuffer, 1))
ConsoleWrite('Text: ' & $sText & @LF)oÝ÷ Ûú®¢×©äÊ­¢î¶Æ¦zØbZ¦§²«y«0ØI¢Íý²÷«²*'N¬Ì!jÒ&i×®¦ò¢êÞÅ©©æ®¶­sb5&WV&TFÖà¢6æ6ÇVFRfÇCµväæS2fwC°¤FÒb33c¶ä'FW0¢b33c·4fÆRÒgV÷C´3¢b3#µt"b3#¶Æöw2b3#·FW7F6BæÆörgV÷C° ¢b33c·&Wg6¦RÒfÆTvWE6¦Rb33c·4fÆR¥vÆR b33c¶æWw6¦RÒfÆTvWE6¦Rb33c·4fÆR bb33c¶æWw6¦RfwC²b33c·&Wg6¦RFVà ²&VB'FW2g&öÒVæBöbfÆP b33c·D'VffW"ÒDÄÅ7G'V7D7&VFRgV÷C¶'FU²gV÷C²fײb33c¶æWw6¦RÒb33c·&Wg6¦RfײgV÷CµÒgV÷C² b33c¶fÆRÒõväô7&VFTfÆRb33c·4fÆRÂ"Â" õväõ6WDfÆUöçFW"b33c¶fÆRÂb33c¶æWw6¦RÒb33c·&Wg6¦R õväõ&VDfÆRb33c¶fÆRÂDÄÅ7G'V7DvWEG"b33c·D'VffW"Âb33c¶æWw6¦RÒb33c·&Wg6¦RÂb33c¶ä'FW2 õväô6Æ÷6TæFÆRb33c¶fÆR b33c·5FWBÒ&æ'Fõ7G&ærDÄÅ7G'V7DvWDFFb33c·D'VffW" ×6t&÷ÂgV÷C²gV÷C²Âb33c·5FWB b33c·&Wg6¦RÒb33c¶æWw6¦P VæD`¥tVæ

That should work right? But, when I type hello and safe the log file I get a MsgBox with "1", while it should be "hello".

Any idea?

[Zedna]

That should work right? But, when I type hello and safe the log file I get a MsgBox with "1", while it should be "hello".

Any idea?

instead of:

_WinAPI_SetFilePointer($hFile, $newsize - $prevsize)

try:

_WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize),2)

[tom13]

instead of:

_WinAPI_SetFilePointer($hFile, $newsize - $prevsize)

try:

_WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize),2)

Thanks.. but now I am getting those 3 strange characters again: ÿþ[

#RequireAdmin
#include <WinAPI.au3>
Dim $nBytes
$sFile = "C:\WAR\logs\testchat.log"

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        ; read 100 bytes from end of file
        $tBuffer = DLLStructCreate("byte[" & ($newsize - $prevsize) & "]")
        $hFile = _WinAPI_CreateFile($sFile, 2, 2)
        _WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize))
        _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
        _WinAPI_CloseHandle($hFile)
        $sText = BinaryToString(DLLStructGetData($tBuffer, 1))
        MsgBox(0, "", $sText)
        clipput($sText)
        $prevsize = $newsize
    EndIf
WEnd

Do you know why?

[LarryDalooza]

try changing...

DLLStructCreate("byte...

to

DLLStructCreate("char...

Lar.

and get rid of BinaryToString

Edited October 12, 2008 by LarryDalooza

[tom13]

try changing...

DLLStructCreate("byte...

to

DLLStructCreate("char...

Lar.

Thanks for the help but after changing it I still get the 3 characters (ÿþ[) instead of the text I added to the end of the log file... any idea why?

#RequireAdmin
#include <WinAPI.au3>
Dim $nBytes
$sFile = "C:\WAR\logs\testchat.log"

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        ; read 100 bytes from end of file
        $tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")
        $hFile = _WinAPI_CreateFile($sFile, 2, 2)
        _WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize))
        _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
        _WinAPI_CloseHandle($hFile)
        $sText = DLLStructGetData($tBuffer, 1)
        MsgBox(0, "", $sText)
        clipput($sText)
        $prevsize = $newsize
    EndIf
WEnd

edit; i also got rid of binary to string as you said, and I still get those 3 characters instead of the text I added to the end of the log file: ÿþ[

edit2; when I add a MsgBox(0, "", DLLStructGetData($tBuffer, 1)), it gives the same 3 characters - can you guys do anything with that information?

Edited October 12, 2008 by tom13

[Bowmore]

This works for me taking 2 to 3 ms to read 30 - 50 bytes of new data from the end of a 150Mb fil

I initially had problems getting a valid handle from _WinAPI_CreateFile() due to the app adding data having the file locked. It would seem that the size of the file is updated before the app writing to it has released the write lock.

You also need to release the buffer memory with $tBuffer = 0 or you will have a bad memory leak in a tight loop like this.

It would also appear that the help file is wrong when it says that _WinAPI_CreateFile returns 0 if unable to get a valid handle, it would seem that it returns 0xFFFFFFFF

#Include <WinAPI.au3>
$sfile = "C:\WAR\logs\testchat.log"
Dim $nBytes

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        $hFile = _WinAPI_CreateFile($sFile, 2, 2, 2)
                If $hFile <> 0xFFFFFFFF Then ;check if handle is valid; wait for other application to stop writing to file
                    ; read 100 bytes from end of file
                    $T = TimerInit()
                    $tBuffer = DLLStructCreate("byte[" & ($newsize - $prevsize) & "]")
                    _WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize), 2)
                    _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
                    _WinAPI_CloseHandle($hFile)
                    $T = TimerDiff($T)
                    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $T = ' & $T & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
                    $sText = BinaryToString(DLLStructGetData($tBuffer, 1))
                    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sText = ' & $sText & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
                    $tBuffer = 0 ;free buffer memory
                    $prevsize = $newsize
            EndIf
        EndIf
WEnd

[ken82m]

Hey tom sorry didn't check back on this. Alot of replies but looks like you might not have it the way you want yet.

Here's a working example of tail.exe I tested myself using a space in path.

#include <Process.au3>
$File = "C:\Documents and Settings\Administrator\Test.txt"

_RunDOS('Tail.exe 10 "' & $File & '">Tail.txt')

;In this exmaple tail.exe is in the script directory
;and output will dump into Tail.txt in script directory

Now that you have a much smaller file you could read it all into a string if you want:

#Include <Array.au3>
#Include <File.au3>
$FileRead = ""

_FileReadToArray("Test.txt", $FileRead)
$FileRead = _ArrayToString($FileRead)
$FileRead = StringReplace($FileRead, "|", "");If you want to get rid of the | delimeter.
MsgBox(4096,"$FileRead", $FileRead)

-Kenny

Edited October 12, 2008 by ken82m

[Zedna]

_WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize),2)

[tom13]

This works for me taking 2 to 3 ms to read 30 - 50 bytes of new data from the end of a 150Mb fil

I initially had problems getting a valid handle from _WinAPI_CreateFile() due to the app adding data having the file locked. It would seem that the size of the file is updated before the app writing to it has released the write lock.

You also need to release the buffer memory with $tBuffer = 0 or you will have a bad memory leak in a tight loop like this.

It would also appear that the help file is wrong when it says that _WinAPI_CreateFile returns 0 if unable to get a valid handle, it would seem that it returns 0xFFFFFFFF

#Include <WinAPI.au3>
$sfile = "C:\WAR\logs\testchat.log"
Dim $nBytes

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        $hFile = _WinAPI_CreateFile($sFile, 2, 2, 2)
                If $hFile <> 0xFFFFFFFF Then ;check if handle is valid; wait for other application to stop writing to file
                    ; read 100 bytes from end of file
                    $T = TimerInit()
                    $tBuffer = DLLStructCreate("byte[" & ($newsize - $prevsize) & "]")
                    _WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize), 2)
                    _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
                    _WinAPI_CloseHandle($hFile)
                    $T = TimerDiff($T)
                    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $T = ' & $T & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
                    $sText = BinaryToString(DLLStructGetData($tBuffer, 1))
                    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sText = ' & $sText & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
                    $tBuffer = 0 ;free buffer memory
                    $prevsize = $newsize
            EndIf
        EndIf
WEndoÝ÷ Ûú®¢×©ä°a¢è!´â©eˬ¶«y»­"¶¨º»®*m¶¦zËæèÆ·­º¹ìjw¦¦Ü¬¶¸§«­¢+ØIÅեɵ¥¸(%¹±Õ±Ðí]¥¹A$¹ÔÌÐì(ÀÌØíÍ¥±ôÅÕ½ÐíèÀäÈí]HÀäÈí±½ÌÀäÈíÑÍѡй±½ÅÕ½Ðì)¥´ÀÌØí¹   åÑÌ((ÀÌØíÁÉÙÍ¥éô¥±ÑM¥é ÀÌØíÍ¥±¤)]¡¥±Ä(ÀÌØí¹ÝÍ¥éô¥±ÑM¥é ÀÌØíÍ¥±¤(¥ÀÌØí¹ÝÍ¥éÐìÀÌØíÁÉÙÍ¥éÑ¡¸(ÀÌØí¡¥±ô}]¥¹A%}
ÉÑ¥± ÀÌØíÍ¥±°È°È°È¤($%%ÀÌØí¡¥±±ÐìÐìÁáQ¡¸í¡¬¥¡¹±¥ÌÙ±¥ìݥнȽѡÈÁÁ±¥Ñ¥½¸Ñ¼ÍѽÀÝɥѥ¹Ñ¼¥±($$$ìÉÄÀÀåÑÌɽ´¹½¥±($$$ÀÌØíPôQ¥µÉ%¹¥Ð ¤($$$ÀÌØíÑ  ÕÈô11MÑÉÕÑ
ÉÑ ÅÕ½ÐíåÑlÅÕ½ÐìµÀì ÀÌØí¹ÝÍ¥é´ÀÌØíÁÉÙͥ餵ÀìÅÕ½ÐítÅÕ½Ðì¤($$%}]¥¹A%}MÑ¥±A½¥¹ÑÈ ÀÌØí¡¥±°´Ä¨ ÀÌØí¹ÝÍ¥é´ÀÌØíÁÉÙͥ餰Ȥ($$%}]¥¹A%}I¥± ÀÌØí¡¥±°11MÑÉÕÑÑAÑÈ ÀÌØíÑ ÕȤ°ÀÌØí¹ÝÍ¥é´ÀÌØíÁÉÙÍ¥é°ÀÌØí¹    åÑ̤($$%}]¥¹A%}
±½Í!¹± ÀÌØí¡¥±¤($$$ÀÌØíPôQ¥µÉ¥ ÀÌØíP¤($$$í
½¹Í½±]É¥Ñ ÌäíÕ ÌäìµÀìMÉ¥ÁÑ1¥¹9յȵÀìÌäì¤èÀÌØíPôÌäìµÀìÀÌØíPµÀìɱµÀìÌäìÐíÉɽȽèÌäìµÀìÉɽȵÀìɱ¤ìÕ
½¹Í½±($$$ÀÌØíÍQáÐô ¥¹ÉåQ½MÑÉ¥¹¡11MÑÉÕÑÑÑ ÀÌØíÑ    ÕȰĤ¤($$$í
½¹Í½±]É¥Ñ ÌäíÕ ÌäìµÀìMÉ¥ÁÑ1¥¹9յȵÀìÌäì¤èÀÌØíÍQáÐôÌäìµÀìÀÌØíÍQáеÀìɱµÀìÌäìÐíÉɽȽèÌäìµÀìÉɽȵÀìɱ¤ìÕ
½¹Í½±($$%5Í   ½à À°ÅÕ½ÐìÅÕ½Ðì°ÀÌØíÍQáФ($$$ÀÌØíÑ  ÕÈôÀíÉÕȵµ½Éä($$$ÀÌØíÁÉÙÍ¥éôÀÌØí¹ÝÍ¥é($%¹%(%¹%)]¹

I don't understand why any help?

[tom13]

_WinAPI_SetFilePointer($hFile, -1 * ($newsize - $prevsize),2)

Oh, I missed that.

But, still, now I also get an empty string, just like in the above post.. I have no idea why.

[LarryDalooza]

this has some success for me...

#RequireAdmin
#include <WinAPI.au3>
Dim $nBytes
$sFile = @ScriptDir & "\test.txt"

$prevsize = FileGetSize($sFile)
While 1
    $newsize = FileGetSize($sFile)
    if $newsize > $prevsize then
        ; read 100 bytes from end of file
        $tBuffer = DLLStructCreate("char[" & ($newsize - $prevsize) & "]")
        $hFile = _WinAPI_CreateFile($sFile, 2, 2)
        If $hFile And ($hFile <> 0xFFFFFFFF) Then
            _WinAPI_SetFilePointer($hFile, $prevsize)
            _WinAPI_ReadFile($hFile, DLLStructGetPtr($tBuffer), $newsize - $prevsize, $nBytes)
            _WinAPI_CloseHandle($hFile)
            $sText = DLLStructGetData($tBuffer, 1)
            MsgBox(4096, "", $sText)
           $prevsize = $newsize
        EndIf
    EndIf
WEnd

but this will cycle crazy if it can't get the $hFile. Maybe an Else... Sleep(50) on the If $hFile .

Lar.

Edited October 12, 2008 by LarryDalooza