Read last 10 lines of text file

[maloysius]

Hi all,

 

I have been searching for like 2 hours and can't seem to quite find anything that suits my need, or that I am capable of understanding easily with my noobishness.  I am trying to open a file for reading in AutoIt that constantly is growing by an undetermined amount daily.  I want it to be able to open the file and read the last 10 lines every time, and then search those lines for a particular set of substrings.

I was messing around with this example from another post for the initial file read, but it doesn't seem to do anything:

$FilePath = "Test.txt"

$sRead = FileRead($FilePath)
$100Lines = StringTrimLeft($sRead, StringInStr($sRead, @CRLF, "", -10)+1)
ConsoleWrite($100Lines)

This is just a literal copy of another's code, I have been messing with it but I have never gotten it to even print anything at all.  I just don't even know what to do to make this work  Does anyone out there have any tips for me or a direction to point towards?

 

Thank you!

Edited April 1, 2020 by maloysius

[Danp2]

Check out _FileCountLines in the help file. Use it to get the line count, subtract 10, then use FileReadLine in a loop.

[maloysius]

Danp2, you have helped me greatly sir!  It's working now.  Thanks a lot!!

[jchd]

A faster way, by not parsing the file twice:

Local $s = FileRead("yourfile.txt")
$s = StringRegExpReplace($s, "(?ms).*?((?:^\N*(?:\R|$)){10}\z)", "$1")
ConsoleWrite($s)

 

[iamtheky]

#include<array.au3>

msgbox(0, '' , _ArrayToString(FileReadToArray("test.txt") , @LF , @extended - 10))

 

this one only pulls the last 10

$fTest = fileopen("test.txt")
$i = 10
$sOut = ""
$total = ubound(filereadtoarray($ftest)) ;_filecountlines($ftest)

do
   $sOut &= FileReadLine($fTest ,  $total - $i) & @LF
   $i -= 1
until $i = 0

fileclose($ftest)
msgbox(0, '' , $sOut)

 

Edited April 2, 2020 by iamtheky

[Zedna]

You can use this my optimization  FileReadLastChar() to read only part of file from end and parse only this part and not the whole (big?) file:

 

 

Here is simplified version using native functions:

msgbox(0, '' , FileReadLastChar('c:\test.txt' , 1024))

Func FileReadLastChar($file , $count)
    $hFile = FileOpen($file)
    $pos = FileSetPos($hFile , $count * -1 , 2)
    $sOut = FileRead($hFile)
    return $sOut
EndFunc

 

Edited April 2, 2020 by Zedna

[Nine]

And yet another way :

#include <Constants.au3>
$s = FileRead ("a.txt")
MsgBox ($MB_SYSTEMMODAL,"",StringMid($s,StringInStr($s,@CRLF,$STR_CASESENSE,-10)+2))

 

[Bitnugger]

On 4/2/2020 at 1:23 PM, Nine said:

MsgBox ($MB_SYSTEMMODAL,"",StringMid($s,StringInStr($s,@CRLF,$STR_CASESENSE,-10)+2))

If the last line ends with @CRLF, you will only see the last 9 lines with text. With the solution from @jchd, 10 lines of text are still displayed and it also works if the line breaks consist only of @LF (Unix), which is not the case with your solution.

Here's how it works better:

#include <Constants.au3>

$s = FileRead ("a.txt")
$s = StringStripWS($s, $STR_STRIPTRAILING)
$s = StringReplace($s, @CRLF, @LF)
$s = StringReplace($s, @LF, @CRLF)
ConsoleWrite(StringMid($s, StringInStr($s, @CRLF, $STR_CASESENSE, -10) + 2) & @CRLF)

... but I would rather take the solution from @jchd.

[Nine]

nvm - I reused the $s, still faster but not as much as I thought. 

#include <Constants.au3>

$s = FileRead ("a.txt")
$hTimer = TimerInit ()
$s = StringStripWS($s, $STR_STRIPTRAILING)
$s = StringReplace($s, @CRLF, @LF)
$s = StringReplace($s, @LF, @CRLF)
ConsoleWrite(StringMid($s, StringInStr($s, @CRLF, $STR_CASESENSE, -10) + 2) & @CRLF)
ConsoleWrite (TimerDiff ($hTimer) & @CRLF)

$s = FileRead ("a.txt")
$hTimer = TimerInit ()
$s = StringReplace(StringStripWS($s, $STR_STRIPTRAILING+$STR_STRIPSPACES),@LF,@CR)
ConsoleWrite (StringMid($s,StringInStr($s,@CR,$STR_CASESENSE,-10)+1) & @CRLF)
ConsoleWrite (TimerDiff ($hTimer) & @CRLF)

$s = FileRead ("a.txt")
$hTimer = TimerInit ()
$s = StringRegExpReplace($s, "(?ms).*?((?:^\N*(?:\R|$)){10}\z)", "$1")
ConsoleWrite($s)
ConsoleWrite (TimerDiff ($hTimer) & @CRLF)

 

Edited April 9, 2020 by Nine

[UEZ]

Try this:

ConsoleWrite(FileReadLastNLines("C:\Temp\swtag.log") &  @CRLF)

Func FileReadLastNLines($sFile, $iLines = 9)
    Local $hFile = FileOpen($sFile)
    If @error Then Return SetError(1, 0, 0)
    Local $sResult = FileRead($hFile)
    FileClose($hFile)
    Local $iPos = FileGetSize($sFile)
    For $i = 1 To $iLines
        $iPos = StringInStr($sResult, @LF, 1, -1, $iPos) - 1
    Next
    Return StringMid($sResult, $iPos + 2) ;assuming that @CRLF is the line break
EndFunc

 

Edited April 9, 2020 by UEZ
added Zedna's suggestion

[Zedna]

@all

For speed optimization always use CaseSensitive flag (=1) in StringInStr() and StringReplace() when searching/replacing @LF or @CRLF (or any other special characters).

The difference is really BIG, especially on BIG texts or inside loops with many repetitions!

 

StringInStr($text, @CRLF) --> StringInStr($text, @CRLF, 1)

StringReplace($text, @CRLF, @LF) --> StringReplace($text, @CRLF, @LF, 0, 1)

Edited April 9, 2020 by Zedna

[MrCreatoR]

Without reading the whole file:

#include <FileConstants.au3>

$sFile = @ScriptDir & '\MyFile.txt'
ConsoleWrite(_FileReadLinesFromEnd($sFile, 10) & @CRLF)

Func _FileReadLinesFromEnd($sFile, $iLines)
    Local $iOffset = -2, $iCount = -1, $sRet = '', $sChar
    Local $hFile = FileOpen($sFile)
    
    FileSetPos($hFile, $iOffset, $FILE_END)
    
    For $i = FileGetPos($hFile) To 0 Step -1
        $sChar = FileRead($hFile, 2)
        $sRet = $sChar & $sRet
        
        If StringInStr($sChar, @LF, 2) Then
            $iCount += 1
            
            If $iCount = $iLines Then
                ExitLoop
            EndIf
        EndIf
        
        $iOffset -= 2
        FileSetPos($hFile, $iOffset, $FILE_END)
    Next
    
    FileClose($hFile)
    Return StringStripWS($sRet, 3)
EndFunc

 

[Bitnugger]

Here is my last version ... copied from @UEZ ... so far the fastest version.

#include <Constants.au3>

$sFile = "f:\Eigene Dateien\TxT\Windows Message Codes.txt"

$hTimer = TimerInit()
ConsoleWrite(FileReadLastNLines($sFile, 10))
ConsoleWrite("> Time: " & TimerDiff($hTimer) & @CRLF)

Func FileReadLastNLines($sFile, $iLines = 9)
    Local $sResult = FileRead($sFile), $iFileSize = @extended
    If @error Then Return SetError(@error, 0, "")
    $sResult = StringStripWS($sResult, $STR_STRIPTRAILING)
    Return StringMid($sResult, StringInStr($sResult, @LF, $STR_CASESENSE, - $iLines, $iFileSize) + 1) & @CRLF
EndFunc

> Time: 0.7427

Edited April 10, 2020 by Bitnugger

[Zedna]

Here is my modification using my previous mentioned FileReadLastChars() to avoid reading of whole content of file into memory

which is important for efficiency on large files (megabytes)

#include <Constants.au3>

$sFile = "c:\test.txt"

$hTimer = TimerInit()
ConsoleWrite(FileReadLastNLines($sFile, 10))
ConsoleWrite("> Time: " & TimerDiff($hTimer) & @CRLF)

Func FileReadLastNLines($sFile, $iLines = 10, $iMaxRowLen = 1000)
    Local $sResult = FileReadLastChars($sFile, $iLines * ($iMaxRowLen+2))
    $sResult = StringStripWS($sResult, $STR_STRIPTRAILING)
    Return StringMid($sResult, StringInStr($sResult, @LF, $STR_CASESENSE, - $iLines) + 1) & @CRLF
EndFunc

Func FileReadLastChars($file, $count)
    $hFile = FileOpen($file)
    $pos = FileSetPos($hFile , -1 * $count, 2)
    $ret = FileRead($hFile)
    FileClose($hFile)
    Return $ret
EndFunc

 

[UEZ]

I would suggest to use both ways depending on file size. Of course it depends also on the lines you want read.

Best it to make some test when which function is faster...

[MrCreatoR]

expandcollapse popup

#include <FileConstants.au3>
#include <StringConstants.au3>

Global $sFile = @TempDir & '\LinesTest.tmp'

If Not FileExists($sFile) Then
    $sLines = ''
    
    For $i = 1 To 10000
        $sLines &= 'Line #' & $i & @CRLF
    Next
    
    $hFile = FileOpen($sFile, 2)
    FileWrite($hFile, $sLines)
    FileClose($hFile)
EndIf

;Call the functions first to avoid first run difference
_Test_FileReadLinesFromEnd()
_Test_FileReadLastNLines()
_Test_FileReadLastNLines2()

_SpeedTest('_Test_FileReadLinesFromEnd', 'By MrCreatoR')
_SpeedTest('_Test_FileReadLastNLines', 'By Zedna')
_SpeedTest('_Test_FileReadLastNLines2', 'By Zedna')

Func _SpeedTest($sTestFunc, $sTitle = 'Testing', $iRepeat_Test = 5)
    Local $aTests[$iRepeat_Test + 1] = [$iRepeat_Test]
    Local $iTotal_Tests = 0
    
    ConsoleWrite(StringFormat('+ Tests for "Func %s" (%s) started...\n=--------------------=\n', $sTestFunc, $sTitle))
    
    Local $iDuration_Timer = TimerInit()
    
    For $x = 1 To $iRepeat_Test
        $iTimer = TimerInit()
        
        ;=== CODE TEST ====
        Call($sTestFunc)
        ;=== CODE TEST ===
        
        $aTests[$x] = Round(TimerDiff($iTimer), 3)
        $iTotal_Tests += $aTests[$x]
        ConsoleWrite(StringFormat('- Test #%i: %.2f ms\n', $x, $aTests[$x]))
    Next
    
    Local $iAverage = Round($iTotal_Tests / $iRepeat_Test, 3)
    ConsoleWrite(StringFormat('======================\n! Tests average: %.2f ms\n+ Test Duration: %.2f ms\n\n', $iAverage, TimerDiff($iDuration_Timer)))
EndFunc

Func _Test_FileReadLinesFromEnd()
    _FileReadLinesFromEnd($sFile, 10)
EndFunc

Func _Test_FileReadLastNLines()
    FileReadLastNLines($sFile, 10, 2000)
EndFunc

Func _Test_FileReadLastNLines2()
    FileReadLastNLines2($sFile, 10)
EndFunc

Func _FileReadLinesFromEnd($sFile, $iLines)
    Local $iOffset = -2, $iCount = -1, $sRet = '', $sChar
    Local $hFile = FileOpen($sFile)
    
    FileSetPos($hFile, $iOffset, $FILE_END)
    
    For $i = FileGetPos($hFile) To 0 Step -1
        $sChar = FileRead($hFile, 2)
        $sRet = $sChar & $sRet
        
        If StringInStr($sChar, @LF, 2) Then
            $iCount += 1
            
            If $iCount = $iLines Then
                ExitLoop
            EndIf
        EndIf
        
        $iOffset -= 2
        FileSetPos($hFile, $iOffset, $FILE_END)
    Next
    
    FileClose($hFile)
    Return StringStripWS($sRet, 3)
EndFunc

Func FileReadLastNLines($sFile, $iLines = 10, $iMaxRowLen = 1000)
    Local $sResult = FileReadLastChars($sFile, $iLines * ($iMaxRowLen+2))
    $sResult = StringStripWS($sResult, $STR_STRIPTRAILING)
    Return StringMid($sResult, StringInStr($sResult, @LF, $STR_CASESENSE, - $iLines) + 1) & @CRLF
EndFunc

Func FileReadLastChars($file, $count)
    $hFile = FileOpen($file)
    $pos = FileSetPos($hFile , -1 * $count, 2)
    $ret = FileRead($hFile)
    FileClose($hFile)
    Return $ret
EndFunc

Func FileReadLastNLines2($sFile, $iLines = 9)
    Local $sResult = FileRead($sFile), $iFileSize = @extended
    If @error Then Return SetError(@error, 0, "")
    $sResult = StringStripWS($sResult, $STR_STRIPTRAILING)
    Return StringMid($sResult, StringInStr($sResult, @LF, $STR_CASESENSE, - $iLines, $iFileSize) + 1) & @CRLF
EndFunc

This speed test results shows (on my machine) that Zenda's FileReadLastNLines with $iMaxRowLen = 2000 even faster then the others.

[UEZ]

I can confirm that Zedna's version is the best for small and large files! For large file reading takes too long.

[MrCreatoR]

It's faster, but problematic if the line length is unknown.

My version is not much slower but not depends on line length.

Edited April 10, 2020 by MrCreatoR

[MrCreatoR]

Ok, here is new version, twice faster!

 

expandcollapse popup

#include <FileConstants.au3>
#include <StringConstants.au3>

Global $sFile = @TempDir & '\LinesTest.tmp'

If Not FileExists($sFile) Then
    $sLines = ''
    
    For $i = 1 To 10000
        $sLines &= 'Line #' & $i & @CRLF
    Next
    
    $hFile = FileOpen($sFile, 2)
    FileWrite($hFile, $sLines)
    FileClose($hFile)
EndIf

;Call the functions first to avoid first run difference
_Test_FileReadLinesFromEnd()
_Test_FileReadLastNLines()

_SpeedTest('_Test_FileReadLinesFromEnd', 'By MrCreatoR')
_SpeedTest('_Test_FileReadLastNLines', 'By Zedna')

Func _SpeedTest($sTestFunc, $sTitle = 'Testing', $iRepeat_Test = 5)
    Local $aTests[$iRepeat_Test + 1] = [$iRepeat_Test]
    Local $iTotal_Tests = 0
    
    ConsoleWrite(StringFormat('+ Tests for "Func %s" (%s) started...\n=--------------------=\n', $sTestFunc, $sTitle))
    
    Local $iDuration_Timer = TimerInit()
    
    For $x = 1 To $iRepeat_Test
        $iTimer = TimerInit()
        
        ;=== CODE TEST ====
        Call($sTestFunc)
        ;=== CODE TEST ===
        
        $aTests[$x] = Round(TimerDiff($iTimer), 3)
        $iTotal_Tests += $aTests[$x]
        ConsoleWrite(StringFormat('- Test #%i: %.2f ms\n', $x, $aTests[$x]))
    Next
    
    Local $iAverage = Round($iTotal_Tests / $iRepeat_Test, 3)
    ConsoleWrite(StringFormat('======================\n! Tests average: %.2f ms\n+ Test Duration: %.2f ms\n\n', $iAverage, TimerDiff($iDuration_Timer)))
EndFunc

Func _Test_FileReadLinesFromEnd()
    _FileReadLinesFromEnd($sFile, 10)
EndFunc

Func _Test_FileReadLastNLines()
    FileReadLastNLines($sFile, 10, 2000)
EndFunc

Func _FileReadLinesFromEnd($sFile, $iLines)
    Local $iOffset = -500, $iCount = -1, $sRet = '', $sChar
    Local $hFile = FileOpen($sFile)
    
    FileSetPos($hFile, $iOffset, $FILE_END)
    
    For $i = FileGetPos($hFile) To 0 Step -1
        $sChars = FileRead($hFile, 500)
        $sRet = $sChars & $sRet
        
        StringReplace($sChars, @LF, '', 0, $STR_CASESENSE)
        
        If @extended >= $iLines Then
            ExitLoop
        EndIf
        
        $iOffset -= 500
        FileSetPos($hFile, $iOffset, $FILE_END)
    Next
    
    FileClose($hFile)
    
    $sRet = StringStripWS($sRet, $STR_STRIPTRAILING)
    Return StringMid($sRet, StringInStr($sRet, @LF, $STR_CASESENSE, - $iLines) + 1)
EndFunc

Func FileReadLastNLines($sFile, $iLines = 10, $iMaxRowLen = 1000)
    Local $sResult = FileReadLastChars($sFile, $iLines * ($iMaxRowLen+2))
    $sResult = StringStripWS($sResult, $STR_STRIPTRAILING)
    Return StringMid($sResult, StringInStr($sResult, @LF, $STR_CASESENSE, - $iLines) + 1) & @CRLF
EndFunc

Func FileReadLastChars($file, $count)
    $hFile = FileOpen($file)
    $pos = FileSetPos($hFile , -1 * $count, 2)
    $ret = FileRead($hFile)
    FileClose($hFile)
    Return $ret
EndFunc