Real-Time log file viewer

[jguinch]

Here is a small application made for my own need.
It allows you to follow the tail of a log file (or other txt file) in real time.
To avoid some memory issues, it loads about 10MB from the end of the file, so you can follow your huge log files in real-time with it.

Here is the code :

expandcollapse popup

#include <FileConstants.au3>
#include <GUIConstantsEx.au3>
#include <GuiEdit.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>

Global $sAppTitle = "RT-LogFileViewer"
Global $iSizeToLoad = 10 * 1024 * 1024
Global $iBlockSize = 1024, $iPause = False
Global $sLastDirectory = RegRead("HKEY_CURRENT_USER\Software\" & $sAppTitle, "LastDirectory")
Global $iFileChange = 0, $sFileDate, $sLogFile, $iFileSize, $hGui
Global $idEdit, $idPause, $idSearch, $idFileName, $idProgress

If Not StringLen($sLastDirectory) Or Not FileExists($sLastDirectory) Then $sLastDirectory = @MyDocumentsDir
Opt("GUIOnEventMode", 1)
Opt("MustDeclareVars", 1)

_LoadGui()
If $CmdLine[0] Then _FileOpen($CmdLine[1])
ProcessWaitClose(@AutoItPID)


Func _LoadGui()
    Local $idOpen, $idAbout
    $hGui = GUICreate($sAppTitle, 800, 600, -1, -1, BitOR($WS_SYSMENU, $WS_SIZEBOX, $WS_MINIMIZEBOX, $WS_MAXIMIZEBOX) )
    $idEdit      = GUICtrlCreateEdit(""            , 10 , 50 , 780, 490, BitOr($ES_READONLY, $ES_AUTOVSCROLL, $WS_VSCROLL, $WS_HSCROLL, $ES_NOHIDESEL))
    $idOpen      = GUICtrlCreateButton(ChrW(0x2f)  , 10 , 10 , 30 , 30)
    $idPause     = GUICtrlCreateButton(ChrW(0x3b)  , 760, 10 , 30 , 30)
    $idSearch    = GUICtrlCreateButton(ChrW(0x2315), 720, 10 , 30 , 30)
    $idFileName  = GUICtrlCreateLabel (""          , 10 , 550, 680, 25)
    $idAbout     = GUICtrlCreateLabel ("About"     , 690, 550, 100, 25, $SS_RIGHT)
    $idProgress  = GUICtrlCreateProgress(10, 540, 780, 5)
    Local $aAccelKeys = [["^o", $idOpen], ["^p", $idPause], ["^f", $idSearch]]

    GUICtrlSetFont($idOpen    , 12, 400, 0, "Wingdings 2")
    GUICtrlSetFont($idPause   , 12, 400, 0, "Webdings")
    GUICtrlSetFont($idSearch  , 19, 800, 0, "Arial")
    GUICtrlSetFont($idEdit    , 10, 400, 0, "Courier")
    GUICtrlSetFont($idFileName, 9 , 400, 0, "Verdana")
    GUICtrlSetFont($idAbout   , 9 , 400, 4, "Verdana")
    GUICtrlSetTip($idOpen  , "Open file (CTRL + O)")
    GUICtrlSetTip($idSearch, "Search... (CTRL + F)")
    GUICtrlSetTip($idPause , "Pause monitoring (CTRL + P)")
    GUICtrlSendMsg($idEdit, $EM_LIMITTEXT, -1, 0)
    GUICtrlSetColor($idAbout, 0x0000ff)

    GUISetOnEvent    ($GUI_EVENT_CLOSE, "_Exit")
    GUICtrlSetOnEvent($idPause        , "_Pause")
    GUICtrlSetOnEvent($idSearch       , "_Search")
    GUICtrlSetOnEvent($idOpen         , "_FileOpenGUI")
    GUICtrlSetOnEvent($idAbout        , "_About")

    GUICtrlSetResizing($idOpen, $GUI_DOCKALL)
    GUICtrlSetResizing($idPause, $GUI_DOCKRIGHT + $GUI_DOCKTOP + $GUI_DOCKSIZE)
    GUICtrlSetResizing($idSearch, $GUI_DOCKRIGHT + $GUI_DOCKTOP + $GUI_DOCKSIZE)
    GUICtrlSetResizing($idEdit, $GUI_DOCKLEFT + $GUI_DOCKRIGHT + $GUI_DOCKTOP + $GUI_DOCKBOTTOM)
    GUICtrlSetResizing($idFileName, $GUI_DOCKLEFT + $GUI_DOCKRIGHT + $GUI_DOCKBOTTOM + $GUI_DOCKHEIGHT)
    GUICtrlSetResizing($idAbout, $GUI_DOCKRIGHT + $GUI_DOCKBOTTOM + $GUI_DOCKSIZE)
    GUICtrlSetResizing($idProgress, $GUI_DOCKLEFT + $GUI_DOCKRIGHT + $GUI_DOCKBOTTOM + $GUI_DOCKHEIGHT)
    GUICtrlSetState($idSearch, $GUI_DISABLE)
    GUICtrlSetState($idPause, $GUI_DISABLE)
    GUICtrlSetState($idProgress, $GUI_HIDE)
    GUISetAccelerators($aAccelKeys, $hGui)
    GUISetIcon(@SystemDir & "\SHELL32.dll", -25)
    GUISetState(@SW_SHOW)
EndFunc

Func _LoadTail($sLogFile)
    Local $sLines, $iPercent
    $iFileSize = FileGetSize($sLogFile)
    Local $hFile = FileOpen($sLogFile, $FO_READ)
    If @error Then Return 0
    $sFileDate = FileGetTime($sLogFile, 0, 1)

    Local $iStartPos =($iSizeToLoad > $iFileSize ? 0 : $iFileSize - $iSizeToLoad)
    GUICtrlSetData($idProgress, 0)
    GUICtrlSetState($idProgress, $GUI_SHOW)
    For $i = $iStartPos To $iFileSize - $iBlockSize Step $iBlockSize
        $iPercent = Int( $i * 100 / $iFileSize)
        GUICtrlSetData($idProgress, $iPercent)
        FileSetPos($hFile, $i, 0)
        $sLines &= FileRead($hFile, $iBlockSize)
        If $i = $iStartPos Then $sLines = StringRegExpReplace($sLines, "^\V*\R", "", 1)
    Next
    $sLines &= FileRead($hFile)
    GUICtrlSetData($idProgress, 100)
    Sleep(100)
    GUICtrlSetState($idProgress, $GUI_HIDE)

    _GUICtrlEdit_AppendText($idEdit, $sLines)
    FileClose($hFile)
    Return 1
EndFunc

Func _Search()
    _GUICtrlEdit_Find($idEdit)
EndFunc

Func _FileOpenGUI()
    _FileOpen()
EndFunc

Func _FileOpen($sFileToOpen = "")
    If Not FileExists($sFileToOpen) Then $sFileToOpen = FileOpenDialog("Select a file", $sLastDirectory, "Log File (*.log)|All (*.*)", 1, "", $hGui)
    If @error Then Return
    If $sFileToOpen = $sLogFile Then
        If not $iFileChange Then Return
        $iFileChange = 0
    EndIf
    GUICtrlSetData($idEdit, "")
    GUICtrlSetState($idPause, $GUI_ENABLE)
    If Not $iPause Then _Pause()
    If _LoadTail($sFileToOpen) Then
        $sLogFile = $sFileToOpen
        GUICtrlSetData($idFileName, $sLogFile)
        $sLastDirectory = StringRegExpReplace($sLogFile, "\\[^\\]+$", "")
        RegWrite("HKEY_CURRENT_USER\Software\" & $sAppTitle, "LastDirectory", "REG_SZ", $sLastDirectory)
    Else
        MsgBox(16, $sAppTitle & " - Error", "Unable to load the specified file.")
    EndIf
    _Pause()
EndFunc

Func _Refresh()
    Local $sLines, $hFile
    Local $iNewSize = FileGetSize($sLogFile)
    Local $sNewFileDate = FileGetTime($sLogFile, 0, 1)
    If $sNewFileDate = $sFileDate Then Return
    $sFileDate = $sNewFileDate

    If $iNewSize > $iFileSize Then
        $hFile = FileOpen($sLogFile, $FO_READ)
        FileSetPos($hFile, $iFileSize, 0)
        $sLines = FileRead($hFile, $iNewSize - $iFileSize)
        FileClose($hFile)
        $iFileSize = $iNewSize
        _GUICtrlEdit_AppendText($idEdit, $sLines)
    Else
        $iFileChange = 1
        _FileOpen($sLogFile)
        Return
    EndIf
EndFunc

Func _Pause()
    $iPause = Not $iPause
    If $iPause Then
        GUICtrlSetData($idPause, ChrW(0x34))
        GUICtrlSetState($idSearch, $GUI_ENABLE)
        AdlibUnRegister("_Refresh")
    Else
        GUICtrlSetData($idPause, ChrW(0x3b))
        GUICtrlSetState($idSearch, $GUI_DISABLE)
        AdlibRegister("_Refresh")
    EndIf
EndFunc

Func _About()
    Local $sAboutText = $sAppTitle & " is a real-time log file viewer. It allows you to monitor huge files by showing you the end of the file (not the whole file)." & @CRLF & _
                                     "Thanks to use it !" & @CRLF & @CRLF & _
                                     "Made with AutoIt version " & @AutoItVersion & " by JGUINCH."
    MsgBox(0, $sAppTitle, $sAboutText, 0, $hGui)
EndFunc

Func _Exit()
    Exit
EndFunc

 

 

Edited September 8, 2020 by jguinch
code update

[JLogan3o13]

Very nice

[Gianni]

Nice idea!

5 hours ago, jguinch said:

It allows you to follow the tail of a log file (or other txt file) in real time.

...it must be said, however, that if the file is modified not by 'appending' new lines, but by modifying the contents of the lines inside the file or also adding new lines in the body of the file (instead of appending new lines as normally happens in a log file) the result displayed in the log viewer is a bit "garbled" and does not match the contents of the monitored file.

 

Edited September 7, 2020 by Chimp

[jguinch]

Thanks guys.

@Chimp : you're right. I just edit the code. Now, it can detect that the content has changed (it reloads the end of the file)

 

[jguinch]

Small update :
 - 10MB of data are read at the file opening, instead of 1000 lines
 - a progress bar appears when the file is loading

 

 

 

[Gianni]

On 9/7/2020 at 5:15 PM, jguinch said:

Thanks guys.

@Chimp : you're right. I just edit the code. Now, it can detect that the content has changed (it reloads the end of the file)

 

Hi @jguinch ,thanks for the update,

... but there are still some imperfections ....
i tried to do the following to check:
saved this small text file and opened in notepad; also opened with your script so to track it.

begin
1
2
3
The end

I put the two windows next to each other and made a change to a line of the text in notepad, for example I changed the line with the number 3 to 333.
Saving the file in notepad, we see that the monitored text also changes but does not match. If you then simply save it a second time without any further changes then it falls into place (in the latter case the script only detects the file date change).
So wouldn't it be better to just check the file date and update the tracking if the date has changed without running the other file length checks?
sorry for the hassle ...

[jguinch]

@Chimp Thanks for your keedback, you're right.
The only way will be to reload the end of the file each time the file changes. I want to avoid it.
I tried with CMTrace (tool from Configuration Manager tools) to see its behaviour , and it's the same result.

So, for the moment, the tool stay as it, since a log file is usually not used as you describe. Maybe an option will be added for the refresh behaviour...

 

 

 

 

[Earthshine]

Isn’t there some free kind of tail DLL out there you could use? This is a perfect use for the tail command from Linux or UNIX

[Earthshine]

Oh you can use power shell to do tail

 

Get-Content filenamehere -Wait -Tail 30

[ur]

There is a Microsoft free tool also, CMTrace

 

 

[jguinch]

@ur : yes , I know. But CMTrace is very long to load huge files. It's for this reason I created this tool

 

 

 

[Earthshine]

That’s why you should use tail instead

[ptrex]

indeed the fast and easy way is to use PS 

Quote

Get-content "C:\pfirewall.log" -Tail 10

You can use the WAIT parameter to only show the new lines added to the log file for montoring

Quote

Get-Content -Path "C:\pfirewall.log" -Wait

If you want the Gui version of FileOpenDialogue

CLS

[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") | Out-Null

$dlg = New-Object System.Windows.Forms.OpenFileDialog

$dlg.initialDirectory = """ + initialDir + """
$dlg.filter = "ZIP files|*.zip|Text Documents|*.txt|Shell Scripts|*.*sh|All Files|*.*"
$dlg.FilterIndex = 4
$dlg.Title = 'Select a file to upload'
$dlg.ShowHelp = $True
$dlg.ShowDialog() | Out-Null
$dlg.FileName

Get-content $dlg.FileName -Tail 10

Enjoy !

[jguinch]

Ah, sorry guys, I thought we were on the Autoit forum...

[Earthshine]

oh well, have at it.

 

maybe you could use Powershell to do the tail and get the output to your gui

 

I would use 

Get-Content D:\log.txt -Tail 10 –Wait

 

to monitor the log and put that in your gui

Edited September 14, 2020 by Earthshine

[ptrex]

@jguinch   

Oh yes I forget it is an autoit forum.

Here you go 😉

 

expandcollapse popup

#AutoIt3Wrapper_UseX64=y

#include <MsgBoxConstants.au3>
#include "C:\CLR.au3"
#include <Clipboard.au3>

Local $PS_Script = " "

_Run_PSHost_Script('Get-content "C:\pfirewall.log" -Tail 10', 1)

Func _Run_PSHost_Script($PSScript, $iOutput = 0)
    Local $oAssembly = _CLR_LoadLibrary("System.Management.Automation")
    ConsoleWrite("!$oAssembly: " & IsObj($oAssembly) & @CRLF)

    ; Get Type
    Local $pAssemblyType = 0
    $oAssembly.GetType_2("System.Management.Automation.PowerShell", $pAssemblyType)
    ConsoleWrite("$pAssemblyType = " & Ptr($pAssemblyType) & @CRLF)

    Local $oActivatorType = ObjCreateInterface($pAssemblyType, $sIID_IType, $sTag_IType)
    ConsoleWrite("IsObj( $oAssemblyType ) = " & IsObj($oActivatorType) & @TAB & @CRLF)

    ; Create Object
    Local $pObjectPS = 0
    $oActivatorType.InvokeMember_3("Create",0x158, 0, 0, 0, $pObjectPS)
    ConsoleWrite("IsObject: " & IsObj($pObjectPS) & @TAB & "$pObject: " & ObjName($pObjectPS) & @CRLF)

; ------------- CONSOLE ---------------
    Local $oAssembly1 = _CLR_LoadLibrary("mscorlib")
    ConsoleWrite("!$oAssembly: " & IsObj($oAssembly1) & @CRLF)

    ; Get Type
    Local $pAssemblyType1 = 0
    $oAssembly1.GetType_2("System.Console", $pAssemblyType1)
    ConsoleWrite("$oAssembly1 = " & Ptr($oAssembly1) & @CRLF)

    Local $oActivatorType1 = ObjCreateInterface($pAssemblyType1, $sIID_IType, $sTag_IType)
    ConsoleWrite("IsObj( $oActivatorType1 ) = " & IsObj($oActivatorType1) & @TAB & @CRLF)

    ; Create Object
    Local $pObjectPS1 = 0
    Local $sText[] = [@CRLF & "AutoIT Rocks !!" & @CRLF & @CRLF]
    $oActivatorType1.InvokeMember_3("Write", 0x158, 0, 0, CreateSafeArray($sText), $pObjectPS1)
;~     ConsoleWrite("$pObjectPS1: " & IsObj($pObjectPS1) & @TAB & "$pObjectPS1: " & ObjName($pObjectPS1) & @CRLF)

; <<<<<<<<<<<<<<<<<<< PS COMMAND HERE >>>>>>>>>>>>>>>>>>>>

    $pObjectPS.AddScript($PSScript) ; Add Script here

; <<<<<<<<<<<<<<<<<<< Output >>>>>>>>>>>>>>>>>>>>
        Switch $iOutput
            Case 0
                $pObjectPS.AddCommand("Clip")
            Case 1
                $pObjectPS.AddCommand("Out-GridView")

            Case 2
                $pObjectPS.AddCommand("Out-Printer")
            Case 3
                $pObjectPS.AddCommand("Out-File")
                    $sFile = @DesktopDir & "\PShost.txt"
                    ConsoleWrite("> " & $sFile & @CRLF)
                $pObjectPS.AddArgument($sFile)
            Case 4
                $pObjectPS.AddCommand("Out-Null")
            Case Else
                MsgBox(0,"PSHost","Wrong Output Choice ?")
        EndSwitch

    $objAsync = $pObjectPS.BeginInvoke()
    ConsoleWrite("$objAsync " & IsObj($objAsync & @TAB & "$pObject: " & ObjName($objAsync) ) & @CRLF)


        While $objAsync.IsCompleted = False
            ContinueLoop
        WEnd
            ConsoleWrite("Completed : " & $objAsync.IsCompleted & @CRLF)

    $objPsCollection = $pObjectPS.EndInvoke($objAsync)

;============================================================
        Switch $iOutput
            Case 0
                MsgBox(0,"PSHost",_ClipBoard_GetData())
                ClipPut("")
                _ClipBoard_Close()
            Case 1
                WinWaitClose("")
            Case 3
                MsgBox(0,"PSHost","File Output Ready on Desktop.")
        EndSwitch
EndFunc

Enjoy !!

[Earthshine]

that is brilliant @ptrex! nice way to get ps output into a nice gui
 

I was trying to write up a script just now to see if I could do it, but you knocked the ball out of the park with this. I need to test this on a huge log file being appended to but I have no HUGE files to test.

i just tested your code on a build that I do and you even have FILTERS!! woot, this is awesome.

Edited September 14, 2020 by Earthshine

[jguinch]

Nice answer , @ptrex,

 

 

[ptrex]

You're welcome.

All credits go to the forum members that are far more gifted than me.

I just put the pieces together.

I am quit confident that performance will work great on huge files. Since the PS will run in a separate x64 process than Autoit.

Let the world know how the testing went.

Enjoy !

[VAN0]

  

On 9/14/2020 at 2:43 PM, ptrex said:

@jguinch   

Oh yes I forget it is an autoit forum.

Here you go 😉

 

Enjoy !!

 

How to get the data as an array without using clipboard or saving into a file? (I'd like to convert the data into a database so it can be filtered/sorted/searched/color formatted, etc)

Edited November 18, 2021 by VAN0