Fade91 Posted August 23, 2015 Posted August 23, 2015 Hello, first time poster here I am working on a project that has to parse a log file in real time. The thing is I know it's hard for Autoit to attach itself to log files when they're already in use by other programs, at least in my experience.I was taking a look at this thread because the log file is quite large and I think Autoit might be a little slow on it's own.The thing is I don't know how to use this properly to extract all data out of a log file or is there a native way to do this using Autoit.Basically , I just need a log parser that is able to read from a log that is 'already opened' and 'being written to' Thanks!
iamtheky Posted August 23, 2015 Posted August 23, 2015 I do not believe that AutoIt has any difficulties, the event log UDF reads from a log that is both already opened and being written to. Since you have told us pretty much zero about the app or file lets begin: when the app is running can you do a simple fileread("logfile.txt")? ,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-. |(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/ (_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_) | | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) ( | | | | |)| | \ / | | | | | |)| | `--. | |) \ | | `-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_| '-' '-' (__) (__) (_) (__)
Fade91 Posted August 23, 2015 Author Posted August 23, 2015 I actually haven't really written any code other than grab the File(Open,Read,Close) functions from the help file and changed the path to where my log is. I just tried it and the read does work when the application is running.What I really would like is to be able to read each "new line" and do something and then move on to the next line.I'm not quite sure how to start though.
iamtheky Posted August 23, 2015 Posted August 23, 2015 (edited) there are a couple hundred ways you could do this, here is my thought: Read to array, check the size, if it is bigger than before then read the last line. If that works you can start getting sexy and return all the lines that are new, since you have both the previous and current size. Something like this pseudo......$oldnumberoflines = 0 While 1 $array = _FileReadToArray("logfile.log") $newnumberoflines = ubound($array) - 1 If $newnumberoflines > $oldnumberoflines Then msgbox(0, '' , $array[ubound($array) - 1]) $newnumberoflines = $oldnumberoflines sleep (5000) ; check every 5 seconds wend Edited August 23, 2015 by boththose Fade91 1 ,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-. |(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/ (_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_) | | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) ( | | | | |)| | \ / | | | | | |)| | `--. | |) \ | | `-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_| '-' '-' (__) (__) (_) (__)
Fade91 Posted August 23, 2015 Author Posted August 23, 2015 (edited) #include <File.au3> #include <Array.au3> #include <MsgBoxConstants.au3> Global $Index = 0 $Path = "output_log.txt" $Count_Log = _FileCountLines($Path) While 1 $Array = _FileReadToArray($Path, $Index, 0) $Recount_Log = UBound($Index) - 1 If $Recount_Log > $Count_Log Then ToolTip($Index[UBound($Index) - 1], 0, 0) ;If $Recount_Log > $Count_Log Then MsgBox(0, '' , $Index[UBound($Index) - 1]) $Recount_Log = $Count_Log Sleep(250) WEndSo far this works great! Thank you! Edited August 23, 2015 by Fade91
iamtheky Posted August 23, 2015 Posted August 23, 2015 (edited) If you need to do this on a log of any decent size, or if you really only want the last line: you may want to consider filereadline($log , -1), as checking that one string will be much less of a penalty than the readtoarray and ubound check. Edited August 23, 2015 by boththose Fade91 1 ,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-. |(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/ (_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_) | | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) ( | | | | |)| | \ / | | | | | |)| | `--. | |) \ | | `-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_| '-' '-' (__) (__) (_) (__)
Fade91 Posted August 23, 2015 Author Posted August 23, 2015 The file is written continuously and my script will perform actions on the most recent action it logs. Which means it will have to pause the script and continue on from where it left off afterwards. So reading just the last line wouldn't work all that well but may be if I were to get the line number and store it and continue on from there, that might just work.Basically the script is taking this log information and projecting it on my monitor in the form of a GUI and when different toggles are on/off, it shows the user (me)I'm not sure how well this is going to work out though because once it reads the last piece of the log it has to decode it and show it in the GUI and continue from where it left off, which is no big deal but the GUI is suppose to be active while its looping through the logs. So it's like I almost need two scripts communicating back and forth or something.It seems like quite the project to be taking on but once I get the reading of the log finalized the GUI part is easier but its getting them to work at the same time that is going to be a bit challenging for me.
iamtheky Posted August 24, 2015 Posted August 24, 2015 Sure, but it would still be a quicker first test in the current loop, because if that string matches then you could just sleep and check again. and if it doesnt match then you can read the file to array without a need to test the ubound, and in the gui display all the lines between $Recount_log and ubound($array) - 1. ,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-. |(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/ (_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_) | | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) ( | | | | |)| | \ / | | | | | |)| | `--. | |) \ | | `-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_| '-' '-' (__) (__) (_) (__)
Fade91 Posted August 24, 2015 Author Posted August 24, 2015 (edited) I will experiment trying different things and see how far I get and if I need more help I will post here , I really appreciate your help! Edited August 24, 2015 by Fade91
kylomas Posted August 24, 2015 Posted August 24, 2015 Fade91,The following is an example of how to return the unread portion of a file from scan to scan...expandcollapse popup; *** Start added by AutoIt3Wrapper *** #include <EditConstants.au3> #include <GUIConstantsEx.au3> ; *** End added by AutoIt3Wrapper *** #include <array.au3> #include <date.au3> #include <File.au3> Local $file_monitored = @ScriptDir & '\log.log' ; file to monitor Local $monitor_interval = .25 ; minute(s) between file check Local $bMonitor = False ; switch to start monitor #AutoIt3Wrapper_Add_Constants=n Local $gui010 = GUICreate('Log Monitor') Local $aSize = WinGetClientSize($gui010) Local $lbl010 = GUICtrlCreateLabel('Log', 10, 10, 50, 20) Local $log010 = GUICtrlCreateEdit('', 10, 30, $aSize[0] - 20, $aSize[0] - 50, $es_readonly) GUICtrlSetFont($log010, 7, 800, -1, 'lucinda console') GUISetState() ; routine to start scan AdlibRegister('_start_monitor', $monitor_interval * 60 * 1000) While 1 Switch GUIGetMsg() Case $gui_event_close Exit EndSwitch If $bMonitor = True Then _scan_file() $bMonitor = False EndIf WEnd ; scan routine Func _scan_file() Local $sTMP = '' Local Static $FilePos = 0 ;read log file entries added since the last scan Local $fl = FileOpen($file_monitored) If $fl = -1 Then Exit MsgBox($MB_ICONERROR, '', 'File open failed') ; set file position to last position read If Not FileSetPos($fl, $FilePos, 0) Then _ Exit MsgBox(0, 'ERROR', 'FileSetPos failed') ; read file from last position read to the end Local $sLogEntries = FileRead($fl) ; save file position for the next loop $FilePos = FileGetPos($fl) FileClose($fl) ; do whatever you want with the log entries...I am just writing them to the gui GUICtrlSetData($log010, 'Log Scan at ' & _Now() & @CRLF & $sLogEntries, 1) If StringRight(GUICtrlRead($log010), 2) <> @CRLF Then GUICtrlSetData($log010, @CRLF, 1) EndFunc ;==>_scan_file ; adlib to set switch that initiates scan Func _start_monitor() $bMonitor = True EndFunc ;==>_start_monitorNote - 1) If lines are deleted and subsequently populated that range will not be returned, although this could be coded for. 2) The file position is only valid while the script is running. You can probably externalize this value, if needed.Cursory comments are included, please ask if you have questions.kylomas Fade91 1 Forum Rules Procedure for posting code "I like pigs. Dogs look up to us. Cats look down on us. Pigs treat us as equals." - Sir Winston Churchill
Fade91 Posted August 24, 2015 Author Posted August 24, 2015 (edited) #include <File.au3> #include <Array.au3> #include <MsgBoxConstants.au3> $Path = "output_log.txt" $Count_Log = _FileCountLines($Path) $Handle = FileOpen($Path) While 1 $Recount_Log = _FileCountLines($Path) If $Recount_Log > $Count_Log Then ConsoleWrite(FileReadLine($Path, _FileCountLines($Path))) ; last written log ; Do Stuff _String(FileReadLine($Path, _FileCountLines($Path))) EndIf ;;; Probably won't use $Count_Log = $Recount_Log ; Would update to the very last written log ;;; Probably won't use ToolTip($Count_Log & @CRLF & $Recount_Log, 0, 0) ; log count compare (test only) WEnd FileClose($Handle) Now I need to somehow count where I left off and do something based on whats actually written and repeat this till I catch up to the 'last count'.I was thinking something like..._String(FileReadLine($Path, _FileCountLines($Path)))and have the function figure out what to do with the infoNow with the useful information of the log im thinking about may be writing it to a seperate log or ini file and having the GUI as a seperate script and having it read the log/ini to show the needed informationThe reason for this is to simply increase the speed of both scripts. I think im quite happy may be going this route. I think it might work Edited August 24, 2015 by Fade91
jchd Posted August 24, 2015 Posted August 24, 2015 _String(FileReadLine($Path, _FileCountLines($Path)))No, just use FileRead for the first time and store the byte size read off the file, which is set in @extended. Next runs, use FileSetPos to start reading up from the start of newly written data, then FileRead up to EOF. Of course this requires that nothing truncates the file in between, i.e. that the logger always appends data. Fade91 1 This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now