Jump to content

Parse time from each line of log (text file)

Vivaed

By Vivaed,
in AutoIt General Help and Support

 Share

Recommended Posts

Vivaed Seeker

Vivaed

Posted
Posted

I have another AutoIT script making a Log file 

Sample of Log file: 

2016/08/22 12:44:18 > Process: [RUNNING] [ACTIVE]
2016/08/22 12:48:35 > Process: [WAS NOT RUNNING]
2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 14:01:10 > Process: [WAS NOT RUNNING]

I am looping through the Log file for  the word "FAILED"

I then want to get all lines that have "FAILED" and get their TIME

 

My Current code to get this far: 

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    ConsoleWrite($filteredLine & @CRLF)     ; this gets me the results I want sans the time parse
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
EndIf

For this part: 

If StringInStr($filteredLine,$search) Then
  ConsoleWrite($filteredLine & @CRLF)   ; this gets me the results I want sans the time parse
EndIf

OUTPUT: 2016/08/22 13:40:00 > Process: [FAILED]

I dont understand how I read the time in that output?

 

I have tried _DateTimeFormat - Dont think this applies 

Tried _DateDiff - I dont have a the date yet so this doesnt work

 

Would love if someone could tell me if I am thinking is the wrong direction and possibly lead me down the correct path to light side of the force :) 

 

Link to comment
Share on other sites
  • Moderators
JLogan3o13 Universalist

JLogan3o13

Posted
  • Moderators
Posted

@Vivaed welcome to the forum. Does the output need to be in date format, or will a string work for your needs? Something like this works, based on the Log file sample you provided (which I expanded a bit). 

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        If StringInStr($sLine, "[FAILED]") Then
            $aTemp = StringSplit($sLine, ">")
            FileWriteLine(@DesktopDir & "\Output.txt", $aTemp[1] & " - Failed")
        EndIf
    Next

It returns the following: 

2016/08/21 13:40:00  - Failed
2016/08/22 13:40:00  - Failed
2016/08/23 13:40:00  - Failed
2016/08/24 13:40:00  - Failed
2016/08/25 13:40:00  - Failed
2016/08/26 13:40:00  - Failed
2016/08/27 13:40:00  - Failed
2016/08/28 13:40:00  - Failed

 

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted

Thanks for the Prompt response!!! @JLogan3o13

 

So I might not have explained it well lol 

2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 13:41:00 > Process: [FAILED]
2016/08/22 13:48:00 > Process: [FAILED]
2016/08/22 13:51:00 > Process: [FAILED]
2016/08/22 14:00:00 > Process: [FAILED]

Above is my actual log file

 

Lets say my current time was  2016/08/22 14:01:00

I want to look at my log file and compare the log files time stamps to _NowCalc(), I will use _DateDiff() (But that comes later)

 

Essentially what I am trying to do:

If the log file has >= 1 FAILED log line(s) in the past 15 mins then send email with these errors inside


I dont know how I parse for the date in the log file, I have tried Googled all I know #StillLearning

Link to comment
Share on other sites
  • Moderators
JLogan3o13 Universalist

JLogan3o13

Posted
  • Moderators
Posted

Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time: 

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next

If you do a consolewrite, you will see that the output is: 

2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

 

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted
13 minutes ago, JLogan3o13 said:

Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time: 

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next

If you do a consolewrite, you will see that the output is: 

2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

 

Interesting, I will have to learn more about this StringSplit() 

 

Thanks for the insight, I hope it leads me down a better path lol

Link to comment
Share on other sites
jchd Universalist

jchd

Posted
Posted

If your log file is significantly large, you'd probably better use a database to store log entries. SQLite is a very good candidate and is simple and efficient to setup and use.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
kylomas Universalist

kylomas

Posted
Posted (edited)

Vivaed,

Another way to do this... 

#include <array.au3>
#include <date.au3>

Local $aFailed = StringRegExp(FileRead(@ScriptDir & '\log.test.txt'), '(?i).*?failed.*', 3)

For $i = 0 To UBound($aFailed) - 1
    If _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) <= 15 Then
        ;
        ; This is where you would do your email routine...I just print the entry to console...
        ;
        ConsoleWrite('! ' & $aFailed[$i] & ' ' & _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) & ' minutes ago...' & @CRLF)
        ;
    EndIf
Next

How large is your log file?

How is the file(s) managed? (hourly, daily, rolling, etc)

How often do you want to scan the file(s)?

kylomas

edit: file used for testing log.test.txt

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted

@kylomas Thanks for the reply!

Another application checks to see if a process is running every minute and logs if [PASS] or [FAILED]

Then this app checks every 15 mins. It looks for if there were any fails in the last 15 mins, if so then email support team.

 

@jchd The log files are short for now, I could see them potentially getting long, but I would just delete them until another solution was created.

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted

Okay, I have the fist part working (Read log get FAILED lines and make new file with all FAILED lines): 

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>


Global $file_output = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 1)
Global $file_output_wipe = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 2)
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

$search = "FAILED"

Global $aFile = FileReadToArray($fileOutput)
Global $sTemp = ''

ConsoleWrite(_FileCountLines($fileLog) & @CRLF)
ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents, $search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog, $i)
                If StringInStr($filteredLine, $search) Then
                    FileWrite($fileOutput,$filteredLine & @CRLF)
                EndIf
            Else
                ConsoleWrite($search & " not found!" & @CRLF)
                FileWrite($file_output_wipe, $search & " not found!" & @CRLF)
            EndIf
        Next
    EndIf

EndIf

;~ ----------------------------------------------BROKEN
;~ For $sLine In $aFile
;~  $aTemp = StringSplit($sLine, ">")
;~  $sTime = $aTemp[1]
;~  ConsoleWrite($sTime & @CRLF)
;~  FileWrite($file_output_wipe,$sTime & @CRLF)
;~  Local $theDiff = _DateDiff('n', $sTime, _NowCalc())
;~ Next
;~ If $theDiff <= 15 Then
;~  ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
;~  FileWrite($file_output,$filteredLine & @CRLF)
;~ Else
;~ EndIf
;~ ----------------------------------------------BROKEN

FileClose($file_output)
Exit


But as you can see the bottom FOR statement doesn't work ( when un-commented lol )

Console Output: 

(65) : ==> Variable must be of type "Object".:
For $sLine In $aFile
For $sLine In $aFile^ ERROR
->09:45:23 AutoIt3.exe ended.rc:1
+>09:45:23 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 0.8435

I am missing something very obvious aren't I?

Link to comment
Share on other sites
  • Moderators
JLogan3o13 Universalist

JLogan3o13

Posted
  • Moderators
Posted

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted
1 minute ago, JLogan3o13 said:

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?

This?

Local $aFile = FileReadToArray($fileOutput)
ConsoleWrite(_ArrayDisplay($aFile))

Link to comment
Share on other sites
  • Moderators
JLogan3o13 Universalist

JLogan3o13

Posted
  • Moderators
Posted

@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this: 

Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)

Does it show you the array you're expecting to see?

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted
1 minute ago, JLogan3o13 said:

@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this: 

Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)

Does it show you the array you're expecting to see?

@JLogan3o13 No it does not, looks like it passes right over it

 

I will do more research on FileReadArray

Link to comment
Share on other sites
  • Moderators
JLogan3o13 Universalist

JLogan3o13

Posted
  • Moderators
Posted (edited)

So, that means it is not finding the file at that location, most likely. You could do something like this to confirm: 

If FileExists("path to file") Then 
   FileReadToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

 

Edited by JLogan3o13

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted
3 minutes ago, JLogan3o13 said:

So, that means it is not finding the file at that location, most likely. You could do something like this to confirm: 

If FileExists("path to file") Then 
   FileReadyToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

 

Its weird but I just put the snippet of code in the original IF statement:
  

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    For $sLine In $aFile
                        $aTemp = StringSplit($sLine, ">")
                        $sTime = $aTemp[1]
                        ConsoleWrite($sTime & @CRLF)
;~                      FileWrite($file_output,$filteredLine & @CRLF)

                        Local $theDiff =  _DateDiff('n', $sTime, _NowCalc())


                    Next
                        If $theDiff < 15 Then
                            ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
                        Else

                        EndIf
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
                FileWrite($file_output,$search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
FileClose($file_output)
EndIf

AND ITS OUTPUTTING????? (Kind of) 

37
NOW CALC TIME 
2016/08/23 12:06:22
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
+>12:06:22 AutoIt3.exe ended.rc:0
+>12:06:22 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.8868

I think this needs to be after FileRead??? I dont really know...

But this gets me a step closer lol
 

Link to comment
Share on other sites
kylomas Universalist

kylomas

Posted
Posted (edited)

Vivaed,

This seems to be what you are trying to do.  I would strongly encourage you to take JL's advice and understand what each function does by reading the Help file. 

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>

; name of log file
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
; name of output file
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

; open output file for write..erase previous content
Global $file_output = FileOpen($fileOutput, 2)

; check that file opened
If $file_output = -1 Then Exit MsgBox(17, 'Output file failed to open', 'File = ' & $fileOutput)

$search = "FAILED"

; read log file to array
Global $aFile = FileReadToArray($fileLog)
If @error Then Exit MsgBox(17, 'Log file read to array failed', 'File = ' & $fileLog)

ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

; iterate through array of log entries lokkiing for a "failed" entry withini the last 15
; minutes.  If found writye it to the output file

For $i = 0 To UBound($aFile) - 1

    If StringInStr($aFile[$i], $search) > 0 Then
        If _DateDiff('n', StringLeft($aFile[$i], 19), _NowCalcDate()) <= 15 Then
            FileWrite($file_output, $aFile[$i])
            ConsoleWrite($aFile[$i] & @CRLF)
        EndIf
    EndIf

Next

FileClose($file_output)

kylomas

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Link to comment
Share on other sites
Vivaed Seeker

Vivaed

Posted
  • Author
Posted

@kylomas Thanks for the insight!

I have been reading ton about these functions.

This script is interesting, it outputs all of the FAILED found but doesn't filter them by only showing the ones from the last 15 mins. since you used _NowCalcDate(), but if I change it to just _NowCalc() I get the results I am looking for.

I am very grateful for the code comments, they help me understand what is what and why things are happening!!!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...