New FAST line counter

[Zedna]

I guess my measly .023 seconds for 10,000 lines or .21 seconds for 88,711 lines is out of the running here.

SRE's people, SRE's

So post RegExp version here for us poor RegExp people.

Share your knowledge.

[GEOSoft]

Read my reply to Jos. I linked to it there. I don't post code in this forum anymore.

[MvGulik]

I don't post code in this forum anymore.

Your sure about that!

In communications and information processing, encoding is the process by which information (thoughts) from a source (George) is converted into symbols (text) to be communicated (forum message)

Edited December 30, 2010 by MvGulik

[Mat]

Your sure about that!

Yes he is. You are looking at code as in encryption code, taking the definition out of context.

(computer science) the symbolic arrangement of data or instructions in a computer program or the set of such instructions

[guinness]

Having said that, we should probably look at the current UDF's for improvements using SRE.

Where could code be posted, instead of writing a bug report? I have made a suggestion for in the Example Forum, as the current _GetIP() is somewhat outdated.

[Jos]

Trac is the place to submit bug reports and possible updates.

Just not sure how long it will take to get a reply as I am sure who is currently working on processing the UDF updates.

Jos

[guinness]

OK, cool. Thanks for the reply

[MvGulik]

Yes he is. You are looking at code as in encryption code, taking the definition out of context.

Bah, spoilsport.

[Tlem]

I All.

All the functions which I was able to see, raise an enormous problem with a file of consequent size.

On my Windows 32 bit XP system, AutoIt cannot read files with a size greater than 183 145 241 octets.

If I try to do that, I'll get an error allocating memory.

For my personal use, I use a simple loop with FileReadLine a file handle and a counter, but this function is 4x slower than tylo's function, having said that, it works for files of any size.

I play a little time with tylo's function, and I noticed something very strange.

if I use it with a large file > 179 639 503 octets, then if I run the function 1 or more times and then I use a MsgBox and again I use the tylo's function, then I have the error allocating memory (even if I wait a long time between the last launch).

If the size is smaller or equal to 179 639 503 octets, then I can do what I want.

This is a sample code that produce the error :

; Test.txt file is a large file > 179 639 503 octets
; 179 639 600 octets is a good choice...

$Lines = _TFileCountLines("Test.txt")
MsgBox(0,"Just a MsgBox", "This message box seems to produce the error if file size if greater than 179 639 503 octets")
; To there, I havn't the error allocating memory

; Before this one finished, then I have the error allocating memory
$begin = TimerInit()
$Lines = _TFileCountLines("Test.txt")
MsgBox(0,"Time Difference for " & $Lines,TimerDiff($begin)/1000)
Exit


Func _TFileCountLines( $sFilePath )
   Local $N = FileGetSize($sFilePath) - 1
   If @error Or $N = -1 Then Return 0
   Return StringLen(StringAddCR(FileRead($sFilePath, $N))) - $N + 1
EndFunc

[ALFJ]

I used the other functions to count the lines on my standard ZIP codes file

I did not do any timed test I just checked the console for the time it took and here's what I get still.

Not watching a movie or anything and I did run the test more than once per function just to be fair (3)times.

Here's what I got from it:

Total Lines : 34417

$FLN = "Zip-Codes.txt"

$FLNcount = _File_CountLines($FLN)

Time: 0.553, Time: 0.542, Time: 0.548

$FLNcount = _FileCountLines($FLN)

Time: 0.331, Time: 0.336, Time: 0.324

$FLNcount = _File_CountLines2($FLN, 1)

Time: 0.546, Time: 0.542, Time: 0.550 (about the same results if you use 0 instead of 1 as an option)

Then again I have no empty lines so it wouldn't matter much.

Here's a sample of what's in each line: 01001,AGAWAM,MA,HAMPDEN,STANDARD

Not sure if what's in it effects the results even though there is not much of a difference I know which function I am still sticking to

I also like the FileReadLine it's great I don't even have to open a file to read the text.

I can just write something like $FileLine = FileReadLine($FLN, $LNum) and it gives me the text that is on the line # referred to by $LNum in the $FLN file name.

[ALFJ]

All the functions which I was able to see, raise an enormous problem with a file of consequent size.

On my Windows 32 bit XP system, AutoIt cannot read files with a size greater than 183 145 241 octets.

If I try to do that, I'll get an error allocating memory.

For my personal use, I use a simple loop with FileReadLine a file handle and a counter, but this function is 4x slower than tylo's function, having said that, it works for files of any size.

; To there, I havn't the error allocating memory

; Before this one finished, then I have the error allocating memory

$begin = TimerInit()

$Lines = _TFileCountLines("Test.txt")

MsgBox(0,"Time Difference for " & $Lines,TimerDiff($begin)/1000)

Exit

Func _TFileCountLines( $sFilePath )

Local $N = FileGetSize($sFilePath) - 1

If @error Or $N = -1 Then Return 0

Return StringLen(StringAddCR(FileRead($sFilePath, $N))) - $N + 1

EndFunc

I use the console for my results (not into Message boxes ), don't know about the size issue you're having, would have to run a test on a huge file to see if I get the same problem and experiment with it. My Zip code file, took a while parsing it and writing it to my needs, if I had to do it manually could have taken way longer hope I never run into a file that is that big. Edited December 31, 2010 by ALFJ

[Tlem]

In fact, I don't use really MsgBox, it's just that I have seen that on my tests.

I think it's not a normal behavior.

Perhaps it's an AutoIt bug?

I said that only for information, because I spent several hours to make tests to see the limits.

[ALFJ]

In fact, I don't use really MsgBox, it's just that I have seen that on my tests.

I think it's not a normal behavior.

Perhaps it's an AutoIt bug?

I said that only for information, because I spent several hours to make tests to see the limits.

I think it could be a size or type of text in the file issue, If it's due to the size then I guess you might have to use some sort of buffering to do parts at a time, which I don't know how to go about doing myself but someone else might know here and know if that's the case..

The way I see it.. is watching a HD movie from a website, they send information in buffer style, same thing when you are watching a movie locally, let's pretend it's a 100GB size movie.. of course if you open the whole movie with the movie app your PC will most likely crash, so I assume some sort of buffering is used with the movie viewers as well.. however you are trying to open a huge file with no buffer so it's failing, my opinion on it anyway.. whether it's right or not.. I am sure someone with more knowledge than me or someone with a similar issue might give us more pointers on this.