Modified _FileCountLines()

[MilesAhead]

It doesn't matter what the default buffer is. I didn't look. The point is I've seen a lot of code, esp. in C/C++ that uses a 1 kb buffer on the stack as in char buf[1024] type of thing. My strategy would be if file size is trivial, suck the whole thing into memory. If not, then pick some buffer size. What is the "best" buffer size I would extrapolate from empirical evidence(iow, trial and error with a phd.)

If it turns out the buffer size never matters(which is difficult to imagine) then use a small buffer. But the optimization that's likely the most useful would be, why is this person counting the lines in a file? Are they doing it 200 times per second? Or once in the whole run of the app to find out how many lines are in an ini file? I would have general text file count lines and industrial strength. But just sucking the whole file up and letting it fail if the file is too big seems a bit sloppy. There should at least be some size check. If file size < heuristic then do it at once, else block read it.

Edited February 2, 2012 by MilesAhead

[jchd]

Of course StringInStr is way faster for "flat match" like this. I know better but half my fucking brain is crunching regexp by itself so that's what gets typed first

About buffer size, I didn't experiment at all. 8Mb looked decent. IMHO varying buffer size won't gain much in the general, average case.

[MilesAhead]

To me when you use the term "general case" it sounds more like trivial case. In the truly trivial case I would read a line inside a while loop and increment a variable. If the last line is blank I would count it or not according to my preference. Otherwise, since we are talking a library function and not something I whip together for a single script, it seems the function should be more robust and have some reasons for the things it does. One reason would be knowing the file size. I don't understand the aversion to taking advantage of the function that provides the information.

I don't think there really is an average case in this case. If I need to know the number of lines in an ini file, my while loop is fine. If I need some optimization because for some strange reason I'm going to parse the library of congress data using scripting, then an optimized version should be called. I don't get the fascination with benchmarks for the trivial case. If it's a big job that needs optimizing then get some facts about the job.

[Beege]

Of course StringInStr is way faster for "flat match" like this. I know better but half my fucking brain is crunching regexp by itself so that's what gets typed first

You meant StringReplace() ?

Edit: Nevermind, I think you were refering to Malkey's post.

If it turns out the buffer size never matters(which is difficult to imagine) then use a small buffer.

No I'm pretty sure it matters. I never verified it but we've talked about it a couple times though out this thread. I think Tlem said he hit 170mb when it fails. Edited February 3, 2012 by Beege

[Tlem]

@MilesAhead

AutoIt does not address necessarily to an elite, thus the general optimization is not still within the reach of each.

Such as it is indicated :

the software was primarily intended to create automation scripts

Usage

AutoIt can be used to produce utility software for Microsoft Windows and automate common tasks, such as website monitoring, network monitoring, disk defragging and backup. It is also used to simulate application users, whereby an application is driven by an AutoIt script in place of manual application control during software testing. It is also commonly used for developing Computer game bots, for automating in-game tasks.

As we know it all, an automation is designed to do repetitive tasks. When in a script, a function is not hurriedly optimized, it burdens the whole program. If for you, the fact of winning 2 seconds represents nothing, when you have to repeat the operation 5000 times per day, it is three hours of lost!!!

I think that your computer has more interesting things to make that to waste time on a not optimized function.

In your daily work, you certainly not have this kind of inconvenience, but you are not alone to use AutoIt, thus think not only of you.

Edited February 3, 2012 by Tlem

[jchd]

Yeah, StringReplace not StringInStr. I was exhausted by too long driving.

When we talk algorithm, what we refer to the "general case" is "all cases that _may_ occur" (regardless of frequency). The "practical case" refers to what you can expect on average real-world use. That may highly depend on your domain you're doing the job for.

Think of a string manipulation fonction that would crash or give erroneous result(s) when two x00 are found in a row in the subject string. If the definition of "string" you refer to allows presence of x00, then it's definitely not useable in the general case. In practice, you may consider things differently.

Now, the "buffer size" question is useless nitpicking. Please ask yourself how frequently _you_ have found yourself compelled to using this function! Counting lines without reading them is very rarely meaningful except when you need to get an idea of how large a project is, or situations like that. IMHO, in such contexts, the actual, precise number is essentially irrelevant (only the order of magnitude matters) and that the function is highly optimized or not is also a non sequitur.

Given the rare need to use that function, it would be ridiculous to even consider adding code to "optimize" buffer size to the filesize to be processed in order to spare an unsignificant fraction of time. Higher optimization would also require to decide if the media is HD or SSD, which sector- or block-size that devices uses and which cache size the device has, probability of parts of the file being in OS cache, rotation speed of HD (probably the most important factor here) ... (I'm joking!)

[Beege]

Please ask yourself how frequently _you_ have found yourself compelled to using this function!

Frequently for me. At least twice every ten years.

[guinness]

I don't think I ever have, because 10 times out of 10 I need to read the file anyway.

[Tlem]

About the frequency of the use of this function, all is a point of view.

Personally, if a work can be do faster, why not to take advantage of it.

Having said that, with the new functions, the size of the file is not a problem any more.

Now, even if I use this function rarely, I have an alternative faster than mine for big file, so for me this subject will not have been useless. ^^

[dplaut]

@JHCD: I just ran this on a file with 11,494,854 lines and your function worked!

[jchd]

Good to know.