Optimized Array.au3

[SmOke_N]

Yes, I agree with the above... amazingly, I think I've only ever used this function 1 or 2 times, and that was helping others.

[-Ultima-]

If I'm remebering correctly, the only two functions that I broke backwards compatibility with (with regards to function parameters) were _ArraySort() and _ArrayDisplay(). Error codes were changed in a bunch of functions too, so I think that might break backwards compatibility for some people that were relying on them. That I've no way to be sure of until it actually happens. I sure hope no one was relying too heavily on _ArraySearch()'s previous error codes, though...

Edited August 26, 2007 by -Ultima-

[Jos]

If I'm remebering correctly, the only two functions that I broke backwards compatibility with (with regards to function parameters) were _ArraySort() and _ArrayDisplay(). Error codes were changed in a bunch of functions too, so I think that might break backwards compatibility for some people that were relying on them. That I've no way to be sure of until it actually happens, but I sure hope no one was relying too heavily on _ArraySearch()'s previous error codes...

Hooooo .... don't go too fast here for me ..... its late here ..

What is the compelling reason for any of the other mentioned backwards compatibility issue ?

[-Ultima-]

Error codes 2-4 for _ArraySearch() redundant, and were handled properly by my new bounds checking for the parameters. As for error 5, it doesn't even matter anymore, so there shouldn't be an error. Finally, for error code 6, I simply moved it to error code 4 (errors 2 and 3 was redefined).

Some of the other error codes were also deprecated because they no longer pertain to the function with my bounds checking/correcting. The only problems that occur might be when I redefined a few error codes, or moved error codes (as in _ArraySearch()). Should I leave the old error codes alone? That would leave gaps in the error codes (like with _ArraySearch(), the old errors 2-5 are no longer pertinent)...

I keep mentiong _ArraySearch() because it was probably the biggest change with regards to returned error codes. A few other functions probably had 1, or maybe at most 2 error codes changed/removed (for the same reasons as to why I changed _ArraySearch()'s error codes). I'll try to list them later on.

Other than these, I don't think there are any other backwards compatibility problems.

Edited August 26, 2007 by -Ultima-

[Jos]

Other than these, I don't think there are any other backwards compatibility problems.

mmmm... always love these statements that gives me that warm and fuzzy feeling that all is well ..

Its important we list all changes as clear as possible (also for documentation in the History log) and these updates need to be properly tested before I am going to update the current Beta version.

George also has been sending me some proposals for _ArraySearch() so like to see you guys and gals all agree on the changes before any final version is submitted.

I will try to have a look at what you posted somewhere the coming week to checkout the changes..

Jos

[-Ultima-]

Alright, cool And yeah,

I'll try to list them [the changed error codes and parameters] later on.

George sent me a copy of his _ArraySearch() too (and various other array-related functions). I'm going to go over it when I find a bit more time.

Edited August 26, 2007 by -Ultima-

[-Ultima-]

@randallc: Meh I've decided to readd the parameters. They'll be there on next update Quick comment/question, though... I'm not that big a fan of setting hard limits unless absolutely necessary, so um... regarding $i_ShowOver4000, would you mind if I changed it into an actual setting for the limit instead of making it a toggle for a hard 4000-item limit? And regarding your updated script... it seems to be around 50% slower than before What kinds of enhancements did you make to _ArrayDisplay()? (I see what changes you made... but isn't making it *that* customizable going a bit too far...? IMHO, it should just be a simple method for viewing an array...)

@GEOSoft: I've gotten through implementing the directional searching (very simple change in my code, really). _ArrayFindAll() was also reimplemented, and is now around 15-16% faster. I've been trying to think of a way to reimplement _ArrayUnique(), but a fast-yet-failure-proof solution has been eluding me. I'm sure you already know this, but the problem with your current implementation is that it can give incorrect results if the delimiter is set badly. I came up with a solution that will always work, but the problem is that it's around 2.5x slower. Basically, I sort the array from $iStart to $iEnd using _ArraySort(), then I go through the array from $iStart to $iEnd and copying elements one by one assuming the next element in $aArray is not equal to the last element in the holding array (taking $iCase into account). The sorting is what's killing the speed for this otherwise accurate solution :\

Quick question... __FileReadToArray() is headed for File.au3, and not Array.au3, right?

Edited August 27, 2007 by -Ultima-

[GEOSoft]

@randallc: Meh I've decided to readd the parameters. They'll be there on next update Quick comment/question, though... I'm not that big a fan of setting hard limits unless absolutely necessary, so um... regarding $i_ShowOver4000, would you mind if I changed it into an actual setting for the limit instead of making it a toggle for a hard 4000-item limit? And regarding your updated script... it seems to be around 50% slower than before What kinds of enhancements did you make to _ArrayDisplay()?

@GEOSoft: I've gotten through implementing the directional searching (very simple change in my code, really). _ArrayFindAll() was also reimplemented, and is now around 15-16% faster. I've been trying to think of a way to reimplement _ArrayUnique(), but a fast-yet-failure-proof solution has been eluding me. I'm sure you already know this, but the problem with your current implementation is that it can give incorrect results if the delimiter is set badly. I came up with a solution that will always work, but the problem is that it's around 2.5x slower. Basically, I sort the array from $iStart to $iEnd using _ArraySort(), then I go through the array from $iStart to $iEnd and copying elements one by one assuming the next element in $aArray is not equal to the last element in the holding array (taking $iCase into account). The sorting is what's killing the speed for this otherwise accurate solution :\

Quick question... __FileReadToArray() is headed for File.au3, and not Array.au3, right?

!st) usually my arrays can not be sorted. If they are I have a problem

2nd) Array Unique is not mine. Read the notes. It's only there so I can try to work out a compatability problem. The note says it all

3ed) __File read to array is still up in the air. Yes it's a file function. Yes it's an array function. The difference is that I removed the 1 parameter so I didn't want it getting mixed up. What I may do is put that parameter back in with a new name and not reference it within the function at all. Redundant, but harmless.

Withe the old one you would do something like

$array = ''

_FileReadtoArray($myFile, $Array)

With my version it' judt

$Array = _FileReadToArray($MyFile)

Also in some of those functions I shrtened them by removing some of the error codes

If $var <> 1 AND $Var <> 0 Then

ReturnSetError(2,2)

EndIf

Is better as

If $Var <>1 Then $Var = 0

[-Ultima-]

1) Indeed, sorting would be a problem for some people, but with my solution, I had changed it so the function read the variable by value, not by reference. Either way, though, it's unimportant now.

2) The notes don't say anything other than that you (or the original author) are working on it, so I couldn't have figured anything out from the note, really. Basically, I shouldn't be adding this function?

3) Yep, I know what you changed the function. Even if you put the variable back in (without using it), though, backwards compatibility would still be broken, since any script that called it previously expected the passed array to be modified. Probably out of the scope of this discussion though

So um, other than the bi-directional searching and _ArrayFindAll(), should I be incorporating any other functions? I was just about to work on _ArrayReplace(), but wasn't too sure if you wanted that too. And if you do want me to incorporate it, do you mind if I place an $iStart and $iEnd before $iCase?

Edited August 26, 2007 by -Ultima-

[GEOSoft]

1) Indeed, sorting would be a problem for some people, but with my solution, I had changed it so the function read the variable by value, not by reference. Either way, though, it's unimportant now.

2) The notes don't say anything other than that you (or the original author) are working on it, so I couldn't have figured anything out from the note, really. Basically, I shouldn't be adding this function?

So um, other than the bi-directional searching and _ArrayFindAll(), should I be incorporating any other functions? I was just about to work on _ArrayReplace(), but wasn't too sure if you wanted that too. And if you do want me to incorporate it, do you mind if I place an $iStart and $iEnd before $iCase?

1) The reason that sort is a problem is because I very often use _FileReadToArray() for file parsing.

Then I do the __ArraySearch() or __ArrayFindAll() To get the elements that require modifying then write the array back to the file. If the array is sorted then I can't write back in the proper order. For Adding the Parms to __ArrayFindAll...... have at it.

2) Don't really care one way or the other. I just put that one in so I could try and do some work with it.

3) All of the array functions should be included. The file I sent you is primarily the functions I modified or added.

[pdaughe]

Ultima,

Good work! On the issue of backwards compatibility -- I'll always favor cleaner, more efficient code. When the changes to FileOpen and BinaryString were made, I had to change several scripts, but the end-result was cleaner, more efficient code.

The courtesy requirement is good documentation regarding the compatibility issues.

I just really appreciate your diligence.

Sincerely,

Paul

[-Ultima-]

Hm... After thinking about my bounds checking, I'm starting to see the point behind _ArraySearch()'s old error codes. I'm not sure if I should be correcting the bounds, or if I should simply fail on improper bounds. Why am I wondering? Because if users rely on a failure notice to know that their bounds are off, then correcting it on-the-fly like I currently do will cause unexpected results. This can happen in easily when the function is called repeatedly in loops where the variable is incrementing (or decrementing). Ideally, I guess the bounds should be checked before the script enters any loops (or at the end of each loop) in the first place, but there may be times where that might not be possible, optimal, or easy... Opinions?

For those wondering, my current method of bounds checking is as follows:

; Bounds checking
If $iEnd <= $iStart Or $iEnd > $iUBound Then $iEnd = $iUBound
If $iStart < 0 Or $iStart > $iEnd Then $iStart = 0

This may vary where necessary for any specific function, but that's basically how I'm doing it at the moment.

Other than the above, I'm done with updating my copy, and have the possible compatibility problems list done. I'm pretty much just awaiting responses to the above now, so that I don't have to release another copy if enough people think I should change my bounds checking/correcting.

__________________________________________________

@GEOSoft: I'm not sure if _ArrayBase() should be included, the sole reason being the fact that it's unreliable. The problem with using it is that if I have an array, Local $avArray[5] = [4, 5, 6, 7, 8], it would return the incorrect base index, since (just by "coincidence") $avArray[0] happens to store the same value as UBound($avArray)-1. There's a lot of guess work involved in using the function, and frankly, you might as well make clear what you expect the array to be like rather than allow the little bit of unreliable leniency Also, I understand you said you don't care much, but just as a heads-up, I opted out of including the current implementation of _ArrayUnique() for the same reason (its unreliability). I'm still not sure if I want to include my implementation, as it's slow, and doesn't preserve order.

So yeah, that was just an update on my progress with your functions; everything else (besides __FileReadToArray()) is, as far as I can tell, incorporated

__________________________________________________

@randallc: I'm still looking for your response to this:

@randallc: Meh I've decided to readd the parameters. They'll be there on next update Quick comment/question, though... I'm not that big a fan of setting hard limits unless absolutely necessary, so um... regarding $i_ShowOver4000, would you mind if I changed it into an actual setting for the limit instead of making it a toggle for a hard 4000-item limit? And regarding your updated script... it seems to be around 50% slower than before What kinds of enhancements did you make to _ArrayDisplay()? (I see what changes you made... but isn't making it *that* customizable going a bit too far...? IMHO, it should just be a simple method for viewing an array...)

/me wonders if he's annoyed randallc because of the apparent unwillingness to change stances before... Edited August 27, 2007 by -Ultima-

[randallc]

/me wonders if he's annoyed randallc because of the apparent unwillingness to change stances before...

No, thanks! Not at all.. busy, and I missed this..

It wouldn't break scripts if you had a once off "if" ;if =1 then =1000! or similar?[then scripts would not be broken, just need to look into new helpfile if they notice less items displayed?...]

I'm wondering why you would want another number, though? - perhaps could be usefule to display an array when already have a lv open and no handles left , or few..?

best, randall

[-Ultima-]

Sounds reasonable, I suppose. I went ahead and added

If $iItemLimit = 1 Then $iLimitItems = $iLVIAddUDFThreshold

As I said, I'm just not a big fan of hard limits unless absolutely necessary, which is why I'd prefer any limitations to be user definable instead of hardcoding

Edited August 27, 2007 by -Ultima-

[randallc]

(I see what changes you made... but isn't making it *that* customizable going a bit too far...? IMHO, it should just be a simple method for viewing an array...)

Hi,

Not sure which ones you mean; which is slowing it?; not the column size adjustment?? [used to prevent crashes, but ? not so necessary since the fix - ]

what else did i customize? - forgotten! I'll have to look...

randall

[-Ultima-]

Dunno, I checked, and it took about 50% more time to generate a window of a few hundred items with (I think) 5 columns I might have to test it again some time later to be sure about this.

Admittedly, I didn't looked too carefully at your changes (I'll try to get around to that when I get more time), but the most obvious changes were the column header customization, and the parameter for setting the window to be maximized at display. I seriously doubt this would cause the slowdown, though...

Edited August 27, 2007 by -Ultima-

[randallc]

Dunno, I checked, and it took about 50% more time to generate a window of a few hundred items with (I think) 5 columns I might have to test it again some time later to be sure about this.

Admittedly, I didn't looked too carefully at your changes (I'll try to get around to that when I get more time), but the most obvious changes were the column header customization, and the parameter for setting the window to be maximized at display. I seriously doubt this would cause the slowdown, though...

H,

I don't think I ever finished tat script; it looks like it was trying to write the items twice?... and nor LVcreate works for more than 253 columns anyway; maybe ignore it all for now..

Later, Randall

[GEOSoft]

@GEOSoft: I'm not sure if _ArrayBase() should be included, the sole reason being the fact that it's unreliable. The problem with using it is that if I have an array, Local $avArray[5] = [4, 5, 6, 7, 8], it would return the incorrect base index, since (just by "coincidence") $avArray[0] happens to store the same value as UBound($avArray)-1. There's a lot of guess work involved in using the function, and frankly, you might as well make clear what you expect the array to be like rather than allow the little bit of unreliable leniency Also, I understand you said you don't care much, but just as a heads-up, I opted out of including the current implementation of _ArrayUnique() for the same reason (its unreliability). I'm still not sure if I want to include my implementation, as it's slow, and doesn't preserve order.

Have you tried the example you gave? I don't really care if it gets included or not. I'm leaving, remember. As it is, it works with anything I've tried it on and soon I will be compiling all of those scripts for the last time so they will still work no matter what changes are made to UDF's or to AutoIt itself.

I wrote that mainly so I didn't have to mess around when deciding to use 1 or 0 for $iStart in some of the other functions like _ArraySort().

[GEOSoft]

Hm... After thinking about my bounds checking, I'm starting to see the point behind _ArraySearch()'s old error codes. I'm not sure if I should be correcting the bounds, or if I should simply fail on improper bounds. Why am I wondering? Because if users rely on a failure notice to know that their bounds are off, then correcting it on-the-fly like I currently do will cause unexpected results. This can happen in easily when the function is called repeatedly in loops where the variable is incrementing (or decrementing). Ideally, I guess the bounds should be checked before the script enters any loops (or at the end of each loop) in the first place, but there may be times where that might not be possible, optimal, or easy... Opinions?

Opinion.........

You can remove the bounds checking.

If $whatever <> 1 Then $whatever = 0

means that it is going to be within bounds so there is no error to return. Much better than

If $whatever <> 0 And $whatever <> 1 Then
   SetError(2)
   Return 1
EndIf

[-Ultima-]

NEW

Array_benchmark.zip (for benchmarking)

Array_UDF.zip (for UDF submission)

In Array_UDF.zip, I've included a list of any possible compatibility problems (Compatibility Concerns.txtthat may arise in the transition from the current official Array.au3 and my version. I honestly tried to track down all changes, but I might have missed something along the way (I doubt it though). Even still, that's why these "new" UDFs need to be tested widely, so please test if you can? Thanks, and enjoy!

Changes since last time...

- $iEnd now defaults to 0 instead of -1 (minor change, but will probably clear up some backwards compatibility problems I forgot about)

- _ArrayDisplay() should now be backwards compatible (old parameters added back)

- Added _ArrayFindAll() [thanks GEOSoft]

- Added backwards searching capability to _ArraySearch() [thanks GEOSoft for the idea]

- Changed error code for _ArraySearch() and _ArraySort() just a bit more; now properly sets $iSubItem if out of bounds instead of simply failing

Edited September 9, 2007 by -Ultima-