Fast New ArrayDisplay with Fast Sort,unlimited columns

[randallc]

Hi,

Here is Array Display with all new features plus Fast vbs sort (WinXP or better;

Best, Randall

** now unimited columns

I have changed to fantastic fast display, thanks @Siao!

3 x10,000 array ;

displays in 0.3 secs max.

Sorts by column in about 1 sec (10,000/sec) [1D now 30-40,000/sec, 2D 8-9,000/sec]

(delay after Display till ready to sort of 0.5sec; or usually half the sort time)

[jsort no longer used; @Siao Clib sort - with thanks ]

Reverts to "_ArrayDisplaySort" UDF included if delimited Array of strings, like CSV

Displays 1D, 2D, or 1D array of strings (eg filereadToArray of a csv)

Best, Randall

_ArrayDisplaySortV.zip ( 21.68K ) Number of downloads: 60

I am very happy with the workaround for unlimited columns, using the slider! - though

Problems still

1. I would rather use Virtual ListView purely if I can work it out. [any more ideas?...]

2. Scrolling horizontally is fast, but reverts to column1 on scrolling laterally.

3. I would appreciate help with the GUI, buttons, resizing Slider control, if anyone has the time...

Best, Randall

[PS - added back extra include...]

_ArrayDisplayVS.zip

Edited February 15, 2008 by randallc

[randallc]

Hi,

removed due to a bug when there were blank cells; now fixed 29th Dec in post #1

Randall

[zxzxzx]

Hi,

Here is Array Display with all new features plus Fast vbs sort (WinXP or better; could be changed t use 98 if needed)

Best, Randall

; fileToArrayDupes2.au3
#include-once
;~ #include "Array.au3"
#include "_ArrayDisplayG.au3"
#include <ArrayUltDisplay.au3>
#include<_FileListToArrayFaster1g.au3>
Local $Message = "all              files    ?", $Exclude, $filter = "*", $Flag = 1, $Recurse = 1, $BasDir = 1
Local $timerstamp1 = TimerInit(), $sPath = @ScriptDir, $filter = '*.*', $filter = '*.*|*.doc'  ;, $filter = '*.doc'
ConsoleWrite("$sPath=" & $sPath & @LF)
$ar_Array = _FileListToArray3 ($sPath, $filter, $Flag, $Recurse, $BasDir);, "", 0) ;
ConsoleWrite($Message & "=" & UBound($ar_Array) - 1 & @TAB & " _FileListToArray3 " & @TAB & _
        ":" & @TAB & Round(TimerDiff($timerstamp1)) & "" & @TAB & " msec" & @LF)
;~ _ArrayUltDisplay ($ar_Array, "Fast Array Display, Gary and  Ultima; Click column header to sort", 0)
_ArrayUltDisplay ($ar_Array, "Fast Array Display, Gary and  Ultima; Click column header to sort", 0,0,"|","~",1,"") ; quicker with no "check for blanks"
;~ _ArrayDisplay ($ar_Array, "Fast Array Display, **Ultima; ")

[EDIT - again removed due to a bug when there were blank cells; now fixed 29th Dec; sorry]

oks o ive posted my code!... its not very good scripting tho.. very noobish

thank

[randallc]

Hi,

See post #1 changed;

Please let me know if you find any bugs..

I have changed to fantastic fast display, thanks @Siao!

3 x10,000 array ;

displays in 0.3 secs max.

Sorts by column in about 1 sec (10,000/sec) (dealy after Display till ready to sort of 0.5sec, usually half the sort time)

Reverts to "_ArrayDispalySort" UDF included if more columns than 5...

Displays 1D, 2D, , or 1D array of strings (eg filereadToArray of a csv)

Fast New ArrayDisplay with Fast Sort(Virtual)

Best, randall

[Siao]

Wanna try this one?

Seems faster than your _ArraySortDelim() in some cases, although I haven't tested extensively, and not sure if it's the right one to test against (I believe I recall you having some faster string sorters)...

Edit: code removed, see updated #471653

Edited January 30, 2008 by Siao

[randallc]

Hi,

Yes, this technique.. (I don't understand it, but it's great!) looks to be 2X faster for 1D array as it stands (and maybe 3-4x for 2D if that is working...), but

1. As you comment, does not sort numeric (yet?..)

2. Does not test if the column is all numeric.. which will be a performance hit, I guess

So I can't look at it for use here yet; I'll try to do some more comparisons.

The previous sort comparison thread was only for numeric and 1D, so if this works for a general solution eventually it will be better than anything I know which can be run from AutoIt.

best, randall

[PS, of course, I'd have to go back to using 2d arrays again, and would need the sort data to be converted differently; not a big deal, but I wouldn't do it here unless I know it is going to work .]

Best, randall

[Siao]

1) I'm gonna try to add numeric sort option somewhen somehow, for I may need such feature in future (right now all I need is string sort). It will have to involve writing ASM code to pass to qsort(), I'm just not sure yet how it should work overall and what wouldn't be too bad for the performance. Treat input column as ints/floats? Treat as strings like now, just with proper numerical ordering? Will have to try and see I guess...

2) What's that for, error checking? To my way of thinking, if I'm sorting I should know what I'm sorting and it's on me to make sure I'm sorting what I want to be sorting.

Edited January 29, 2008 by Siao

[randallc]

1) I'm gonna try to add numeric sort option somewhen somehow, for I may need such feature in future (right now all I need is string sort). It will have to involve writing ASM code to pass to qsort(), I'm just not sure yet how it should work overall and what wouldn't be too bad for the performance. Treat input column as ints/floats? Treat as strings like now, just with proper numerical ordering? Will have to try and see I guess...

2) What's that for, error checking? To my way of thinking, if I'm sorting I should know what I'm sorting and it's on me to make sure I'm sorting what I want to be sorting.

Hi,

OK, thanks..

1. not sure,.. hope you find it1

2. yes, OK, but if someone is to use your func to throw an unknown array... we can guess from a s ample, and make corretcions as an option.

3. You were right about at least one of my previous funcs being faster than the Clib; one of the jscript funcs can return an array as string very fast (for 1D), and I had forgotten; the stringsplit still leaves itr 2-3 x as fast as yours for strings, case insensitive, so I will try that for that specific situation; works really well. [i will update attachment in post #1 later..]

4. what your function could do really fast, if I knew how, would be to do the sort using $hMsvcrt or whatever, then -not- read the array in that function, but only small bits via the call-back; but I can't get it to work from the call-back, even with every variable relevant put into global... Any idea if it is possible?

Best, randall

Edited January 30, 2008 by randallc

[randallc]

Hi,

See last post for changes; new faster 1D sort fixed in post #1 attachment.

Best, randall

[Siao]

Not sure I completely understand what you wanted to say in #4.

qsort call itself is smokin fast. Problem is getting the input data to it from the array, and getting the sorted data back into the array. These two things take most of the running time (though not as much as I initially feared). But I don't know what else can be done about this fact. If I had direct low-level access to AutoIt array (that is, would be able to get its memory address somehow), maybe that could be a different story.

Anyway, here's updated version of my sorter. Still no numerical compare yet. Much faster 2D though. Variant types of original array elements preserved in 1D too (used to be forced to string).

Edit: code removed again, see http://www.autoitscript.com/forum/index.php?showtopic=63525

Edited February 2, 2008 by Siao

[randallc]

Hi,

Thanks;Just looking, i'm not sure if that will help or not;

My Question is whether you could do half the script in one func; up to the sort, -not- free the dll, then read values into the array or part of it, in another func? If yu could show me a demo of that it would make it doable and I will keep trying; for the display I don't need to wait for the whole array to be read back, only individual items....

Best, Randall

[EDIT - never mind, for now; I see that I can, indeed, retieve them from outside the func; now to get it working .. thanks again]

Edited January 30, 2008 by randallc

[Siao]

Have $tSource in global scope (and remove it from local scope definitions at the start of _ArraySortClib). Comment/delete the ";; read back the result" part of the code inside the function (leave the FreeLibrary call be, you don't need the dll to be loaded anymore at that point).

When the function returns, you can access any $tSource struct element with DllStructGetData.

Also you could modify the function to optionally accept dllstruct as a parameter instead of array, and skip reading from array to memory part in such case, so you could quickly resort the same struct again if you need that for whatever reason.

I actually had the same line of thinking, not use the autoit arrays as a mediator at all when I need more speed sorting/searching simple but large enough data, because getting data from memory (lets say, $x = DllStructGetData($struct, 5)) takes about the same time as getting from array (lets say, $x = $array[4]) anyway. That's why I experimented with bsearch from msvcrt too in my previous version... of course, right now a lot of memory is wasted, better would be to have memory array of pointers to strings instead of array of strings, that would take more time to initialize (when reading from array StringLen each element, etc) also would need a custom compare procedure for qsort/bsearch, but it wouldn't hog as much RAM and would be easier to add/remove elements to/from it if need be. Would be kind of like vbs/autoit arrays keep their data. Not sure if it's worth going that far.

Edited January 30, 2008 by Siao

[randallc]

Have $tSource in global scope (and remove it from local scope definitions at the start of _ArraySortClib). Comment/delete the ";; read back the result" part of the code inside the function (leave the FreeLibrary call be, you don't need the dll to be loaded anymore at that point).

When the function returns, you can access any $tSource struct element with DllStructGetData.

Also you could modify the function to optionally accept dllstruct as a parameter instead of array, and skip reading from array to memory part in such case, so you could quickly resort the same struct again if you need that for whatever reason.

I actually had the same line of thinking, not use the autoit arrays as a mediator at all when I need more speed sorting/searching simple but large enough data, because getting data from memory (lets say, $x = DllStructGetData($struct, 5)) takes about the same time as getting from array (lets say, $x = $array[4]) anyway. That's why I experimented with bsearch from msvcrt too in my previous version... of course, right now a lot of memory is wasted, better would be to have memory array of pointers to strings instead of array of strings, that would take more time to initialize (when reading from array StringLen each element, etc) also would need a custom compare procedure for qsort/bsearch, but it wouldn't hog as much RAM and would be easier to add/remove elements to/from it if need be. Would be kind of like vbs/autoit arrays keep their data. Not sure if it's worth going that far.

Hi,

Yes, the initial speed is a problem. I have got it working, but probably won't use it; unless we input the data quicker by another method? Disappointing because the sort is probably quite fast..

Otherwise, I'll stick to the other sorts for now and find the best one for each situation..

Best, randall

[Siao]

Fair enough. If performance is the most important thing, tweaking everything depending on the situation makes sense.

I'll continue working on my own though, mostly because I want to have a convenient solution that works from one function, and which I'm familiar with because I've written it myself

(not having to browse through several files of multiple functions written in multiple languages, pulling hairs while trying to get each of them working the way I want, etc.).

For example that javascript sort... yeah, it's very fast, but it's clearly case sensitive (which in most cases is the opposite of what's needed) despite you stating otherwise, and the function param $icase doesn't seem to do anything to change that... also it doesn't work with StringSplit-format arrays too, despite having a param switch that, jugding by its name, is supposed to help. In fact, none of the options seem to have any effect on the outcome. Maybe all that is very well possible and I just failed to make it work, but that just proves my point. So it's just a big teaser ("look how fast I am"), but not practical in general sense, requiring some conditions and decisions to set up (have the freedom to get rid of mixed case with StringUpper/StringLower or knowing for sure that the input won't be mixed case, have that custom function to split the string when building the array to begin with, etc).

[randallc]

Fair enough. If performance is the most important thing, tweaking everything depending on the situation makes sense.

I'll continue working on my own though, mostly because I want to have a convenient solution that works from one function, and which I'm familiar with because I've written it myself

(not having to browse through several files of multiple functions written in multiple languages, pulling hairs while trying to get each of them working the way I want, etc.).

For example that javascript sort... yeah, it's very fast, but it's clearly case sensitive (which in most cases is the opposite of what's needed) despite you stating otherwise, and the function param $icase doesn't seem to do anything to change that... also it doesn't work with StringSplit-format arrays too, despite having a param switch that, jugding by its name, is supposed to help. In fact, none of the options seem to have any effect on the outcome. Maybe all that is very well possible and I just failed to make it work, but that just proves my point. So it's just a big teaser ("look how fast I am"), but not practical in general sense, requiring some conditions and decisions to set up (have the freedom to get rid of mixed case with StringUpper/StringLower or knowing for sure that the input won't be mixed case, have that custom function to split the string when building the array to begin with, etc).

OK!

Thanks for pointing out the bugs; I have to get organised to check them again, but at least I have the same reasoning as you; I wrote them an dhave some motivation and familiarity.

Thanks for your help.

best, randall

[randallc]

Hi,

The jsort seems to be working fine case-insensitive for the situation (1D array) where I have used it... Can you give an example where it is not?

Best, Randall

[EDIT - later; Sorry, Yes, case-sensitive only, and i knew that before, but didn't listen to myself;

[strings case insensitive not tested yet, and will obviously be slower overall]

Array sort in scriptcontrol8 Edited February 1, 2008 by randallc

[Siao]

Well, lets say:

expandcollapse popup

#include <Array.au3>

Global $a[8] = ['','Abcd','aaaa','ToOt','','Toot']

_ArrayDisplay($a)

_JSort1Da($a, False, 1, 0, 1, Chr(2))
$a = StringSplit($a, Chr(2))

_ArrayDisplay($a)


Func _JSort1Da(ByRef $arEither2D1D, $bSkipFirst = False, $iDesc = 0, $iNumeric = 1, $icase = 1, $sSeparator = "|"); mod for numeric
    Local $code = '   function SortArray(arrArray,Case,Sep) {'
    $code &= @LF & '        arrArray=arrArray.toArray().sort();';numerical asc
    $code &= @LF & '        if (Case=0){'
    $code &= @LF & '            arrArray=bubbleSort(arrArray);'
    $code &= @LF & '        }'
    $code &= @LF & '        return arrArray.join(Sep);';
;~  $code &= @LF & '        return arrArray.join("|");';
    $code &= @LF & '      }'
;===============================================================
    $code &= @LF & 'function bubbleSort(inputArray) {'
    $code &= @LF & '    for (var i = inputArray.length - 1; i >= 0;  i--) {'
    $code &= @LF & '        for (var j = 0; j <= i; j++) {'
    $code &= @LF & '            if (inputArray[j+1].toUpperCase() < inputArray[j].toUpperCase()) {'
    $code &= @LF & '                var tempValue = inputArray[j];'
    $code &= @LF & '                inputArray[j] = inputArray[j+1];'
    $code &= @LF & '                inputArray[j+1] = tempValue;'
    $code &= @LF & '      }'
    $code &= @LF & '   }'
    $code &= @LF & '}'
    $code &= @LF & 'return inputArray;'
    $code &= @LF & '}'
;===============================================================
    Local $codeNumeric = '   function SortArray(arrArray,Case,Sep) {'
    $codeNumeric &= @LF & '         return arrArray.toArray().sort(sNumAsc).join(Sep);';numerical asc
    $codeNumeric &= @LF & '     }'
    $codeNumeric &= @LF & '   function sNumAsc(a, b) {'
    $codeNumeric &= @LF & '         return ((+a > +b) ? 1 : ((+a < +b) ? -1 : 0));'
    $codeNumeric &= @LF & '     }'
;===============================================================
;~  If Not $icase And Not $iNumeric Then $code = $codeNoCase
    If $iNumeric Then $code = $codeNumeric
;===============================================================
    Local $jvs = ObjCreate("ScriptControl")
    $jvs.language = "jscript"
    $jvs.Timeout = -1
    $jvs.addcode($code)
;~  FileWrite(@ScriptDir & "\subsortJSau3a.jvs", $code)
;~  Local $arEither2D1DSt = $jvs.Run("SortArray", $arEither2D1D, ($icase Or $iNumeric), $sSeparator);, $iIndex, $icase)
    Local $arEither2D1DSt = $jvs.Run("SortArray", $arEither2D1D, $icase, $sSeparator);, $iIndex, $icase)
    $arEither2D1D = $arEither2D1DSt
;~  $arEither2D1D = StringSplit($arEither2D1DSt, "|")
;~  _ArrayDelete($arEither2D1D,0)
;~  If Not $icase And Not $iNumeric Then
;~      Local $arEither2D1DSt = $jvs.Run("quick_sort", $arEither2D1D);, $iIndex, $icase)
;~      $arEither2D1D = StringSplit($arEither2D1DSt, "|")
;~  EndIf
    
;===============================================================
;~  If Not $icase And Not $iNumeric Then _Array1DIntroSort( $arEither2D1D);_ArraySort($arEither2D1D, $iDesc, 1)
;~  If Not $icase And Not $iNumeric Then _Array2D1DFieldSortSt($arEither2D1D);_ArraySort($arEither2D1D, $iDesc, 1)
;===============================================================
    If $iDesc Then _ArrayRev($arEither2D1D)
    $jvs = ""
    Return True
EndFunc;==>_JSort1Da
Func _ArrayRev(ByRef $avArray, $i_Base = 0, $i_ubound = 0)
    If Not IsArray($avArray) Then
        SetError(1)
        Return 0
    EndIf
    Local $tmp, $last = UBound($avArray) - 1
    If $i_ubound < 1 Or $i_ubound > $last Then $i_ubound = $last
    For $i = $i_Base To $i_Base + Int(($i_ubound - $i_Base - 1) / 2)
        $tmp = $avArray[$i]
        $avArray[$i] = $avArray[$i_ubound]
        $avArray[$i_ubound] = $tmp
        $i_ubound = $i_ubound - 1
    Next
    Return 1
EndFunc;==>_ArrayRev

The two funcs copied as is from _ArrayDisplaySortV.au3 (same result if I #include the file)

No matter if I set any of $iDesc and $icase to 0 or 1, the result is always this:

which is ascending case sensitive.

Now it's obvious why descending doesn't work - at the point when _ArrayRev is called inside _JSort1Da, there is no array, just a string. Not sure why case doesn't work though.

And another thing. Did a test with array read from file - 100000 lines x 150 chars per line, javascript sort chokes a bit, RAM usage goes way up, and finishes the job slower than my last sorter (which has even slower initialization than the first version I posted a few days ago).

Edited February 1, 2008 by Siao

[randallc]

Well, lets say:The two funcs copied as is from _ArrayDisplaySortV.au3 (same result if I #include the file)

No matter if I set any of $iDesc and $icase to 0 or 1, the result is always this:

which is ascending case sensitive.

Now it's obvious why descending doesn't work - at the point when _ArrayRev is called inside _JSort1Da, there is no array, just a string. Not sure why case doesn't work though.

And another thing. Did a test with array read from file - 100000 lines x 150 chars per line, javascript sort chokes a bit, RAM usage goes way up, and finishes the job slower than my last sorter (which has even slower initialization than the first version I posted a few days ago).

OK, I'll look again.

Thanks again for all the feedback.

Randall

[EDIT - later; Sorry, Yes, you are so right, and i knew that before, but didn't listen to myself;

[strings case insensitive not tested yet, and will obviously be slower overall]

Array sort in scriptcontrol8 Edited February 1, 2008 by randallc

[randallc]

Well, lets say:

Hi again,

So I have done some more detailed testing with 1D arrays; my conclusion is to stick with your first version for speed in 1D.

Do you see the "variant" problem as an issue?

Have you done any comparisons with 2D sorts?Oh - first tests show huge difference from mine [your second version, at least!] if there are large numbers of cols......

Best, Randall

Edited February 2, 2008 by randallc

[Siao]

Hi again,

So I have done some more detailed testing with 1D arrays; my conclusion is to stick with your first version for speed in 1D.

Do you see the "variant" problem as an issue?

Have you done any comparisons with 2D sorts?Oh - first tests show huge difference from mine [your second version, at least!] if there are large numbers of cols......

Well, the change between first and current version when writing from array to memory, even being slower, is what makes 2D sort much faster than first version. Additionally it allows using same output technique for 1D too, which doesn't destroy variant types, and I do think it is pretty important. If I sort something I expect only the order to change, not the elements... lets say, if input array consists of 3 strings and 2 number types, then the output should also be 3 strings and 2 numbers, just ordered by a given criteria. Also consider the case if the input array strings are unicode and have some non-ASCII chars. In first version, they would be forced to ansi and those chars would be lost. In second version, they would stay exactly the same. I think some speed loss is a fair price to pay. Of course, in the specific case when you know input is all strings anyway (read from file) and definitely not unicode, then using first method makes sense; but I want my sorter to be as generic and non-destructive as possible.

Anyway, the latest version is here, added numerical sort option and some tweaks.

Didn't really want to start a new topic, but it's easier for me this way, than to post in your topic and keep all the posts up to date...

Edited February 2, 2008 by Siao