Why is _ArraySort so broken? Updated 1/9/22 630pm g2g

[markyrocks]

 

@Melba23

At the time I'm sure it was ground breaking stuff.  I'm more referring to the fact that it hasn't changed in all that time.  That's the embarrassing part.   I don't have the answer for every problem as far as portability and whatnot that's something that you deal with after the fact.   I know it's possible to include the bits for a dll directly in autoit code so having the dll be seamlessly integrated don't sound like that high of a bar.  

As far as dissatisfaction and us "having to life with it" that's just a defeatist mentality.   I don't have to live with anything I don't want to. In reference to the comment about using alternative methods (which has been brought up b4)  that's just allowing the language control your control paths, I program the computer it doesn't program me.  

As far as whose jumping into volunteer... 8 years ago I don't remember getting asked for help. Matter of fact everytime I broach the subject of pushing the language ahead I'm usually met with hostility.  I will admit I say some abrasive things sometimes but you got to break a couple eggs to make an omelet.    

@pixelsearch. That's most likely an error in the array math.  The resulting struct is probably getting an extra element or the array isn't being copied completely.   I'll take a look here in a min.

func ArrayToStruct($arr)     
$sz=UBound($arr)-1 
$s=DllStructCreate("int[" & $sz+1 & "]")  
For $i=1 to $sz + 1 DllStructSetData($s,1,$arr[$i-1],$i)
Next  
return $s
EndFunc

Edit actually this loop was being shorted by one.  So the above should fix that.  It previously was only going to $sz.  I should have formatted this a little better.  I'll do a couple tests tonight and make sure it gets fixed in the original post.

Edited January 3, 2022 by markyrocks

[Melba23]

markyrocks,

You just keep on feeling embarrassed for us - no-one else here does.

Quote

everytime I broach the subject of pushing the language ahead I'm usually met with hostility.  I will admit I say some abrasive things sometimes

Could it be the two things are connected? A point worth pondering...... Or could it be the fact that:

Quote

 I don't use autoit that much anymore except for trying to break it or make it do stuff that it doesn't want to

Anyway I am out of here - enjoy your coding.

M23

[pixelsearch]

6 hours ago, markyrocks said:

Edit: actually this loop was being shorted by one.  So the above should fix that.  It previously was only going to $sz.

Yes it fixed it, thanks !
This dll could be useful at times and I'm keeping it on my computer.

I have to tell that before using it the 1st time, I uploaded it to VirusTotal (it's a reflex everyone should have before using a new dll) : so no virus found (out of 66 AV) as shown in the pic below (re-checked right now), great !

I just added 2 ByRef when calling both functions, also changed some variable names in them and got rid of some + 1 ... - 1, trying to make the 2 functions "matchable when read" as most as possible (lol) but it's just cosmetic :

;=====================================
Func ArrayToStruct(ByRef $arr)
    Local $sz = UBound($arr)
    Local $struct = DllStructCreate("int[" & $sz & "]")
    For $i = 0 To $sz - 1
        DllStructSetData($struct, 1, $arr[$i], $i+1)
    Next
    Return $struct
EndFunc   ;==>ArrayToStruct

;=====================================
Func StructToArray(Byref $struct, $sz)
    Local $arr[$sz]
    For $i = 0 To $sz - 1
        $arr[$i] = DllStructGetData($struct, 1, $i+1)
    Next
    Return $arr
EndFunc   ;==>StructToArray

Just a question. Let's forget strings. Would it be hard to have the dll take care of all numeric variables (double) instead of integers only ?

If it's too hard then please forget it and we'll stick with the dll you already provided.
Thanks.

[markyrocks]

17 hours ago, pixelsearch said:

Yes it fixed it, thanks !
This dll could be useful at times and I'm keeping it on my computer.

I have to tell that before using it the 1st time, I uploaded it to VirusTotal (it's a reflex everyone should have before using a new dll) : so no virus found (out of 66 AV) as shown in the pic below (re-checked right now), great !

I just added 2 ByRef when calling both functions, also changed some variable names in them and got rid of some + 1 ... - 1, trying to make the 2 functions "matchable when read" as most as possible (lol) but it's just cosmetic :

 

Just a question. Let's forget strings. Would it be hard to have the dll take care of all numeric variables (double) instead of integers only ?

If it's too hard then please forget it and we'll stick with the dll you already provided.
Thanks.

Hey I don't blame ya for the scan the trust factor did cross my mind.  I'm not like that tho.  I'm actually glad you posted that.  

 

As far as adding ByRef etc I had already done the same and also fixed the mixed math.  In my own copy.  I haven't got around to fixing the example yet.  It's just an example/test case than a finished tool.  I have already implemented all number types in my own code so it's no problem at all to do.  I already have the shell of a udf that once the array is passed will grab the type and return an enum that will point to an associated array of strings that will punch in for int or whatever.   I'm working on much bigger things that should hopefully make this all obsolete anyways.   That's why I really haven't pushed on this.  All I can say is that I've achieved hypervisor level status in my coding game and I have some big ideas that could not only make this operation exponentially faster I could translate to more speed in many (possibly all?) areas of autoit.   That being said it might even open the door to multithreading but let's not get ahead of ourselves.    

 

Edit..I just want to clarify that I don't claim that this example is anything extraordinary or over the top ground breaking.  The proved solution and cpp is actually mind numbingly unspectacular.  

Edited January 4, 2022 by markyrocks

[pixelsearch]

48 minutes ago, markyrocks said:

The proved solution and cpp is actually mind numbingly unspectacular. 

Ok, but your result is fast as light and easily scriptable, so it can be useful at times. 2 more noticeable points :

1) There will be times where nothing needs to be returned from StructToArray(), for example when $g_array has been declared Global in main, or if the array has been declared local and passed as a 2nd ByRef parameter to the function StructToArray()

Func StructToArray(ByRef $struct)
    For $i = 0 To Ubound($g_array) - 1
        $g_array[$i] = DllStructGetData($struct, 1, $i+1)
    Next
EndFunc   ;==>StructToArray

2) There will be other times where we won't need the function StructToArray() at all, for example if the structure has been declared with elementnames and the user is confident enough in using structure names like $s.arr($i+1) instead of array names like $g_array[$i] after the sort has been done :
        

;=================================
Func ArrayToStruct(ByRef $g_array)
    Local $struct = DllStructCreate("int arr[" & UBound($g_array) & "]")

    For $i = 0 To UBound($g_array) - 1
        $struct.arr($i+1) = $g_array[$i] ; note that $struct.arr($i) = ... doesn't work (Nine found this bug, trac ticket 3838, my 201.au3)
    Next
    Return $struct
EndFunc   ;==>ArrayToStruct

;=================================
Func StructToArray(ByRef $struct) ; not a mandatory function in some cases
    For $i = 0 To Ubound($g_array) - 1
        $g_array[$i] = $struct.arr($i+1)
    Next
EndFunc   ;==>StructToArray

So thanks for your dll and have a great year 2022 

[jugador]

result using JavaScript Array sort
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort

expandcollapse popup

#include <Array.au3>
Global $ObjErr = ObjEvent("AutoIt.Error", "_ErrorHandler")

__Example1()
Func __Example1()
    Local $size = 100000
    Local $array[$size]
    GenerateArray($array)

#cs
    Local $hTimer0 = TimerInit()
    _ArraySort($array)
    ConsoleWrite("for [ " & $size & " ] time = " & TimerDiff($hTimer0) & @CRLF)
    ;for [ 10000 ] time = 501.983565939219
    ;for [ 100000 ] time = 6123.68485518709
#ce

;#cs
    Local $hTimer1 = TimerInit()
    Local $x_Result = __ArraySortJs($array, True)
    ConsoleWrite("for [ " & $size & " ] time = " & TimerDiff($hTimer1) & @CRLF)
    ;for [ 10000 ] time = 81.1680221733835 (if numeric = False)
    ;for [ 10000 ] time = 113.865363887061 (if numeric = True)

    ;for [ 100000 ] time = 978.238867813598 (if numeric = False)
    ;for [ 100000 ] time = 1291.74552402816(if numeric = True)
    ;_ArrayDisplay($x_Result)
;#ce
EndFunc

; #FUNCTION# =============================================================================
; Name...........: __ArraySortJs
; ========================================================================================
Func __ArraySortJs($o_arry, $o_Isnumeric = False)
    ;====
    Local $o_Check = ($o_Isnumeric ? 'return JsArray.sort(NumericalSort)' : 'return JsArray.sort()' )
    Local $o_CBlock = 'function GetArray(arr){' & @CRLF & _
    'var oArray = new VBArray(arr)' & @CRLF & _
    'return oArray.toArray()' & @CRLF & _
    '}' & @CRLF & _
    @CRLF & _
    'function NumericalSort(a, b){' & @CRLF & _
    'return a - b;' & @CRLF & _
    '}' & @CRLF & _
    @CRLF & _
    'function ArraySort(arr){' & @CRLF & _
    'var JsArray = GetArray(arr)' & @CRLF & _
    $o_Check & @CRLF & _
    '}'
    ;====

    ;====
    Local $o_Obj = 0
    $o_Obj = ObjCreate("ScriptControl")
    $o_Obj.Language = "JScript"
    $o_Obj.AddCode($o_CBlock)
    Local $o_GData = $o_Obj.run("ArraySort" , $o_arry)
    $o_Obj = 0
    ;====

    ;====
    Local $oDict = ObjCreate("Scripting.Dictionary")
    For $vKeys in $o_GData
        $oDict.Item($vKeys)
    Next
    Local $oValue = $oDict.Keys()
    $oDict = 0
    ;====

    Return $oValue
EndFunc

Func _ErrorHandler($oError)
EndFunc

Func GenerateArray(ByRef $a)
  For $i = 0 To UBound($a) - 1
    $a[$i] = Random(-1000000, +1000000, 1)
  Next
EndFunc   ;==>GenerateArray

 

[markyrocks]

@jugadorNice job bro.  What kinda hardware are you running on?  That does have something to do with the time aswell.  Time across different machines are going to vary wildly.  This computer has needed a wipe for long time.   Probably has more malware running in the kernel than system files lol.  Anyways. This test case isn't even remotely representative of what I want the final solution to be.  I honestly wanted to abandon this script completely but @pixelsearch asked me to add in double/float for shitz and giggles.  I'm pretty confident that I can virtually eliminate autoit from the equation for the most part to where all a user would have to do is pass a variable and params to a wrapper function and the dll would take care of everything else.   Its going to be epic.   Even the updated .cpp is like amateur hour compared to the solution i have in mind.

[markyrocks]

During my travels I've kinda stumbled upon the reason at least in part of the performance related problems.  It appears that something like a 4 byte int takes up 48 bytes of virtual memory in autoit.    I'm assuming it's in some type of object and the other data is like associated information related to the fact its an int but that would kinda explain why everything takes 10x as long to do.  Bc instead of copying a 4 byte in its copying 48 bytes instead.... 🤯  so that being said it may take up considerably more space on 64 bit.  Haven't looked into that yet.

Edited January 8, 2022 by markyrocks

[LarsJ]

Before you get too well started, you might want to read this post very carefully.

[argumentum]

On 1/1/2022 at 11:48 PM, markyrocks said:

anyways I'll post the test script here and attach the source.cpp and

Would you compress all the files ( and add the DLL itself too )  into one ZIP file. Thanks.

Edited January 8, 2022 by argumentum

[markyrocks]

On 1/8/2022 at 7:03 AM, LarsJ said:

Before you get too well started, you might want to read this post very carefully.

Ya the thread where you insisted I was some kinda crazy person who had no idea what i was talking about.  lol.  you'll really like this test.  It returns the actual pointer to the data element of an autoit type variable.  Only tested with ints at this point the structure for strings is a little different but its a true pointer to dereference vs some kinda hacked work around.  

Like it or not this is how progress is made. 

expandcollapse popup

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Change2CUI=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
;~ #AutoIt3Wrapper_Change2CUI=y
#include <Array.au3>

Global Enum $INT32_,$INT64_,$FLOAT_,$DOUBLE_,$HANDLE_,$POINTER_,$HWND_,$STRUCT_,$BINARY_,$BOOL_,$STRING_,$ARRAY_
Global Const $TYPE_[12]=["Int_32","Int_64","Float","Double","Handle","Pointer","Hwnd","Struct","Binary","Bool","String","Array"]
Global $size = 5
Global $array[$size]
;~ GenerateArray($array)

Global $f = 0xFFFFFFF ;<~~~~~~~$f
ArraysortTest()

func ArraysortTest()


$dll=DllOpen(@ScriptDir & "\autoitdll.dll")
if not $dll then Exit

        $result = DllCall($dll,"handle:cdecl","memscan","int",0);

        if IsArray($result) Then
            $fptr=$result[0] ;<~~~~~~~~~~actual pointer to $f.
            ConsoleWrite("sucess" & @CRLF)
        Else
        ConsoleWrite("faile")
        EndIf

        $str=DllStructCreate("int",$fptr)
        ConsoleWrite(DllStructGetData($str,1)) ;<~~to show the current value should be 0xfffffff or a really big number
        DllStructSetData($str,1,1776)  ;<~~ this should change $f as outputted below.

DllClose($dll)

$i=0
while $f
    ConsoleWrite($f & @CRLF)
    sleep(1000)
    $i+=1
    if $i=10 Then
        DllStructSetData($str,1,0) ;<~~to break
    EndIf
    WEnd

EndFunc

@argumentum  and everyone else whose interested, this is completely different than the files on the first post so the original sort functions won't work.

edit edit.  This test might not seem relevant to the subject at hand but to avoid the copy and copy back this is kinda the first step that needs to be taken.  I'm not going to get into a lengthy explanation as to the whys but take my word for it.  Anyways this works as expected on my machine.  I'm on windows 10 64 but autoit is set to run everything on 32.  If it doesn't work for you then it may have something to do with your settings.  

 

Edited January 14, 2022 by markyrocks

[markyrocks]

11 hours ago, argumentum said:

Would you compress all the files ( and add the DLL itself too )  into one ZIP file. Thanks.

Sorry lol I totally missed the fact that the dll wasn't included.  Wow duh.  Anyways the files on the first page have been updated.  Should hopefully work.  Like I said tho that's just like the demonstration portion.  It may be useful in limited circumstances but the main event is yet to come. 

 

As a side note the more i dig ... the more I just shake my head.  I can't elaborate bc its a secret. ssshhhhh

[argumentum]

On 1/3/2022 at 3:56 AM, Melba23 said:

... the way he implemented arrays in AutoIt - a major factor in the slowness of their manipulation ...

@markyrocks, there's the "AutoIt issue tracker" and if presented a problem along with a solution, the solution is evaluated and processed "according with the guidelines of the elders**", and in my experience, I learned to respect their decisions.
Since you can CPP, do open a track with a reference to this thread and a function on C to be adapted to the stub. If the integration does not gravely affect the legacy code, I'm sure it'll have a good chance of adoption. Attitude affects these outcomes, so a humble state of clarity over the presentation is important. Let it run it's course and in due time it will be attended to.

Thank you for bringing attention to this. This sorting function as a script/UDF, is a slow process. An internal function would be quite welcomed.
( Trivia time: back in v3.3.8.1 there was _FileReadToArray() but no internal FileReadToArray() but now there is  )

PS: Again, the product is AutomateIt and flexible/able as it is, was never meant to be anything else other than that.

[pixelsearch]

@markyrocks : thanks for the update in the initial post of 1st page (which includes a very rapid sort on 1D arrays of floats, doubles etc...)

I had to change 1 line in the updated AutoIt script or the sort goes wrong when it comes to integers (all values returned = 0)

Your update :

Func EnumToType($Enum)
    if $Enum>0 And $Enum<12 Then
        return $TYPE_[$Enum]
    endIf
    return -1
EndFunc

ConsoleWrite would show something goes wrong :

$sType = 0   EnumToType($sType) = -1

This should fix it :

Func EnumToType($Enum)
    if $Enum>=0 And $Enum<12 Then ; 0 for $INT32
        return $TYPE_[$Enum]
    endIf
    return -1
EndFunc

ConsoleWrite would then correctly display :

$sType = 0   EnumToType($sType) = int

And the integer values returned are now correctly sorted.

Note: in case someone wants to display the results (unsorted, then sorted) using _ArrayDisplay, please consider using the last beta version of _ArrayDisplay which uses a virtual listview. It will display the 100.000 elements at the speed of light (thanks to @LarsJ for the initial idea and @jpm for the coding)

[markyrocks]

@pixelsearch nice catch.  Its been updated.  

[markyrocks]

Ok So a slight update.  Been working on this all weekend so its good to see some kinda solid result.  This may or may not work for you i'd honestly like to hear if it works as intended.  This current test only sorts ints that being said it will do so on every array in the script lol (at least I believe so haven't tested that).  umm its just kinda a byproduct of the way its currently written.   it will have no effect on an array of strings.  if you make an array of doubles it will just return a bunch of garbage.  

Also this will not work on a compiled autoit script.  Don't ask me why I haven't gotten that far yet.  I'm sure its nothing that can't be sorted. 

On my machine it sorts an array of 750000 ints in 2.5 seconds.  which is faster than the previous examples of 100000 that required being copied to a struct and then back.  

One million seems to break it.  Probably exceeds the limits of a std::vector.  I just tested 100000 with this test and on my machine it takes 300 ms which as I said previously is roughly 200 times faster than arraysort?   66000 vs 300... not too bad.

That being said even something as simple as copying an array from one to another could benefit from something like this.  This is why I been talking about blowing doors off... This example fits into that category.   

Again I'm on windows 10 64 but autoit should be running everything in 32. 

Thanks and good luck.  Fingers crossed

 

Edit:: I'm also open to ideas how TO go about targeting a specific array.  I grab them in order of being created so should I simply use an index or some other kinda identifier?

 

 

Edited January 14, 2022 by markyrocks
I really must be an idiot....

[pixelsearch]

Hi markyrocks
Just downloaded the autoitdll beta .01.zip from your last post and ran the dummyscript.au3  found in it
I tried it on 10 elements (Global $size = 10) but it returned an @error 3 ("function" not found in the DLL file) :

$result = DllCall($dll,"handle:cdecl","findArray","int",5,"int",0);
If @error Then Exit Msgbox(0, "DllCall", "error " & @error & " occured") ; added that line

I notice there is a zip file within the zip (autoitdll.zip within autoitdll beta .01.zip)
Both dll's in the zip files are the same.
But both dllmain.cpp aren't the same at all : one contains the function findArray, the other one doesn't.

Could it be possible that the dll needs to be recompiled with the most recent dllmain.cpp (the one containing the function findArray ?

[markyrocks]

3 hours ago, pixelsearch said:

Hi markyrocks
Just downloaded the autoitdll beta .01.zip from your last post and ran the dummyscript.au3  found in it
I tried it on 10 elements (Global $size = 10) but it returned an @error 3 ("function" not found in the DLL file) :

$result = DllCall($dll,"handle:cdecl","findArray","int",5,"int",0);
If @error Then Exit Msgbox(0, "DllCall", "error " & @error & " occured") ; added that line

I notice there is a zip file within the zip (autoitdll.zip within autoitdll beta .01.zip)
Both dll's in the zip files are the same.
But both dllmain.cpp aren't the same at all : one contains the function findArray, the other one doesn't.

Could it be possible that the dll needs to be recompiled with the most recent dllmain.cpp (the one containing the function findArray ?

lol let me take A look I'm not good with the file distribution part... wth...

edit fixed should be g2g sry just was excited to let it drop

 

Edit: I think I'm pretty confident how to get this current example working with compiled script (not a big deal) .  Besides that I'm already optimizing this to get rid of the while loops and checking for bounds.  That was kinda a hack just get get things moving.   I'm finding new information that makes that all obsolete.  

 

EDIT EDIT...sry I put the new test on the first page.  OMG.  I fixed and put the new test on the above post.  Going to fix the files on the first page now.  These files are the least of my concerns.  1/9/22 930pm 

Edited January 10, 2022 by markyrocks

[markyrocks]

This is going to be my last update for at least 12-24 hrs.  So I was able to get this current iteration to work with compiled scripts and I've tested it up to 2million elements. I don't think going much higher makes much sense but It seems to only fail at that level if I don't give it a few seconds between tests and make sure all of the autoit icons in the tray are cleared.  If you're going to test it like that maybe do a fresh restart and make sure no other windows are open. 

For 200000 elements......

The doors are off the trunk lid is up,  I got to walk back down the road to grab the bumper... 

 

It would actually be a little faster but I have to iterate through the array to do an actual count of the elements and cross reference that with the capacity.  Due to the nature of this type of programming if those numbers don't align then I have to realign and realloc an allocated block of memory in my dll otherwise it results in dereferencing a null pointer.  I'm careful not to do that but the Qsort function isn't smart enough apparently.  I really like that function realign and realloc.  Rolls off the tongue.  

I guess I should mention that this new method works with all number types now, I havn't dealt with strings yet but I have somewhat of a clear path to that.  Really just more of the same.  I laid the ground work for my own variant type I know that @Lars loves him some variants.  I'm assuming in the future I will need it if I'm going to manipulate arrays of mixed types.  This will obviously be slower but not a ton.  I need to think about that a bit. Might be useful for other reasons down the line.    I'm pretty confident that I should be able to grab and array of multiple types and sort it based on a row or col as long as the type thoughout that row or column is the same.  Anyways, food for thought.  I'd still like to hear about ideas for how to identify a particular array.  If it has a unique size? index in the script based on where it was declared? (using redim may shift it around in memory so that might not be the best answer) .  first element could have a code that I can key in on? It really doesn't matter to me.  

Are these tests working for other people? I've heard about a couple hickups.  I believe I got the files in order this time around.  Idk.  Heads spinning.  Have a good one thanks.

 

Edited January 14, 2022 by markyrocks

[markyrocks]

Just a quick update,  I wasted alot of time last night jerkin my gerkin without a whole lot to show for it.  I cleaned up a bunch of stuff and in the process of attempting to add more features ended up boogering it up pretty good.  I did put in the ability to single out a particular array which is a step forward.   That all gets handled in my script side function.  A major development was that this morning I was able to dereference a pointer without using readprocessmemory(). 🎉🎆😎.  I know crazy right.   It's actually a much bigger deal than it seems on the surface bc it will drastically simplify things and greatly improve efficiency.   Rn I have alot of overhead and I can get rid of most of it due to this fact.  The other big part is that this opens the door to pointer swaps instead of doing everything by value.  

The only thing I have to add is thank God bc damn all that read and write to process memory is super annoying and really clutters up the place.   No files this time but I should have something worth looking at tonight.   I'm probably going to stop including the source bc im getting to the point that if the devs want it they probably going to have to pay me for it.  I probably showed too much as it is.