ArrayUnique returns an empty array

[francoiste]

in my large project ArrayUnique returns an empty array.

note: the following code snippet does not show this error symptom when run in a "standalone" script, though.

i'm only seeing it when run as part of the large project.

; snippet for AutoIt 3.3.12.0

#include <Array.au3>

Local $aArray[1] = [0]

_ArrayAdd($aArray, 1)
_ArrayAdd($aArray, 1)
$aArray[0] = UBound($aArray, 1) - 1

_ArrayDisplay($aArray, "BEFORE Unique")

$aArray = _ArrayUnique($aArray, 0, 1, 0, 1)
If (UBound($aArray, 1) < 1) Then
    MsgBox(0, "ERROR", "Empty array!")
EndIf

_ArrayDisplay($aArray, "AFTER Unique")

i have even drilled down into the func "_ArrayUnique" and added two sanity checks there. see the two lines with MsgBox:

; walk the input array
For $i = $iBase To UBound($aArray) - 1
    If $iDims = 1 Then
        ; 1D array
        $vElem = $aArray[$i]
    Else
        ; 2D array
        $vElem = $aArray[$i][$iColumn]
    EndIf
    ; add key to dictionary
    ; NOTE: accessing the value (.Item property) of a key that doesn't exist creates the key :)
    ; keys are guaranteed to be unique
    $oDictionary.Item($vElem)
    If (Not $oDictionary.Exists($vElem)) Then
        MsgBox(0, "ERROR", "Dictionary key failed to add: " & $vElem) ;## ADDED for troubleshooting purposes
    EndIf
Next

If ($oDictionary.Count < 1) Then
    MsgBox(0, "ERROR", "Dictionary count: " & $oDictionary.Count) ;## ADDED for troubleshooting purposes
EndIf
; return the array of unique keys
If BitAND($iFlags, 1) = 1 Then
    Local $aTemp = $oDictionary.Keys()
    _ArrayInsert($aTemp, 0, $oDictionary.Count)
    Return $aTemp
Else
    Return $oDictionary.Keys()
EndIf

in my project both message boxes are kicking in.

as a result the "oDictionary.Count" is "0" and hence an empty array is returned.

anyone seen such symptoms before?

any idea how to troubleshoot further?

 

---
edit (as response to the replies so far):

i'm aware that "Array.au3" (and "_ArrayUnique", in particular) got re-written.
mainly between versions 3.3.8.1 and 3.3.10.0.

this was a "script breaking change".
i did adjust my script accordingly and am passing in the correct parameters so it matches the implementation of the 3.3.12.0 version.
i verified this by adding some tracing and sanity checking to the "Array.au3" include file, itself.

now, the speed enhancement in "ArrayUnique" mainly is because of introducing a "dictionary".
and this seems to be the root cause of my problem.
the dictionary is getting created by $oDictionary = ObjCreate("Scripting.Dictionary").

in my project it is (for some reasons) not successful when adding a key through $oDictionary.Item($vElem).
please refer the checks as outlined in above code excerpt.

for now i have implemented a custom version of "MyArrayUnique" as workaround.

so far so good.

but obviously i'd prefer to avoid this custom workaround.

when asking for "ideas how to troubleshoot further" i mean to say:
 - could it be failing when doing the "ObjCreate"?
 - if not: why would "$oDictionary.Item($vElem)" not be adding the key?

what status flags or error codes should be checked for?
the answers to those questions will serve to extend the "error handling" and "sanity checking" available in "Array.au3" itself.

Edited September 5, 2014 by francoiste

[guinness]

I see no issue.

[Gianni]

I noticed different behaviour of _arrayunique if used in AutoIt v3.3.8.1 or in newer versions

for example, it seems that the second parameter (was called $iDimension while now is called $iColumn) it was 1 based in v3.3.8.1, while is 0 based in newer versions.

Are you using the above snippet and your larger project on the same version of AutoIt?

Could you try this _arrayUnique statement in your larger project to see if works:

$aArray = _ArrayUnique($aArray, 1, 1)

Edited September 3, 2014 by Chimp

[BrewManNH]

_ArrayUnique is a total rewrite from the old version, the old one was PAINFULLY slow on large arrays.

[Gianni]

_ArrayUnique is a total rewrite from the old version, the old one was PAINFULLY slow on large arrays.

 

the differences from previous version is not only on speed, but it seems also some lack of compatibility....(?)

Edited September 3, 2014 by Chimp

[BrewManNH]

In what way, you can't just throw out a statement like that with nothing backing it up? The parameters have changed from the old version to the new, but it appears to work the same, as far as removing duplicate entries goes.

[Melba23]

Chimp,

When the new Array library was released we add this to the main changelog:

 

Changed: Re-wrote Array UDF to add 2D support and add some functions. THIS IS A SCRIPT BREAKING CHANGE.

We also added this to the "Script Breaking Changes in Recent Versions" page

 

Array UDF was re-written. Functions and/or parameters have been changed, removed or added. A detailed list can be found here.

The link leads you to a page listing the changes to each of the functions - among which you find:

 

_ArrayUnique: For consistency with all other functions $iColumn is now 0-based, not 1-based

What more do you want us to do to advertise the fact that scripts may have to be changed?

We think long and hard before breaking backwards compatibility - you should be surprised how little this happens given the huge improvements made to the language over the past few releases. In this case it seemed the sensible thing to do as all other Array functions a used a 0-based column parameter and so many others were being modified at the same time - particularly as the change to this specific function was so simple. As BrewManNH has indicated, the function itself was rewritten to be much faster so there were benefits as well as disadvantages to the change, which made the decision even easier to make.

M23

[Gianni]

ok, sorry, nothing against changes on my side.
I just try to understand the cause of OP issue

p.s.
I'm missing the "Script Breaking Changes in Recent Versions" page link
where can I found it?
thanks

[Melba23]

Chimp,

Open the Help file and select <History /Changelog>. The first line of that page reads:

 

IMPORTANT: See here for recent script-breaking changes.

The link ("See here") takes you to the <Script Breaking Changes in Recent Versions> page. Because of the universal use of the Array library we kept the link for the changes page in v3.3.12.0 as well - look under the "UDFs" section and you will see the link to the special page listing all the changes.

M23

[Gianni]

got it, thanks.

edit:

@francoiste, sorry for the little hijacking

Edited September 4, 2014 by Chimp

[francoiste]

update:

this is weird.

the following change in _ArrayUnique() seems to cure the symptom:

$oDictionary.Item( $vElem )  ;## original line

$oDictionary.Item( "" & $vElem & "")  ;## tweaked line

by prepending and appending an empty string it is kind of forcing the concatenated key to be a valid string?

this way my keys are getting added to the dictionary successfully.

and in turn the "oDictionary.Count" no longer is "0".

everything working as expected. i'm no longer getting back empty arrays.

 

what's happening behind the scenes?

any clue what effect this particular change has?

Edited September 5, 2014 by francoiste

[Melba23]

francoiste,

Interesting. MSDN tells us that " The key is used to retrieve an individual item and is usually an integer or a string, but can be anything except an array" so why does it seem necessary to force the value to a string? What datatype are the values you have in the original array?

M23

[francoiste]

it should be all integers, i think.

finally i have a repro script.

(apologies for not stripping it down, further ...)

to reproduce: make sure you have opened one additional instance of "windows explorer".

expandcollapse popup

; snippet for AutoIt 3.3.12.0

#include <Array.au3>
#include <security.au3>

Global Const $WTS_CURRENT_SERVER_HANDLE = 0

Global Const $tagWTS_PROCESS_INFO = _
        "DWORD SessionId;" & _
        "DWORD ProcessId;" & _
        "PTR pProcessName;" & _
        "PTR pUserSid"

If @AutoItX64 Then MsgBox(4096 + 16, "ERROR", "Error: Running as x64!" & @CRLF & @CRLF & "You must change the compiler options.")

;# do the test:
Local $a = _GetProcessContexts("explorer.exe")


;*****************************************
;# returns an array of unique session IDs running the specified process. first element holds the count: $arReturn[0]
Func _GetProcessContexts(Const $sProcessName)
    Local $aContexts[1] = [0]
    Local Const $aProcesses = _GetProcessList()
    If @error Or (UBound($aProcesses) <= 0) Then
        MsgBox(4096 + 48, "Error", "GetProcessContexts() failed to retrieve running processes. " & @CRLF & @CRLF & "Terminating!")
        Exit (1)
    EndIf

    For $i = 0 To UBound($aProcesses, 1) - 1
        If (0 == StringCompare($aProcesses[$i][0], $sProcessName)) Then
            _ArrayAdd($aContexts, $aProcesses[$i][2])
        EndIf
    Next
    $aContexts[0] = UBound($aContexts, 1) - 1

    _ArrayDisplay($aContexts, "Before") ;## DEBUG <-----
    $aContexts = _ArrayUnique($aContexts, 0, 1, 0, 1) ;## this is ArrayUnique  <-----
    _ArrayDisplay($aContexts, "After") ;## DEBUG  <-----

    If ((Not IsArray($aContexts)) Or (UBound($aContexts) < 1)) Then
        MsgBox(4096 + 48, "Error", "$aSessionIDs is NOT an array!") ;## DEBUG <-----
    EndIf

    Return $aContexts
EndFunc   ;==>_GetProcessContexts


;*****************************************
;# returns an array: ProcessName | ProcessID | SessionID | ProcessOwner
;# on error: returns -1, @error is set to 1
Func _GetProcessList()
    Local $i, $aRet, $ret1, $mem

    $aRet = DllCall("WTSApi32.dll", "int", "WTSEnumerateProcesses", "int", $WTS_CURRENT_SERVER_HANDLE, "int", 0, "int", 1, "ptr*", 0, "int*", 0)
    If @error Or ($aRet[0] == 0) Then
        MsgBox(4096 + 48, "Error", "Failed invoking WTSEnumerateProcesses") ;# DEBUG
        Return (SetError(1, 0, -1))
    EndIf

    Local $array[$aRet[5]][4]
    $mem = DllStructCreate($tagWTS_PROCESS_INFO, $aRet[4])

    For $i = 0 To $aRet[5] - 1
        $mem = DllStructCreate($tagWTS_PROCESS_INFO, $aRet[4] + ($i * DllStructGetSize($mem)))
        Local $string = DllStructCreate("char[256]", DllStructGetData($mem, "pProcessName"))
        $array[$i][0] = DllStructGetData($string, 1)
        $array[$i][1] = DllStructGetData($mem, "ProcessId")
        $array[$i][2] = DllStructGetData($mem, "SessionId")
        $ret1 = _Security__LookupAccountSid(DllStructGetData($mem, "pUserSid"))
        If IsArray($ret1) Then $array[$i][3] = $ret1[1] & "\" & $ret1[0]
    Next

    DllCall("WTSApi32.dll", "int", "WTSFreeMemory", "int", $aRet[4])

    Return $array
EndFunc   ;==>_GetProcessList

edit:

i'm using Win7 x64.

running (and compiling) the script as x86 (32bit).

Edited September 5, 2014 by francoiste

[Melba23]

francoiste,

Running that script shows that the elements in the array to be "uniqued" are 64-bit integers - on my machine (Win7 x32) the dictionary object crashes AutoIt when they are passed as keys. Converting them to 32-bit integers or strings gives the correct return. Are you also running a 32-bit OS?

M23

[BrewManNH]

I noticed this only happens when the array contains numbers, it seems to be stripping the numbers out of the array because the line "$oDictionary.Item($vElem)" sees .Item($vElem) as trying to access the element number of the scripting.dictionary object, rather than trying to add it to the dictionary. When that element number isn't found, it errors out, and nothing gets added.

[Melba23]

BrewManNH,

 

it seems to be stripping the numbers out of the array

Not so sure about that - as I mentioned above, converting the numbers to 32-bit integers worked fine for me.

M23

[BrewManNH]

Running as-is, it strips numbers out of the array. Converting the numbers to strings and then adding them to the array works, straight numbers do not. Probably not what we want the function to do.

[Melba23]

BrewManNH & francoiste,

Could you both please run this modified version of the function and tell me if you get the same Int64 datatype returned as I do when I run francoiste's script:

expandcollapse popup

Func _ArrayUnique_Mod(Const ByRef $aArray, $iColumn = 0, $iBase = 0, $iCase = 0, $iFlags = 1)

    If $iColumn = Default Then $iColumn = 0
    If $iBase = Default Then $iBase = 0
    If $iCase = Default Then $iCase = 0
    If $iFlags = Default Then $iFlags = 1
    ; Start bounds checking
    If UBound($aArray, $UBOUND_ROWS) = 0 Then Return SetError(1, 0, 0) ; Check if array is empty, or not an array
    ; Parameters can only be 0 or 1, if anything else return with an error
    If $iBase < 0 Or $iBase > 1 Or (Not IsInt($iBase)) Then Return SetError(3, 0, 0)
    If $iCase < 0 Or $iCase > 1 Or (Not IsInt($iCase)) Then Return SetError(3, 0, 0)
    If $iFlags < 0 Or $iFlags > 1 Or (Not IsInt($iFlags)) Then Return SetError(4, 0, 0)
    Local $iDims = UBound($aArray, $UBOUND_DIMENSIONS), $iNumColumns = UBound($aArray, $UBOUND_COLUMNS)
    If $iDims > 2 Then Return SetError(2, 0, 0)
    ; Checks the given dimension is valid
    If $iColumn < 0 Or ($iNumColumns = 0 And $iColumn > 0) Or ($iNumColumns > 0 And $iColumn >= $iNumColumns) Then Return SetError(5, 0, 0)
    ; create dictionary
    Local $oDictionary = ObjCreate("Scripting.Dictionary")
    ; compare mode for strings
    ; 0 = binary, which is case sensitive
    ; 1 = text, which is case insensitive
    ; this expression forces either 1 or 0
    $oDictionary.CompareMode = Number(Not $iCase)
    Local $vElem = 0
    ; walk the input array
    For $i = $iBase To UBound($aArray) - 1
        If $iDims = 1 Then
            ; 1D array
            $vElem = $aArray[$i]
        Else
            ; 2D array
            $vElem = $aArray[$i][$iColumn]
        EndIf
        ; add key to dictionary
        ; NOTE: accessing the value (.Item property) of a key that doesn't exist creates the key :)
        ; keys are guaranteed to be unique
        ConsoleWrite("Elem: " & $vElem & " - " & VarGetType($vElem) & @CRLF)    ; Get datatype
        $iRet = $oDictionary.Item(Int($vElem, 1))                               ; Convert to Int32
        ConsoleWrite($oDictionary.Count & @CRLF)                                ; Check count
    Next
    ; return the array of unique keys
    If BitAND($iFlags, 1) = 1 Then
        Local $aTemp = $oDictionary.Keys()
        _ArrayInsert($aTemp, 0, $oDictionary.Count)
        Return $aTemp
    Else
        Return $oDictionary.Keys()
    EndIf
EndFunc

Once I have converted the elements to Int32 one of them adds without problem.

M23

[francoiste]

@M23:

oh, yes. that's interesting.
my console output states it was "Int64". (BTW: i didn't know "VarGetType", yet).

here's now a simple, stripped-down repro script:

; snippet for AutoIt 3.3.12.0

#include <Array.au3>

If @AutoItX64 Then MsgBox(4096 + 16, "ERROR", "Error: Running as x64!" & @CRLF & @CRLF & "You must change the compiler options.")

Local $aArray[1] = [0]

_ArrayAdd($aArray, Int(1, 2)) ;# force Int64
_ArrayAdd($aArray, Int(1, 2)) ;# force Int64
$aArray[0] = UBound($aArray, 1) - 1

_ArrayDisplay($aArray, "BEFORE Unique")
$aArray = _ArrayUnique($aArray, 0, 1, 0, 1)
_ArrayDisplay($aArray, "AFTER Unique")

If (UBound($aArray, 1) < 1) Then
    MsgBox(0, "ERROR", "Empty array!")
EndIf

your "_ArrayUnique_Mod" now works on my environment just fine.

the problem with our tweaked versions of "_ArrayUnique" is:

• i made it work by converting all elements to string: $oDictionary.Item( String($vElem) )
• you made it work by converting all elements to int32: $oDictionary.Item( Int($vElem, 1) )

but you never know what type of array will be passed in.

• string or int?
• or maybe mixed types, even?

 

as workaround i can force Int32 datatype when filling my array:

$array[$i][1] = Int(DllStructGetData($mem, "ProcessId"), 1) ;## converting to Int32
$array[$i][2] = Int(DllStructGetData($mem, "SessionId"), 1) ;## converting to Int32

but "_ArrayUnique" should get some more error handling or smart logic to handle different types of input arrays - including mixed types.

Edited September 5, 2014 by francoiste

[Melba23]

francoiste,

Googling is giving me hints that the standard scripting dictionary will not accept Int64 keys - but I cannot find a definitive statement to that effect. In this case, as the values concerned are nowhere near Int64 size, so chopping off the leading bits has no effect, but I can appreciate that it might in other cases. What type of OS do you run - 32 or 64 bit?

Let me have a think about how the function might be adapted to cater for this - we might have to add an error case to indicate that values in the array to be "uniqued" are unacceptably large. Further down the road, another route might be a switch to using maps once they are past the Beta stage.

M23