Improving _ArrayDisplay speed

[pixelsearch]

Hi everybody
when using _ArrayDisplay, I found a way to drastically reduce a listview load time (and later its sorting time when clicking on a column header) . We discussed of this a year ago with @jpm in this thread and I applied it to _ArrayDisplay just now.

1) The 1st script below uses the original functions of AutoIt
It works on a 2D array of 10.000 rows & 10 columns (you may change that if you want)
When you run this 1st script, the Console will indicate the number of seconds it takes to populate the listview. Please note this number somewhere so you'll compare it later with the 2nd script's number.

expandcollapse popup

#include <Array.au3> ; original
#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Opt("MustDeclareVars", 1)
Global $hTimer

Example()

;==================================================
Func Example()

    _SplashOn("1 - Filling Array")
    Local $iRowMax = 10000, $iColMax = 10, $aArray[$iRowMax][$iColMax]
    For $i = 0 To $iRowMax - 1
        For $j = 0 To $iColMax - 1
            $aArray[$i][$j] = "Row " & $i & " / Col " & $j
        Next
    Next
    _ArrayShuffle($aArray)
    SplashOff()

    MsgBox($MB_TOPMOST, "Shuffled Array is ready", "Click to launch _ArrayDisplay original")

    _SplashOn("2 - Populating Listview")
    AdLibRegister(TimeToLoad)
    $hTimer = TimerInit()
    _ArrayDisplay($aArray, "Speed test ArrayDisplay Original")
EndFunc   ;==>Example

;==================================================
Func TimeToLoad() ; Adlib

    If WinActive("Speed test ArrayDisplay Original") Then
        ConsoleWrite(TimerDiff($hTimer) / 1000 & @lf)
        AdLibUnRegister(TimeToLoad)
        SplashOff()
    EndIf
EndFunc   ;==>Slider

;==================================================
Func _SplashOn($sFirstLine, $sSecondLine = "please wait...")

    SplashTextOn("", $sFirstLine & @CRLF & $sSecondLine, _
        250, 50, -1, -1, $DLG_NOTITLE + $DLG_TEXTVCENTER)
EndFunc   ;==>_SplashOn

2) The 2nd script below uses 2 "speed" files : ArrayDisplayInternals2.au3 and ArrayDisplayInternals_Fill.au3
You should notice that the listview appears much quicker and the Console should reflect it, maybe 2, 3 or 4 times faster.

expandcollapse popup

; Array.au3 is not used during this speed test

#include "ArrayDisplayInternals2.au3" ; increased speed (listview fill & sort)
#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Opt("MustDeclareVars", 1)
Global $hTimer

Example()

;==================================================
Func Example()

    _SplashOn("1 - Filling Array")
    Local $iRowMax = 10000, $iColMax = 10, $aArray[$iRowMax][$iColMax]
    For $i = 0 To $iRowMax - 1
        For $j = 0 To $iColMax - 1
            $aArray[$i][$j] = "Row " & $i & " / Col " & $j
        Next
    Next
    _ArrayShuffle($aArray)
    SplashOff()

    MsgBox($MB_TOPMOST, "Shuffled Array is ready", "Click to launch _ArrayDisplay2")

    _SplashOn("2 - Populating Listview")
    AdLibRegister(TimeToLoad)
    $hTimer = TimerInit()
    _ArrayDisplay2($aArray, "Speed test ArrayDisplay2 Patched")
EndFunc   ;==>Example

;==================================================
Func TimeToLoad() ; Adlib

    If WinActive("Speed test ArrayDisplay2 Patched") Then
        ConsoleWrite(TimerDiff($hTimer) / 1000 & @lf)
        AdLibUnRegister(TimeToLoad)
        SplashOff()
    EndIf
EndFunc   ;==>Slider

;==================================================
Func _SplashOn($sFirstLine, $sSecondLine = "please wait...")

    SplashTextOn("", $sFirstLine & @CRLF & $sSecondLine, _
        250, 50, -1, -1, $DLG_NOTITLE + $DLG_TEXTVCENTER)
EndFunc   ;==>_SplashOn

; #FUNCTION# ====================================================================================================================
; Author ........: randallc, Ultima
; Modified.......: Gary Frost (gafrost), Ultima, Zedna, jpm, Melba23, AZJIO, UEZ - _ArrayDisplay() renamed _ArrayDisplay2() in this speed test
; ===============================================================================================================================
Func _ArrayDisplay2(Const ByRef $aArray, $sTitle = Default, $sArrayRange = Default, $iFlags = Default, $vUser_Separator = Default, $sHeader = Default, $iMax_ColWidth = Default)
    #forceref $vUser_Separator
    Local $iRet = __ArrayDisplay_Share($aArray, $sTitle, $sArrayRange, $iFlags, Default, $sHeader, $iMax_ColWidth, 0, False)
    Return SetError(@error, @extended, $iRet)
EndFunc   ;==>_ArrayDisplay2

; #FUNCTION# ====================================================================================================================
; Author ........: Melba23
; Modified.......: copied here because Array.au3 is not used during this speed test
; ===============================================================================================================================
Func _ArrayShuffle(ByRef $aArray, $iStart_Row = 0, $iEnd_Row = 0, $iCol = -1)

    ; Fisher–Yates algorithm

    If $iStart_Row = Default Then $iStart_Row = 0
    If $iEnd_Row = Default Then $iEnd_Row = 0
    If $iCol = Default Then $iCol = -1

    If Not IsArray($aArray) Then Return SetError(1, 0, -1)
    Local $iDim_1 = UBound($aArray, $UBOUND_ROWS)
    If $iEnd_Row = 0 Then $iEnd_Row = $iDim_1 - 1
    If $iStart_Row < 0 Or $iStart_Row > $iDim_1 - 1 Then Return SetError(3, 0, -1)
    If $iEnd_Row < 1 Or $iEnd_Row > $iDim_1 - 1 Then Return SetError(3, 0, -1)
    If $iStart_Row > $iEnd_Row Then Return SetError(4, 0, -1)

    Local $vTmp, $iRand
    Switch UBound($aArray, $UBOUND_DIMENSIONS)
        Case 1
            For $i = $iEnd_Row To $iStart_Row + 1 Step -1
                $iRand = Random($iStart_Row, $i, 1)
                $vTmp = $aArray[$i]
                $aArray[$i] = $aArray[$iRand]
                $aArray[$iRand] = $vTmp
            Next
            Return 1
        Case 2
            Local $iDim_2 = UBound($aArray, $UBOUND_COLUMNS)
            If $iCol < -1 Or $iCol > $iDim_2 - 1 Then Return SetError(5, 0, -1)
            Local $iCol_Start, $iCol_End
            If $iCol = -1 Then
                $iCol_Start = 0
                $iCol_End = $iDim_2 - 1
            Else
                $iCol_Start = $iCol
                $iCol_End = $iCol
            EndIf
            For $i = $iEnd_Row To $iStart_Row + 1 Step -1
                $iRand = Random($iStart_Row, $i, 1)
                For $j = $iCol_Start To $iCol_End
                    $vTmp = $aArray[$i][$j]
                    $aArray[$i][$j] = $aArray[$iRand][$j]
                    $aArray[$iRand][$j] = $vTmp
                Next
            Next
            Return 1
        Case Else
            Return SetError(2, 0, -1)
    EndSwitch

EndFunc   ;==>_ArrayShuffle

You may also notice that sorting on a column is much faster with the 2nd script, compared to the 1st script.

Thanks for readers who intend to test this. Just place all 4 files in a folder (i.e. the 2 preceding scripts + ArrayDisplayInternals2.au3 + ArrayDisplayInternals_Fill.au3) . There is nothing to change in AutoIt folders.

Please report if something went wrong or should be improved as I just ended this today. I already know that Global variable names which were added are perfectible, because they should follow the way ArrayDisplay "names" its variables, so they won't interfere with user's variables names.

For the curious ones, the differences between the original file ArrayDisplayInternals.au3 and the "speed" file ArrayDisplayInternals2.au3 are found in 5 sections, delimited like this :

;********** Speed patch 1 start *********
...
;**********  Speed patch 1 end  *********

Here are the 2 files to download :

ArrayDisplayInternals2.au3

ArrayDisplayInternals_Fill.au3

[jpm]

Is it intended that the $ARRAYDISPLAY_TRANSPOSE is not speed up

I attach the test file and the corresponding include that can be included in the standard UDF after validation

Cheers

Jpm

test ArrayDisplay2 Patched.au3 ArrayDisplayInternals2.au3

[pixelsearch]

Hi Jpm
Glad you found some interest in this ArrayDisplay speed process.

1 hour ago, jpm said:

Is it intended that the $ARRAYDISPLAY_TRANSPOSE is not speed up

It would have required 2 more additional functions and I wanted first to see where all this was going.
If these 2 functions are not scripted & added, then the (very ?) few people using transposed 2D arrays should continue to use the actual slow way to fill their LV

Something doesn't work for me with the file you uploaded a couple of hours ago (ArrayDisplayInternals2.au3)
* The filling of the LV is speeded, that's great.
* But the sort process (after a click on a column header) isn't speeded at all.

This is because patch #5 isn't anymore in the file :

;********** Speed patch 5 start *********
;~     Local $sVal1 = __ArrayDisplay_GetItemText($hWnd, $nItem1, $__g_aArrayDisplay_SortInfo[3])
    DllStructSetData($g_tItem, "SubItem", $__g_aArrayDisplay_SortInfo[3])
    GUICtrlSendMsg($g_idListView, $g_GetItemText, $nItem1, $g_pItem)
    Local $sVal1 = DllStructGetData($g_tBuffer, "Text")

;~     Local $sVal2 = __ArrayDisplay_GetItemText($hWnd, $nItem2, $__g_aArrayDisplay_SortInfo[3])
    DllStructSetData($g_tItem, "SubItem", $__g_aArrayDisplay_SortInfo[3])
    GUICtrlSendMsg($g_idListView, $g_GetItemText, $nItem2, $g_pItem)
    Local $sVal2 = DllStructGetData($g_tBuffer, "Text")
    ;**********  Speed patch 5 end  *********

I don't know if it was intentional, you'll tell us
In case you add it again, it requires 2 variables that should be renamed ($g_GetItemText and $g_idListView)

Thanks for your interest and the time it took you to rename all these variables !

[jpm]

Certainly I miss patch5 in the merge

for the transpose do what you want perhaps less speed is enough if the 2 functions to add is to much

I include the updated file

ArrayDisplayInternals2.au3

Edited April 2, 2021 by jpm

[pixelsearch]

Hi Jpm,
Thanks for the rework. Everything works fine now, speed at its max in both cases (LV fill & LV sort)
Let's keep the transpose part as it is now, no big deal.
Have a great special week-end

 

[jpm]

Committed  for next beta

[pixelsearch]

@jpm: I notice there are new variables in Beta 3.3.15.3, for example :

Global $g_sArrayDisplay_RowPrefix = "#"

So we probably should amend 3 lines in ArrayDisplayInternals2.au3, based on what is found in the beta script :

Line 357 (comment stays comment)
;~    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, "Row " & $i, 0)
becomes
;~    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, $g_sArrayDisplay_RowPrefix & " " & $i, 0)


Lines 658 AND 692 :
DllStructSetData($__g_tArrayDisplay_Buffer, "Text", "Row " & $i)
becomes
DllStructSetData($__g_tArrayDisplay_Buffer, "Text", $g_sArrayDisplay_RowPrefix & " " & $i)

Now both examples in help file (_ArrayDisplay & _DebugArrayDisplay) will correctly show the content of 1st col (# 0 instead of Row 0)

[jpm]

Thanks, updated

[pixelsearch]

great, thanks & see you soon

[pixelsearch]

@jpm @Melba23 : could you please explain us the reason why the While loop had to be changed in __ArrayDisplay_Share though there aren't 2 windows to deal with inside the function ?

Actual version (3.3.14.5)

While 1
    $iMsg = GUIGetMsg() ; (0)
    Switch $iMsg
        Case $_ARRAYCONSTANT_GUI_EVENT_CLOSE
            ExitLoop

        Case ...
             
    EndSwitch
Wend

Beta version (3.3.15.3)

While 1
    $aMsg = GUIGetMsg($_ARRAYCONSTANT_GUI_EVENT_ARRAY) ; (1)
    If $aMsg[1] = $hGUI Then
        Switch $aMsg[0]
            Case $_ARRAYCONSTANT_GUI_EVENT_CLOSE
                ExitLoop

            Case ...

        EndSwitch
    EndIf
Wend

Maybe there was some unwanted interaction between a main GUI and an ArrayDisplay GUI (closing the main GUI does in fact close the ArrayDisplay GUI with 3.3.14.5, but now with beta version closing the main GUI won't close anything until you close explicitly the ArrayDisplay GUI first)

#include <Array.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>

Opt("GUICloseOnESC", 0)

Example()

Func Example()
    Local $aSortTest[2][3] = [["1a", "1b", "1c"], ["2a", "2b", "2c"]]

    Local $hGUI = GUICreate("Test", 200, 100, -1, 80, -1, $WS_EX_TOPMOST)
    Local $idButton = GUICtrlCreateButton("Click me", 40, 40, 100, 30)

    GUISetState(@SW_SHOW)

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                ExitLoop

            Case $idButton
                GUICtrlSetData($idButton, "Close GUI  X")
                _ArrayDisplay($aSortTest)
                GUICtrlSetData($idButton, "Click me")

        EndSwitch
    WEnd
    GUIDelete($hGUI)
EndFunc   ;==>Example

Other possible explanations : no need to test any Case if a clicked control doesn't belong to the ArrayDisplay GUI. Or maybe you encountered a more serious issue ?

Thanks for your explanations

[Melba23]

pixelsearch,

Quote

there was some unwanted interaction between a main GUI and an ArrayDisplay GUI

That is my vague recollection of why this happened - but I cannot find the precise thread in which it was discussed.

M23

[LarsJ]

If you want to improve _ArrayDisplay loading speed, why not use a virtual listview. Since data isn't stored in a virtual listview, there's no loading time and the listview shows up instantly.

Here an example with 1,000,000 rows and 10 columns. Run the code in SciTE with F5.

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GuiListView.au3>

Global $iRows = 1000000, $iCols = 10, $aArray[$iRows][$iCols]
Global $hListView

Example()

Func Example()
  ; Create array
  Local $l, $hTimer = TimerInit()
  For $i = 0 To 9
    For $j = 0 To $iRows / 10 - 1
      $l = $i * $iRows / 10 + $j
      For $k = 0 To 9
        $aArray[$l][$k] = 10 * $l + $k
      Next
    Next
    ConsoleWrite( "Rows = " & ( $i + 1 ) * $j & @CRLF )
  Next
  ConsoleWrite( TimerDiff( $hTimer ) & @CRLF ) ; Approx. 8/11 seconds on Windows 7/10

  ; Create GUI
  Local $hGui = GUICreate( "Virtual ListView", 788+20, 788+20 )

  ; Create ListView
  Local $idListView = GUICtrlCreateListView( "", 10, 10, 788, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE )
  _GUICtrlListView_SetExtendedListViewStyle( $idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT )
  $hListView = GUICtrlGetHandle( $idListView )
  For $i = 0 To $iCols - 1
    _GUICtrlListView_AddColumn( $idListView, "Col" & $i, 75 )
  Next
  GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $iRows, 0 )

  ; Register WM_NOTIFY message handler through subclassing
  Local $pNotifyHandler = DllCallbackGetPtr( DllCallbackRegister( "NotifyHandler", "lresult", "hwnd;uint;wparam;lparam;uint_ptr;dword_ptr" ) )
  DllCall( "comctl32.dll", "bool", "SetWindowSubclass", "hwnd", $hGui, "ptr", $pNotifyHandler, "uint_ptr", 0, "dword_ptr", 0 ) ; $iSubclassId = 0, $pData = 0

  ; Show GUI
  GUISetState( @SW_SHOW )

  ; Loop
  While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
        ExitLoop
    EndSwitch
  WEnd

  ; Cleanup
  DllCall( "comctl32.dll", "bool", "RemoveWindowSubclass", "hwnd", $hGui, "ptr", $pNotifyHandler, "uint_ptr", 0 ) ; $iSubclassId = 0
  GUIDelete( $hGui )
EndFunc

Func NotifyHandler( $hWnd, $iMsg, $wParam, $lParam, $iSubclassId, $pData )
  If $iMsg <> 0x004E Then Return DllCall( "comctl32.dll", "lresult", "DefSubclassProc", "hwnd", $hWnd, "uint", $iMsg, "wparam", $wParam, "lparam", $lParam )[0] ; 0x004E = $WM_NOTIFY
  Local $tNMHDR = DllStructCreate( $tagNMHDR, $lParam )
  Switch HWnd( DllStructGetData( $tNMHDR, "hWndFrom" ) )
    Case $hListView
      Switch DllStructGetData( $tNMHDR, "Code" )
        Case $LVN_GETDISPINFOW
          Local $tNMLVDISPINFO = DllStructCreate( $tagNMLVDISPINFO, $lParam )
          Local Static $tText = DllStructCreate( "wchar[50]" ), $pText = DllStructGetPtr( $tText )
          DllStructSetData( $tText, 1, $aArray[DllStructGetData($tNMLVDISPINFO,"Item")][DllStructGetData($tNMLVDISPINFO,"SubItem")] )
          DllStructSetData( $tNMLVDISPINFO, "Text", $pText )
          Return
      EndSwitch
  EndSwitch
  Return DllCall( "comctl32.dll", "lresult", "DefSubclassProc", "hwnd", $hWnd, "uint", $iMsg, "wparam", $wParam, "lparam", $lParam )[0]
  #forceref $iSubclassId, $pData
EndFunc

 

[pixelsearch]

Dev's if you read this, it's up to you if you want to rewrite _ArrayDisplay code and change it to LarsJ idea.
Good luck.

PS: thank you Melba23 for the explanation

Edited April 3, 2021 by pixelsearch

[Nine]

With a virtual ListView, how sorting (asc/desc) by column will be managed ?  (as you can tell I am not familiar with VLV)

[argumentum]

2 hours ago, Nine said:

how sorting (asc/desc) by column will be managed ?

...faster  

[LarsJ]

Easy. You generate a sort index (1d array of integers) for each column you want sorted. These sort indexes should be calculated in advance just like the array itself. When you click a column header, the sorted array is displayed instantly. In my example, there is room for 40 rows in the listview. Showing 40 rows with or without an index makes only a very small difference. The sorted array will thus be displayed as fast as the unsorted array. There aren't really any good reasons not to use a virtual listview.

True argumentum. Of course. Sorting an array of many rows directly in the listview will necessarily be slow. Even in real compiled code.

Edited April 3, 2021 by LarsJ

[pixelsearch]

@jpm: I notice these 2 lines in ArrayDisplayInternals2.au3

Case "Map"
    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, "{Map}", $iColFill)

These 2 lines are found in the Transpose part, but not in the "Regular" part. Should they be also there ?

Edited April 5, 2021 by pixelsearch

[Nine]

On 4/3/2021 at 3:28 PM, argumentum said:

...faster

Ya, Ya faster.  But when will I see an example of it ?

[pixelsearch]

@Melba23: thanks for having fixed the following fatal error, your patch is useful in many cases, for example :

#include <Array.au3>
Local $aArray[1] = [0]
_ArrayDisplay($aArray, "Title", "1:")

1) Actual version ArrayDisplayInternals.au3 (3.3.14.5)

"ArrayDisplayInternals.au3" (345) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:

$vTmp = $aArray[$i]
$vTmp = ^ ERROR

Line 138 creates this issue :

If $iItem_Start > $iItem_End Then

 

2) Beta version ArrayDisplayInternals.au3 (3.3.15.3) as found on AutoIt web site :
Line 155 fixed the issue :

If $iItem_Start > $iItem_End And $iItem_End > 0 Then

Well done MB23

[jpm]

here is as version working for Map

I am trying to work with Larj suggestion without success for the time being ...

 

ArrayDisplayInternals.au3

Edited April 5, 2021 by jpm