Improve performance of TreeView creation

[water]

Hi everybody,

I'm using _GuiCtrlTreeView_Add and _GuiCtrlTreeView_AddChild to create a treeView with about 42000 items. This takes about 22 seconds.

When I use GUICtrlCreateListViewItem it only needs 5 seconds.

My understandind is that the Autoit and the UDF way of creating a TreeView can't be mixed.

Do you have any tips to improve performance of my script?

Thanks in advance

Thomas

For $i = 1 to $asTree[0]
        ; Tabelle um jeweils 10000 Einträge erweitern
        If Mod($i,50000)=0 Then ReDim $asTreeGUI[UBound($asTreeGUI,1)+50000][5]
        If Mod($i,1000)=0  Then ProgressSet($i*100/$asTree[0],StringFormat("%02i",$i*100/$asTree[0]) & "%" & " - Datensatz " & $i & " von " & $asTree[0])
        ; Pfad von den Berechtigungen trennen (beim ersten TAB)
        $iPos     = StringInStr($asTree[$i],@TAB)
        $sItem    = StringLeft($asTree[$i],$iPos-1)
        $sItemEx  = StringMid($asTree[$i],$iPos+1)
        $level    = StringInStr($sItem,"#")
        $iPos     = StringInStr($sItem,"|")
        If $level = 0 Then exitloop
        If $level = 1 Then 
            $hNode[$level] = _GuiCtrlTreeView_Add($hTree, 0, StringMid($sItem, $level+1, $iPos-$level-1))
        Else
            $hNode[$level] = _GuiCtrlTreeView_AddChild($hTree, $hNode[$level-1], StringMid($sItem, $level+1 ,$iPos-$level-1))
        Endif
        $asTreeGUI[$i][0] = StringMid($sItem, $level+1, $iPos-$level-1)
        $asTreeGUI[$i][1] = $sItemEx
        $asTreeGUI[$i][2] = $hNode[$level]
        $asTreeGUI[$i][3] = $level
        $asTreeGUI[$i][4] = StringMid($sItem, $iPos+1)
    Next
    $asTreeGUI[0][0]  = $asTree[0]
    $asTree = ""        ; Delete Array

[Zedna]

Look here: http://www.autoitscript.com/forum/index.php?showtopic=67829

Princip:

Make your own modified optimized versions _GuiCtrlTreeView_Add(), _GuiCtrlTreeView_AddChild()

take out of them all structures as global variables and create/prepare them once at start of your app.

By default this is executed many times at creation of each treeview item

Note: my link is about ListView (not TreeView) but princip is exactly the same

Times for creating 4000 ListView items in my case are folowing:

native: 110 ms

UDF: 1125 ms

UDF optimized: 160ms

so you can make it 7x faster Edited November 23, 2008 by Zedna

[water]

Hi Zedna,

thanks a lot for your support!

I was able to reduce the required time by more than 60%.

Kind regards

Thomas

[Zedna]

Hi Zedna,

thanks a lot for your support!

I was able to reduce the required time by more than 60%.

Kind regards

Thomas

Post your code (related to this optimization).

I think speed up by 60% is not as much as my speed up by 700% 85% on ListView item add :-)

Maybe we can do more ...

Edited November 24, 2008 by Zedna

[Zedna]

Here are another little optimizations:

- make ReDim $asTreeGUI[] only once --> remove: If Mod($i,50000)=0 Then ReDim $asTreeGUI[...]

- access $asTree[] array less times (copy it to helper variable if you access to the same array field more times)

- If $level = 0 Then exitloop should be at begin of FOR LOOP (to not do unneccessary parsing when we will quit the loop)

- If Mod($i,1000)=0 Then ProgressSet --> should be at end of FOR LOOP

$iCount = $asTree[0]
 ReDim $asTreeGUI[$iCount][5]
    For $i = 1 to $iCount
        ; Pfad von den Berechtigungen trennen (beim ersten TAB)
    $sItemOrig = $asTree[$i]
        $iPos     = StringInStr($sItemOrig,@TAB)
        $sItem    = StringLeft($sItemOrig,$iPos-1)
        $level    = StringInStr($sItem,"#")
        If $level = 0 Then exitloop
  
        $sItemEx  = StringMid($sItemOrig,$iPos+1)
        $iPos     = StringInStr($sItem,"|")
        If $level = 1 Then 
            $hNode[$level] = _GuiCtrlTreeView_Add($hTree, 0, StringMid($sItem, $level+1, $iPos-$level-1))
        Else
            $hNode[$level] = _GuiCtrlTreeView_AddChild($hTree, $hNode[$level-1], StringMid($sItem, $level+1 ,$iPos-$level-1))
        Endif
        $asTreeGUI[$i][0] = StringMid($sItem, $level+1, $iPos-$level-1)
        $asTreeGUI[$i][1] = $sItemEx
        $asTreeGUI[$i][2] = $hNode[$level]
        $asTreeGUI[$i][3] = $level
        $asTreeGUI[$i][4] = StringMid($sItem, $iPos+1)
  
        ; Tabelle um jeweils 10000 Einträge erweitern
        If Mod($i,1000)=0  Then ProgressSet($i*100/$iCount,StringFormat("%02i",$i*100/$iCount) & "%" & " - Datensatz " & $i & " von " & $iCount)
    Next
    $asTreeGUI[0][0]  = $iCount
    $asTree = ""        ; Delete Array

Edited November 24, 2008 by Zedna

[water]

Hi Zedna,

thanks a lot for taking the time to optimize my code.

After stripping down the UDF code and incorporating your suggestions my code now runs 61.64% faster (down from 18.441 to 7.073 seconds).

I think our users can live with that.

Thomas

expandcollapse popup

Global $asTreeGUI[50000][5]=[[1]]
    Local $hNode[50]
    Local $sItem, $sItemEx, $level
    Local $tInsert, $iMask = $TVIF_TEXT
    DllStructSetData($tInsert, "InsertAfter", $TVI_LAST)
          $tInsert = DllStructCreate($tagTVINSERTSTRUCT)
    Local $pInsert = DllStructGetPtr($tInsert)
    Local $tBuffer = DllStructCreate("wchar Text[1024]")
    Local $pBuffer = DllStructGetPtr($tBuffer)  
    DllStructSetData($tInsert, "Mask", $iMask)
    DllStructSetData($tInsert, "Text", $pBuffer)

    $iCount = $asTree[0]
    If $iCount > 50000 Then ReDim $asTreeGUI[$iCount-50000+1][5]
    For $i = 1 to $iCount
        ; Pfad von den Berechtigungen trennen (beim ersten TAB)
        $sTmp     = $asTree[$i]
        $iPos     = StringInStr($sTmp,@TAB)
        $sItem    = StringLeft($sTmp,$iPos-1)
        $sItemEx  = StringMid($sTmp,$iPos+1)
        $level    = StringInStr($sItem,"#")
        $iPos     = StringInStr($sItem,"|")
        If $level = 0 Then exitloop
        $sText = StringMid($sItem, $level+1 ,$iPos-$level-1)
        If $level = 1 Then 
            DllStructSetData($tInsert, "Parent", "")
        Else
            DllStructSetData($tInsert, "Parent", $hNode[$level-1])
        Endif
        DllStructSetData($tBuffer, "Text", $sText)
        $hNode[$level] = _SendMessage($hTree, $TVM_INSERTITEMW, 0, $pInsert, 0, "wparam", "ptr", "hwnd")
        $asTreeGUI[$i][0] = $sText
        $asTreeGUI[$i][1] = $sItemEx
        $asTreeGUI[$i][2] = $hNode[$level]
        $asTreeGUI[$i][3] = $level
        $asTreeGUI[$i][4] = StringMid($sItem, $iPos+1)
        If Mod($i,1000)=0  Then ProgressSet($i*100/$iCount,StringFormat("%02i",$i*100/$iCount) & "%" & " - Datensatz " & $i & " von " & $iCount)
    Next
    $asTreeGUI[0][0]  = $iCount
    $asTree = ""        ; Delete Array

[Zedna]

Hi Zedna,

thanks a lot for taking the time to optimize my code.

After stripping down the UDF code and incorporating your suggestions my code now runs 61.64% faster (down from 18.441 to 7.073 seconds).

I think our users can live with that.

Thomas

Your code looks good to me but I think there are possibilities to speed-up it:

- try to use GUICtrlSendMsg() instead of _SendMessage() for $TVM_INSERTITEMW inside FOR loop

- add _SendMessage($hlistview,$WM_SETREDRAW,0,0) before FOR loop

- try to remove your ProgressSet

If Mod($i,1000)=0  Then ProgressSet($i*100/$iCount,StringFormat("%02i",$i*100/$iCount) & "%" & " - Datensatz " & $i & " von " & $iCount)

If it's only for 7s then rather use:

GUISetCursor(15,1); to set HourGlass mouse cursor before your FOR LOOP and

GUISetCursor(2); to restore default mouse cursor at end

Edited November 25, 2008 by Zedna

[water]

I've removed the ProgressSet (yields a few milliseconds) and tried to replace _SendMessage() with GUICtrlSendMsg().

Unfortunately this two function calls need different numbers of arguments.

I can't translate from _SendMessage() to GUICtrlSendMsg() - because I don't know what's going on behind the covers

Do you have an idea how the _SendMessage() line should look like?

I think it would reduce execution time from 7 to below 4 seconds.

[Zedna]

I've removed the ProgressSet (yields a few milliseconds) and tried to replace _SendMessage() with GUICtrlSendMsg().

Unfortunately this two function calls need different numbers of arguments.

I can't translate from _SendMessage() to GUICtrlSendMsg() - because I don't know what's going on behind the covers

Do you have an idea how the _SendMessage() line should look like?

I think it would reduce execution time from 7 to below 4 seconds.

Post code for creating your TreeView control.

Do you use GUICtrlCreateTreeView or _GUICtrlTreeView_Create?

[water]

The whole GUI looks like:

;----
; GUI
;----
#Region ### START Koda GUI section ###
Global $hForm1              = GUICreate($sFormTitle, 942, 717, -1, -1, BitOR($WS_CAPTION, $WS_POPUP, $WS_BORDER, $WS_MAXIMIZEBOX, $WS_MINIMIZEBOX))
Global $hTree               = _GUICtrlTreeView_Create($hForm1, 8, 24, 313, 610, BitOR($TVS_HASBUTTONS, $TVS_HASLINES, $TVS_LINESATROOT, $TVS_DISABLEDRAGDROP, $TVS_SHOWSELALWAYS), $WS_EX_CLIENTEDGE)
Global $hGUIGroupList       = GUICtrlCreateListView("Gruppe|Zugriffsrechte (Verzeichnis und Dateien)", 328, 24, 601, 305,$LVS_SHOWSELALWAYS)
Global $hButtonMail         = GUICtrlCreateButton("Auswahl -> Mail", 553, 664, 120, 25)
Global $hButtonClip         = GUICtrlCreateButton("Auswahl -> Clipboard", 679, 664, 122, 25)
Global $hButtonExcel        = GUICtrlCreateButton("Auswahl -> Excel", 808, 664, 120, 25)
Global $hButtonStrgA        = GUICtrlCreateDummy()
Global $hButtonStrgC        = GUICtrlCreateDummy()
Global $hGUITreeLabel       = GUICtrlCreateLabel("", 8, 8, 309, 14)
Global $hGUIGroupLabel      = GUICtrlCreateLabel("", 328, 8, 591, 14)
Global $hGUIGroupUserList   = GUICtrlCreateListView("Name|Typ", 328, 352, 601, 282, $LVS_SHOWSELALWAYS)
Global $hGUIGroupUserLabel  = GUICtrlCreateLabel("", 328, 334, 589, 14)
Global $hGUIPathLabel       = GUICtrlCreateLabel("", 8, 640, 921, 16, $SS_SUNKEN)
Global $hButtonGroups       = GUICtrlCreateButton("Gruppen", 8, 664, 66, 25)
Global $hButtonExpand       = GUICtrlCreateButton("Expand", 80, 664, 57, 25, 0)
Global $hButtonCollapse     = GUICtrlCreateButton("Collapse", 144, 664, 57, 25)
; Menu
Global $hDateiMenu          = GUICtrlCreateMenu("&Datei")
Global $hDateiMenuOeffnen   = GUICtrlCreateMenuItem("Ö&ffnen...", $hDateiMenu)
Global $hDateiMenuBeenden   = GUICtrlCreateMenuItem("&Beenden", $hDateiMenu)
Global $hHilfeMenu          = GUICtrlCreateMenu("&?")
Global $hHilfeMenuHilfe     = GUICtrlCreateMenuItem("&Hilfe", $hHilfeMenu)
Global $hHilfeMenuUeber     = GUICtrlCreateMenuItem("&Über", $hHilfeMenu)
                              GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

; Set accelerators for Strg+a and Strg+c
Global $asAccelKeys[2][2]=[["^a", $hButtonStrgA],["^c", $hButtonStrgC]]
GUISetAccelerators($asAccelKeys)

GUIRegisterMsg($WM_NOTIFY, "WM_NOTIFY")

[Zedna]

I think there is no reason to use UDF TreeView (create) instead of AutoIt's native one:

expandcollapse popup

;Global $hTree = _GUICtrlTreeView_Create($hForm1, 8, 24, 313, 610, BitOR($TVS_HASBUTTONS, $TVS_HASLINES, $TVS_LINESATROOT, $TVS_DISABLEDRAGDROP, $TVS_SHOWSELALWAYS), $WS_EX_CLIENTEDGE)Global 
Global $TreeId = GUICtrlCreateTreeView(8, 24, 313, 610, BitOR($TVS_HASBUTTONS, $TVS_HASLINES, $TVS_LINESATROOT, $TVS_DISABLEDRAGDROP, $TVS_SHOWSELALWAYS), $WS_EX_CLIENTEDGE)
Global $hTree = GUICtrlGetHandle($TreeId)

Global $asTreeGUI[50000][5]=[[1]]
    Local $hNode[50]
    Local $sItem, $sItemEx, $level
    Local $tInsert, $iMask = $TVIF_TEXT
    DllStructSetData($tInsert, "InsertAfter", $TVI_LAST)
          $tInsert = DllStructCreate($tagTVINSERTSTRUCT)
    Local $pInsert = DllStructGetPtr($tInsert)
    Local $tBuffer = DllStructCreate("wchar Text[1024]")
    Local $pBuffer = DllStructGetPtr($tBuffer)  
    DllStructSetData($tInsert, "Mask", $iMask)
    DllStructSetData($tInsert, "Text", $pBuffer)

    $iCount = $asTree[0]
    If $iCount > 50000 Then ReDim $asTreeGUI[$iCount-50000+1][5]
    For $i = 1 to $iCount
        ; Pfad von den Berechtigungen trennen (beim ersten TAB)
        $sTmp     = $asTree[$i]
        $iPos     = StringInStr($sTmp,@TAB)
        $sItem    = StringLeft($sTmp,$iPos-1)
        $sItemEx  = StringMid($sTmp,$iPos+1)
        $level    = StringInStr($sItem,"#")
        $iPos     = StringInStr($sItem,"|")
        If $level = 0 Then exitloop
        $sText = StringMid($sItem, $level+1 ,$iPos-$level-1)
        If $level = 1 Then 
            DllStructSetData($tInsert, "Parent", "")
        Else
            DllStructSetData($tInsert, "Parent", $hNode[$level-1])
        Endif
        DllStructSetData($tBuffer, "Text", $sText)
;        $hNode[$level] = _SendMessage($hTree, $TVM_INSERTITEMW, 0, $pInsert, 0, "wparam", "ptr", "hwnd")
        $hNode[$level] = GUICtrlSendMsg($TreeId, $TVM_INSERTITEMW, $pInsert, 0)
        $asTreeGUI[$i][0] = $sText
        $asTreeGUI[$i][1] = $sItemEx
        $asTreeGUI[$i][2] = $hNode[$level]
        $asTreeGUI[$i][3] = $level
        $asTreeGUI[$i][4] = StringMid($sItem, $iPos+1)
        If Mod($i,1000)=0  Then ProgressSet($i*100/$iCount,StringFormat("%02i",$i*100/$iCount) & "%" & " - Datensatz " & $i & " von " & $iCount)
    Next
    $asTreeGUI[0][0]  = $iCount
    $asTree = ""        ; Delete Array

EDIT: correction

$hNode[$level] = GUICtrlSendMsg($TreeId, $TVM_INSERTITEMW, 0, $pInsert)

Edited November 26, 2008 by Zedna

[Zedna]

If you still need to use _GUICtrlTreeView_Create() then do this:

expandcollapse popup

Global $hTree = _GUICtrlTreeView_Create($hForm1, 8, 24, 313, 610, BitOR($TVS_HASBUTTONS, $TVS_HASLINES, $TVS_LINESATROOT, $TVS_DISABLEDRAGDROP, $TVS_SHOWSELALWAYS), $WS_EX_CLIENTEDGE)

Global $userdll = DllOpen("user32.dll")

Global $asTreeGUI[50000][5]=[[1]]
    Local $hNode[50]
    Local $sItem, $sItemEx, $level
    Local $tInsert, $iMask = $TVIF_TEXT
    DllStructSetData($tInsert, "InsertAfter", $TVI_LAST)
          $tInsert = DllStructCreate($tagTVINSERTSTRUCT)
    Local $pInsert = DllStructGetPtr($tInsert)
    Local $tBuffer = DllStructCreate("wchar Text[1024]")
    Local $pBuffer = DllStructGetPtr($tBuffer)  
    DllStructSetData($tInsert, "Mask", $iMask)
    DllStructSetData($tInsert, "Text", $pBuffer)

    $iCount = $asTree[0]
    If $iCount > 50000 Then ReDim $asTreeGUI[$iCount-50000+1][5]
    For $i = 1 to $iCount
        ; Pfad von den Berechtigungen trennen (beim ersten TAB)
        $sTmp     = $asTree[$i]
        $iPos     = StringInStr($sTmp,@TAB)
        $sItem    = StringLeft($sTmp,$iPos-1)
        $sItemEx  = StringMid($sTmp,$iPos+1)
        $level    = StringInStr($sItem,"#")
        $iPos     = StringInStr($sItem,"|")
        If $level = 0 Then exitloop
        $sText = StringMid($sItem, $level+1 ,$iPos-$level-1)
        If $level = 1 Then 
            DllStructSetData($tInsert, "Parent", "")
        Else
            DllStructSetData($tInsert, "Parent", $hNode[$level-1])
        Endif
        DllStructSetData($tBuffer, "Text", $sText)
;        $hNode[$level] = _SendMessage($hTree, $TVM_INSERTITEMW, 0, $pInsert, 0, "wparam", "ptr", "hwnd")
    $aResult = DllCall($userdll, "wparam", "SendMessage", "hwnd", $hTree, "int", $TVM_INSERTITEMW, "ptr", $pInsert, "hwnd", 0)
        $hNode[$level] = $aResult[0]
        $asTreeGUI[$i][0] = $sText
        $asTreeGUI[$i][1] = $sItemEx
        $asTreeGUI[$i][2] = $hNode[$level]
        $asTreeGUI[$i][3] = $level
        $asTreeGUI[$i][4] = StringMid($sItem, $iPos+1)
        If Mod($i,1000)=0  Then ProgressSet($i*100/$iCount,StringFormat("%02i",$i*100/$iCount) & "%" & " - Datensatz " & $i & " von " & $iCount)
    Next
    $asTreeGUI[0][0]  = $iCount
    $asTree = ""        ; Delete Array

DllClose($userdll)

EDIT:

Important is: DllOpen, DllClose

EDIT: correction

$aResult = DllCall($userdll, "hwnd", "SendMessage", "hwnd", $hTree, "int", $TVM_INSERTITEMW, "wparam", 0, "ptr", $pInsert)

Edited November 26, 2008 by Zedna

[water]

I tried your last code example - and it really runs faster: 4.3 to 7 seconds

Unfortunately it now takes GUICtrlCreateTreeView 4 seconds before it displays the GUI

So this now totals to 8.5 seconds compared to 7 seconds before.

I think I'll stick with the 7 seconds example - my users will be happy anyway.

I really would like to thank you for the time you've taken to help me with my problem!

Greatings from snowy Austria

Thomas

[Zedna]

I tried your last code example - and it really runs faster: 4.3 to 7 seconds

Unfortunately it now takes GUICtrlCreateTreeView 4 seconds before it displays the GUI

So this now totals to 8.5 seconds compared to 7 seconds before.

I think I'll stick with the 7 seconds example - my users will be happy anyway.

I really would like to thank you for the time you've taken to help me with my problem!

Greatings from snowy Austria

Thomas

Then try my last example with _GUICtrlTreeView_Create, DllOpen, DllCall, DllClose

Note: Sometimes I go for skiing to Austria (from Czech) :-)

[water]

I tried your last example but unfortunately the TreeView remains empty.

I stripped down the script code, attached a short example of the input file and the results of a test run with the example input file for _SendMessage and DLLCall.

I noticed that the results of DLLCall don't change whereas the results of _SendMessage return different values for each call.

[Zedna]

I tried your last example but unfortunately the TreeView remains empty.

I stripped down the script code, attached a short example of the input file and the results of a test run with the example input file for _SendMessage and DLLCall.

I noticed that the results of DLLCall don't change whereas the results of _SendMessage return different values for each call.

Sorry I had a bug in DllCall types/params

This is correct version:

$aResult = DllCall($userdll, "hwnd", "SendMessage", "hwnd", $hTree, "int", $TVM_INSERTITEMW, "wparam", 0, "ptr", $pInsert)

Tested on your datafile and works fine now (the same results as _SendMessage).

Post what time can you achieve with this latest version :-)

[water]

The modified (stripped down) script runs in 3.8 seconds.

The "production" script (with the addional Array processing for $asTreeGUI) takes 5.3 seconds - so this is a reduction of 70% from where we started.

Our users will be very satisfied with this results.

I really would like to thank you for your efforts!

Thomas

[Zedna]

The modified (stripped down) script runs in 3.8 seconds.

The "production" script (with the addional Array processing for $asTreeGUI) takes 5.3 seconds - so this is a reduction of 70% from where we started.

Our users will be very satisfied with this results.

I really would like to thank you for your efforts!

Thomas

I'm glad I could help.

Have a nice further AutoIt scripting :-)