ListView export speed

pixelsearch · December 5, 2019

Hi all
I would like to share with you the speed results of a few tests concerning a ListView (LV) exportation, with a question to our "DllStruct gurus" in the end, to make sure I tested this correctly.

First of all, here is the GUI aspect of a "nearly-ended" project I'm working on, where you can import a CSV file (we choose a comma delimiter in the pic below, also indicating that the 1st row of the imported file will contain the listview headers)

1973445466_ListViewtests.png.9443cf4083cb7a3b52dd7d1f3269fdc5.png

After the file is imported and the LV dynamically created, we edit "in place" the cells which need to be modified, we add, insert or suppress rows, then we're ready to export the content of the amended LV.

During these tests, we're gonna focus on the export phase to a 2D array, so you will notice how many seconds it takes to get the final 2D array filled, depending on how we're gonna fill it.

I tested 3 ways to export the LV, here is the code for this export test phase :

;============================================
Func Export_csv()

    _GUICtrlListView_GetAllTextArray($hListView, "Export (handle)")
    _GUICtrlListView_GetAllTextArray($idListView, "Export (id)")
    _GUICtrlListView_GetAllTextArray_personal($idListView, "Export (personal)")
 
EndFunc ; ==> Export_csv

;============================================
Func _GUICtrlListView_GetAllTextArray($hWnd, $Filename)

    Local $nBegin = TimerInit()
    Local $iRows = _GUICtrlListView_GetItemCount($hWnd)
    Local $iCols = _GUICtrlListView_GetColumnCount($hWnd)
    Local $aWrite[$iRows][$iCols]
    For $i = 0 To $iRows -1
        For $j = 0 To $iCols -1
            $aWrite[$i][$j] = _GUICtrlListView_GetItemText($hWnd, $i, $j)
        Next
    Next
    Local $nEnd = TimerDiff($nBegin)
    ConsoleWrite($nEnd & @CRLF)
    _FileWriteFromArray("G:\Temp\CSV Files tests\" & $Filename & " " & _
        (Int($nEnd) / 1000) & "s.txt", $aWrite, Default, Default, ",")
    $aWrite = 0

EndFunc   ;==>_GUICtrlListView_GetAllTextArray

;============================================
Func _GUICtrlListView_GetAllTextArray_Personal($idListView, $Filename)

    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ; trying this, instead of _GUICtrlListView_GetItemText()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    Local $nBegin = TimerInit()
    Local $iRows = _GUICtrlListView_GetItemCount($idListView)
    Local $iCols = _GUICtrlListView_GetColumnCount($idListView)
    Local $aWrite[$iRows][$iCols]
    Local $bUnicode = GUICtrlSendMsg($idListView, $LVM_GETUNICODEFORMAT, 0, 0) <> 0
    Local $tBuffer, $pBuffer, $tItem, $pItem
    $tBuffer = DllStructCreate(($bUnicode ? "wchar Text[4096]" : "char Text[4096]"))
    $pBuffer = DllStructGetPtr($tBuffer)
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tItem, "TextMax", 4096)
    DllStructSetData($tItem, "Text", $pBuffer)
    For $i = 0 To $iRows -1
        For $j = 0 To $iCols -1
            DllStructSetData($tItem, "SubItem", $j)
            GUICtrlSendMsg($idListView, ($bUnicode ? $LVM_GETITEMTEXTW : $LVM_GETITEMTEXTA), $i, $pItem)
            $aWrite[$i][$j] = DllStructGetData($tBuffer, "Text")
        Next
    Next
    Local $nEnd = TimerDiff($nBegin)
    ConsoleWrite($nEnd & @CRLF)
    _FileWriteFromArray("G:\Temp\CSV Files tests\" & $Filename & " " & _
        (Int($nEnd) / 1000) & "s.txt", $aWrite, Default, Default, ",")
    $aWrite = 0

EndFunc   ;==>_GUICtrlListView_GetAllTextArray_Personal

_FileWriteFromArray() found in both functions won't create a pure CSV file at all, the function is here just to make sure that the text files created are exactly the same (and they are), no matter which test created them. Also the script was exited between each test, to browse here & there, then back to the script (so cache shouldn't be re-used for the next test) . Here are the results :

621730577_ListViewtestsresults.png.c434fcb180f09ee3d8bf12690b6d758c.png

1) 1st test was based on an imported CSV file having 985 rows / 12 cols
As you can see in the pic above, on my antique computer, it took 5.8s using the handle way, 2.7s the id way, 0.9s the personal way

2) 2nd test based on an imported CSV file having 9859 rows / 12 cols
50.2s using handle way, 22.8s id way, 4.5s personal way

(I even did a 3rd test based on 36635 rows / 18 cols : 278s handle way, 132s id way, 26s personal way)

This personal way has simply been done by "reworking" 2 functions extracted from GuiListView.au3, I paste the originals below so you won't have to open GuiListView.au3 to find them :

; #FUNCTION# ====================================================================================================================
; Author ........: Paul Campbell (PaulIA)
; Modified.......: Gary Frost (gafrost)
; ===============================================================================================================================
Func _GUICtrlListView_GetItemText($hWnd, $iIndex, $iSubItem = 0)
    Local $bUnicode = _GUICtrlListView_GetUnicodeFormat($hWnd)

    Local $tBuffer
    If $bUnicode Then
        $tBuffer = DllStructCreate("wchar Text[4096]")
    Else
        $tBuffer = DllStructCreate("char Text[4096]")
    EndIf
    Local $pBuffer = DllStructGetPtr($tBuffer)
    Local $tItem = DllStructCreate($tagLVITEM)
    DllStructSetData($tItem, "SubItem", $iSubItem)
    DllStructSetData($tItem, "TextMax", 4096)
    If IsHWnd($hWnd) Then
        If _WinAPI_InProcess($hWnd, $__g_hLVLastWnd) Then
            DllStructSetData($tItem, "Text", $pBuffer)
            _SendMessage($hWnd, $LVM_GETITEMTEXTW, $iIndex, $tItem, 0, "wparam", "struct*")
        Else
            Local $iItem = DllStructGetSize($tItem)
            Local $tMemMap
            Local $pMemory = _MemInit($hWnd, $iItem + 4096, $tMemMap)
            Local $pText = $pMemory + $iItem
            DllStructSetData($tItem, "Text", $pText)
            _MemWrite($tMemMap, $tItem, $pMemory, $iItem)
            If $bUnicode Then
                _SendMessage($hWnd, $LVM_GETITEMTEXTW, $iIndex, $pMemory, 0, "wparam", "ptr")
            Else
                _SendMessage($hWnd, $LVM_GETITEMTEXTA, $iIndex, $pMemory, 0, "wparam", "ptr")
            EndIf
            _MemRead($tMemMap, $pText, $tBuffer, 4096)
            _MemFree($tMemMap)
        EndIf
    Else
        Local $pItem = DllStructGetPtr($tItem)
        DllStructSetData($tItem, "Text", $pBuffer)
        If $bUnicode Then
            GUICtrlSendMsg($hWnd, $LVM_GETITEMTEXTW, $iIndex, $pItem)
        Else
            GUICtrlSendMsg($hWnd, $LVM_GETITEMTEXTA, $iIndex, $pItem)
        EndIf
    EndIf
    Return DllStructGetData($tBuffer, "Text")
EndFunc   ;==>_GUICtrlListView_GetItemText


; #FUNCTION# ====================================================================================================================
; Author ........: Gary Frost (gafrost)
; Modified.......:
; ===============================================================================================================================
Func _GUICtrlListView_GetUnicodeFormat($hWnd)
    If IsHWnd($hWnd) Then
        Return _SendMessage($hWnd, $LVM_GETUNICODEFORMAT) <> 0
    Else
        Return GUICtrlSendMsg($hWnd, $LVM_GETUNICODEFORMAT, 0, 0) <> 0
    EndIf
EndFunc   ;==>_GUICtrlListView_GetUnicodeFormat

Now my question to the "DllStruct gurus" : when you compare the original functions and the personal _GUICtrlListView_GetAllTextArray_Personal() , is there something wrong in the personal function that could be dangerous ?

Because in the way I restructured _GUICtrlListView_GetItemText() lines, I always use the same buffer, the same pointers etc... they're all placed outside For... Next loops and are created only once, that's why the speed is so much accelerated.

And in case there is something wrong in the reworked function, isn't it strange that the exported text file is exactly the same as when the original functions are used (a software named Beyond Compare has been used to compare all files byte by byte, indicating they are exactly the same). Even the 3rd test (36635 rows / 18 cols) produces exactly the same text files of 4Mb, no matter the test that creates them.

Thanks for your expertise

Edited December 6, 2019 by pixelsearch
used ternary twice, replacing each block of 5 lines with 1

jpm · December 6, 2019

did you run them in reverse order your personal first?

Func Export_csv()

     _GUICtrlListView_GetAllTextArray_personal($idListView, "Export (personal)")
     _GUICtrlListView_GetAllTextArray($idListView, "Export (id)")
    _GUICtrlListView_GetAllTextArray($hListView, "Export (handle)")
 
EndFunc ; ==> Export_csv

;=============================

pixelsearch · December 6, 2019

Hi jpm
Very pleased to read you in this thread, your nickname is found in so many include files

After reading your message, this is what I did :
* Reboot the computer
* Commented 2 lines to keep only the personal test active :

_GUICtrlListView_GetAllTextArray_personal($idListView, "Export (personal)")
;~  _GUICtrlListView_GetAllTextArray($idListView, "Export (id)")
;~  _GUICtrlListView_GetAllTextArray($hListView, "Export (handle)")

* Run the script on 1st file (985 rows, 12 cols), re-run it on 2nd test file (9859 rows, 12 cols), here are the results just below, which are nearly the same as yesterday. But you know, even yesterday, I never run the 3 tests one after the other, I always run one test only, close the GUI and exit SCite, then do plenty of other stuff (outside AutoIt) before running the 2nd test etc...

1951451242_ListViewtestsafterreboot.png.c996ae4f92ad41fd3613d74eba21a7f0.png

* Compare files with yesterday results : same content (gladly !)
If you think there's no problem using the personal script, well I'm gonna use it very often from now on !

Talking about speed, a couple of days ago, after reading what Zedna wrote here in 2008, I did a quick test, trying to reduce the waiting time during _ArrayDisplay (this can be helpful in case you got big arrays to display) and it worked, waiting time was reduced about 50%, by simply doing this :

In ArrayDisplayInternals.au3, instead of using __ArrayDisplay_AddItem() and __ArrayDisplay_AddSubItem() to populate the listview, I replaced both of them with a local empty string, concatenated with its delimiter each time a new Item (and its subitems) were found to get a complete row, each row being added with GUICtrlCreateListViewItem(... , $idListView) then empty the string again : about 50% of waiting time disappeared before the array was displayed !

Edit : back to our message, I'm gonna do same now concerning the remaining tests, reboot computer between each test, then run one test only... though I'm pretty sure there won't be major changes compared to yesterday. The thing to avoid absolutely is to run the 3 tests one after the other.

Edited December 6, 2019 by pixelsearch

jpm · December 6, 2019

so now I need to think more deeper.

for ArrayDisplayInternals i completly agree that GUICtrlCreateListViewITem() will be more efficient. Just check the amount of extra memory.

Big array will perhaps reach some memory limit.

Cheers

pixelsearch · December 7, 2019

Just now, jpm said:

so now I need to think more deeper.

@jpm : many thanks for your thoughts
Today was a lucky day because I finally found the quickest way to import / export a CVS file, after it has been modified by a listview control. Here are the steps that led me here :

1) Import phase : _CSVSplit() by Czardas, found here :
I haven't found another function that is so reliable. We'll talk about its speed a bit later.

Melba23 indicated here to Water that, from AutoIt v3.3.12.0, _FileReadToArray() could be used to parse a CSV file into an array.
That's true... as long as there aren't any delimiters embedded in any field. But let's say your CSV file is comma delimited and an adress field contains a comma (for example 326, Highway Road) then _FileReadToArray() won't return any Array at all (error 3). Now good luck to find why, in this file full of commas delimiters. I'm not sure that the $FRTA_INTARRAYS flag (creating a 1D "array of arrays") will be enough to quickly debug what's happening in your file. Also better avoid embedded quotes in fields, or fields surrounded by quotes etc... They won't generate an error but will appear like this in the Array """Sacramento""" instead of "Sacramento"

That's why _CSVSplit() seems imho a more reliable solution, as it handles all preceding situations.

2) Export phase : _ArrayToCSV() by Czardas
Unfortunately this export function, though reliable, takes a lot of time, which made me try something else :

2) Export phase : _GUICtrlListView_SaveCSV() by Guinness, found here or there :
This one is quick as it's based on surrounding each and every listview cell with a quote, also doubling any eventual quote found within the cell. Here is a part of guinness function :

For $i = 0 To $iItemCount
    For $j = 0 To $iColumnCount
        $sReturn &= $sQuote & StringReplace(_GUICtrlListView_GetItemText($hListView, $i, $j), $sQuote, $sQuote & $sQuote, 0, 1) & $sQuote
        ...
    Next
    ...
Next

Using guinness export function, you end up with a CSV file where every field is surrounded with quotes but guess what ?
When you want to import it back with Czardas _CSVSplit(), now it takes a very long time because of all the quotes found everywhere in guinness CSV file ! That's because Czardas _CSVSplit() is very fast... as long as most fields aren't surrounded by quotes.

Facing that dilemma, I decided to rework a bit guinness function, not only to change _GUICtrlListView_GetItemText() with the "personal" function as we discussed yesterday, but to avoid having any field surrounded by quotes when it's not absolutely necessary (99% of all cases), here is the result :

;============================================
Func Export_csv()

    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ; trying this, instead of _GUICtrlListView_GetItemText()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    Local $nBegin = TimerInit()
    Local $iRows = _GUICtrlListView_GetItemCount($idListView)
    Local $iCols = _GUICtrlListView_GetColumnCount($idListView)
    Local $sWrite = "", $sText = "", $sQuote = chr(34), $sDoubleQuote = chr(34) & chr(34), $sPattern = "\R|" & $sDelimiter
    If $bHeaders1stRow = True Then ; 1st row contained headers in the imported CSV file
        $sWrite = $g_sHeadersReadLine & @CRLF
    EndIf
    Local $bUnicode = GUICtrlSendMsg($idListView, $LVM_GETUNICODEFORMAT, 0, 0) <> 0
    Local $tBuffer, $pBuffer, $tItem, $pItem
    $tBuffer = DllStructCreate(($bUnicode ? "wchar Text[4096]" : "char Text[4096]"))
    $pBuffer = DllStructGetPtr($tBuffer)
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tItem, "TextMax", 4096)
    DllStructSetData($tItem, "Text", $pBuffer)
    For $i = 0 To $iRows -1
        For $j = 0 To $iCols -1
            DllStructSetData($tItem, "SubItem", $j)
            GUICtrlSendMsg($idListView, ($bUnicode ? $LVM_GETITEMTEXTW : $LVM_GETITEMTEXTA), $i, $pItem)
            $sText = DllStructGetData($tBuffer, "Text")
            If StringRegExp($sText, $sQuote) Then $sText = $sQuote & StringReplace($sText, $sQuote, $sDoubleQuote) & $sQuote ; 1st If +++
            If StringRegExp($sText, $sPattern) And StringLeft($sText, 1) <> $sQuote Then $sText = $sQuote & $sText & $sQuote ; 2nd If +++
            $sWrite &= $sText & (($j < $iCols - 1) ? $sDelimiter : @CRLF)
        Next
    Next
    Local $nEnd = TimerDiff($nBegin)
    ConsoleWrite($nEnd & @CRLF)
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    
    Local $sFileName_Export = FileSaveDialog(...)

And now Czardas _CSVSplit() is very quick again when importing a CSV file created by guinness amended export function, while tests done this afternoon looked very promising (speed & reliability) :thumbsup:

Edited December 11, 2019 by pixelsearch
Export phase : StringRegExp() takes care of eventual quotes, delimiters, or line-breaks found within cells

jpm · December 8, 2019

Awsome work

I check carefully my conclusion you optimise carefullt the use of _GUICtrlListView_GetItemText() when use in loops

😎

pixelsearch · December 21, 2019

On 12/8/2019 at 9:52 AM, jpm said:

I check carefully my conclusion : you optimise carefully the use of _GUICtrlListView_GetItemText() when use in loops

Thank you jpm
So now the challenge was to optimize its speed outside loops, I tried something that looks promising : waiting time can be divided by 4 during a column sort on a native-created listview, depending on the functions you call (tests results : 2s vs 8s on one cvs file, 25s vs 107s on another much bigger file). Here is the detailed process to speed it up :

Func WM_NOTIFY(...)
    ...
    Case $LVN_COLUMNCLICK ; Notifies a list-view control's parent window that a column header
                          ; was clicked while the list-view control was in report mode
        ...
        _GUICtrlListView_RegisterSortCallBack($g_hListView, 1, True, "__GUICtrlListView_Sort")
        _GUICtrlListView_SortItems($g_hListView, DllStructGetData($tInfo, 'SubItem'))
        
        ; LVM_SORTITEMSEX message, found in _GUICtrlListView_SortItems() will use an
        ; application-defined comparison function to sort the items of a list-view control.
        ; The index of each item will change to reflect the new sequence.
        ...

Function __GUICtrlListView_Sort() is found in GuiListView.au3 and described as "Our sorting callback function". I paste it here so you won't have to check it in GuiListView.au3 :

; #INTERNAL_USE_ONLY# ===========================================================================================================
; Name...........: __GUICtrlListView_Sort
; Description ...: Our sorting callback function
; Syntax.........: __GUICtrlListView_Sort ( $nItem1, $nItem2, $hWnd )
; Parameters ....: $nItem1      - Param of 1st item
;                  $nItem2      - Param of 2nd item
;                  $hWnd        - Handle of the control
; Return values .: None
; Author ........: Gary Frost (gafrost)
; Modified.......:
; Remarks .......: For Internal Use Only
; Related .......:
; Link ..........:
; Example .......:
; ===============================================================================================================================
#Au3Stripper_Ignore_Funcs=__GUICtrlListView_Sort
Func __GUICtrlListView_Sort($nItem1, $nItem2, $hWnd)
    Local $iIndex, $sVal1, $sVal2, $nResult

    For $x = 1 To $__g_aListViewSortInfo[0][0]
        If $hWnd = $__g_aListViewSortInfo[$x][1] Then
            $iIndex = $x
            ExitLoop
        EndIf
    Next

    ; Switch the sorting direction
    If $__g_aListViewSortInfo[$iIndex][3] = $__g_aListViewSortInfo[$iIndex][4] Then ; $nColumn = nCurCol ?
        If Not $__g_aListViewSortInfo[$iIndex][7] Then ; $bSet
            $__g_aListViewSortInfo[$iIndex][5] *= -1 ; $nSortDir
            $__g_aListViewSortInfo[$iIndex][7] = 1 ; $bSet
        EndIf
    Else
        $__g_aListViewSortInfo[$iIndex][7] = 1 ; $bSet
    EndIf
    $__g_aListViewSortInfo[$iIndex][6] = $__g_aListViewSortInfo[$iIndex][3] ; $nCol = $nColumn
    $sVal1 = _GUICtrlListView_GetItemText($hWnd, $nItem1, $__g_aListViewSortInfo[$iIndex][3])
    $sVal2 = _GUICtrlListView_GetItemText($hWnd, $nItem2, $__g_aListViewSortInfo[$iIndex][3])

    If $__g_aListViewSortInfo[$iIndex][8] = 1 Then
        ; force Treat as Number if possible
        If (StringIsFloat($sVal1) Or StringIsInt($sVal1)) Then $sVal1 = Number($sVal1)
        If (StringIsFloat($sVal2) Or StringIsInt($sVal2)) Then $sVal2 = Number($sVal2)
    EndIf

    If $__g_aListViewSortInfo[$iIndex][8] < 2 Then
        ; Treat as String or Number
        $nResult = 0 ; No change of item1 and item2 positions
        If $sVal1 < $sVal2 Then
            $nResult = -1 ; Put item2 before item1
        ElseIf $sVal1 > $sVal2 Then
            $nResult = 1 ; Put item2 behind item1
        EndIf
    Else
        ; Use API handling
        $nResult = DllCall('shlwapi.dll', 'int', 'StrCmpLogicalW', 'wstr', $sVal1, 'wstr', $sVal2)[0]
    EndIf

    $nResult = $nResult * $__g_aListViewSortInfo[$iIndex][5] ; $nSortDir

    Return $nResult
EndFunc   ;==>__GUICtrlListView_Sort

This function can be called thousands of times during the sort process (8494 times on a cvs file containing 986 rows, or 91017 times on a cvs file containing 8160 rows, when the items of the column are all bad placed and need sort & resort), which means that 2 usual _GUICtrlListView_GetItemText() lines found in the function will be called twice more :

$sVal1 = _GUICtrlListView_GetItemText($hWnd, $nItem1, $__g_aListViewSortInfo[$iIndex][3])
$sVal2 = _GUICtrlListView_GetItemText($hWnd, $nItem2, $__g_aListViewSortInfo[$iIndex][3])

As discussed in the precedents posts, each time you call _GUICtrlListView_GetItemText(), buffers & structures & pointers need to be created at each call, which takes time when the function is called thousands of times. So I tried what we already discussed before, but this time, it's not in a loop anymore :

;=====================================================
Func WM_NOTIFY(...)
    ...
    Case $LVN_COLUMNCLICK
        ...    
        ; _GUICtrlListView_RegisterSortCallBack($g_hListView, 1, True, "__GUICtrlListView_Sort")
        _GUICtrlListView_RegisterSortCallBack($g_hListView, 1, True, "__GUICtrlListView_Sort2")
        
        _GUICtrlListView_SortItems($g_hListView, DllStructGetData($tInfo, 'SubItem'))        
        ...
EndFunc   ;==>WM_NOTIFY

;=====================================================
Func __GUICtrlListView_Sort2($nItem1, $nItem2, $hWnd)
    ...
    ; all lines are same as in __GUICtrlListView_Sort() except those 2 :
    
;~  $sVal1 = _GUICtrlListView_GetItemText($hWnd, $nItem1, $__g_aListViewSortInfo[$iIndex][3])
;~  $sVal2 = _GUICtrlListView_GetItemText($hWnd, $nItem2, $__g_aListViewSortInfo[$iIndex][3])

    $sVal1 = _Personal_LVGetItemText($nItem1, $__g_aListViewSortInfo[$iIndex][3])
    $sVal2 = _Personal_LVGetItemText($nItem2, $__g_aListViewSortInfo[$iIndex][3])
    ...
EndFunc   ;==>__GUICtrlListView_Sort2

;=====================================================
Func _Personal_LVGetItemText($iIndex, $iSubItem)

    Local Static $iCounter = 0
    $iCounter += 1

    Local Static $bUnicode = GUICtrlSendMsg($g_idListView, $LVM_GETUNICODEFORMAT, 0, 0) <> 0
    Local Static $tBuffer = DllStructCreate(($bUnicode ? "wchar Text[4096]" : "char Text[4096]"))
    Local Static $pBuffer = DllStructGetPtr($tBuffer)
    Local Static $tItem = DllStructCreate($tagLVITEM)
    Local Static $pItem = DllStructGetPtr($tItem)
    If $iCounter = 1 Then
        DllStructSetData($tItem, "TextMax", 4096)
        DllStructSetData($tItem, "Text", $pBuffer)
    EndIf
    DllStructSetData($tItem, "SubItem", $iSubItem)
    GUICtrlSendMsg($g_idListView, ($bUnicode ? $LVM_GETITEMTEXTW : $LVM_GETITEMTEXTA), $iIndex, $pItem)
    Return DllStructGetData($tBuffer, "Text")

EndFunc   ;==>_Personal_LVGetItemText

$g_idListView found in _Personal_LVGetItemText() is the native-created listview id (global variable)
Comparing files exported using __GUICtrlListView_Sort() or __GUICtrlListView_Sort2() shows no differences between them, while the sort process is 4 times quicker

Edited December 21, 2019 by pixelsearch

pixelsearch · January 5, 2020

What follows also tries to improve the speed of _GUICtrlListView_GetItemText() and _GUICtrlListView_SetItemText() ... by not calling them directly from GuiListView.au3

I'm experimenting a function _BufferCreate() that will create once only the buffer & pointers involving my Listview needs. This function will be called only once, from any function that requires it first (it could be export phase, numeric & string sorts, natural sort etc...), which means that the buffer will always stay opened during the whole script (as soon as it is created) and reused by any function that requires it.

Could it be a dangerous way of scripting ?
Plenty of tests will give the answer.

Because if I don't do that, the same following code will appear at least 3 times within the script, making it bigger & bigger and harder to maintain

As the variable $g_idListView isn't defined until a csv file is imported, then I'll have to create global variables within the function (though it may be better to define them global at the beginning of the script, then assign their values in this function, we'll see)

Global $g_bBufferExist = False
...

;============================================
Func Export_csv()
    If Not $g_bBufferExist Then _BufferCreate()
    ...
EndFunc   ;==>_Export_csv

;============================================
Func AnyFunc_ThatNeedsIt
    If Not $g_bBufferExist Then _BufferCreate()
    ...
EndFunc   ;==>AnyFunc_ThatNeedsIt

;============================================
Func _BufferCreate()

    Global $g_bUnicode = GUICtrlSendMsg($g_idListView, $LVM_GETUNICODEFORMAT, 0, 0) <> 0
    Global $g_InsertItem = ($g_bUnicode ? $LVM_INSERTITEMW : $LVM_INSERTITEMA)
    Global $g_SetItem = ($g_bUnicode ? $LVM_SETITEMW : $LVM_SETITEMA)
    Global $g_GetItemText = ($g_bUnicode ? $LVM_GETITEMTEXTW : $LVM_GETITEMTEXTA)
    Global $g_SetItemText = ($g_bUnicode ? $LVM_SETITEMTEXTW : $LVM_SETITEMTEXTA)
    Global $g_tBuffer = DllStructCreate(($g_bUnicode ? "wchar Text[4096]" : "char Text[4096]"))
    Global $g_tItem = DllStructCreate($tagLVITEM)
    Global $g_pItem = DllStructGetPtr($g_tItem)
    DllStructSetData($g_tItem, "Text", DllStructGetPtr($g_tBuffer))
    DllStructSetData($g_tItem, "TextMax", 4096)
   
    $g_bBufferExist = True

EndFunc   ;==>_BufferCreate

Exported files using this function (found in script a.au3 for example) will be compared to exported files not using it (script b.au3 for example). Let's cross fingers that files exported will be exactly the same after the same treatment has been applied to them (sort, headers drag, export etc...) . Beyond Compare will check that,

To be continued...

Edited January 8, 2020 by pixelsearch

pixelsearch · January 8, 2020

Hi everybody

Even if they're not recent, both following links are interesting if you want to know how to improve speed in your listview control :

https://www.autoitscript.com/forum/topic/67829-_guictrllistview_additem-much-slower-than-guictrlcreatelistviewitem/
https://www.autoitscript.com/forum/topic/132703-correct-way-to-clear-a-listview-solved/

Also LarsJ , in this thread, wrote what follows :

"A general approach to listviews is to create listview and listview items with native functions due to speed and then use UDF functions for everything else."

I'd like to add 2 comments based on the little experience I got with my "CSV file editor" script :

1) If the data you need to populate your listview is contained in an array, then do not use the native function GUICtrlCreateListViewItem() in a loop, but choose instead the function _GUICtrlListView_AddArray()

This function, found in GuiListView.au3 and written by Paul Campbell & Gary Frost is optimized and its speed will be 3 times faster than a loop with many GUICtrlCreateListViewItem() needed to populate the listview.

Why is it faster ?
Because there's a loop on items & subitems, the loop is included in the function and the buffer involved is opened only once. That's the reason why the next release of "CSV file editor" will be based on _GUICtrlListView_AddArray() instead of GUICtrlCreateListViewItem(), which will allow to populate the listview 3 times faster during the import phase (tests done successfully on arrays of 1000, 10000, 36000 rows) :thumbsup:

2) Now let's talk about deleting all items in a listview :
=> if you added your items with GUICtrlCreateListViewItem() then deleting thousands of items will take a lot of time because each control has to be deleted one by one in a loop, using GuiCtrlDelete() as explained by Gary Frost himself, here

=> if you added your items with _GUICtrlListView_AddArray() then deleting all items could be done in a snap, like this :

$g_idListView = GUICtrlCreateListView(...) ; native-created listview
$g_hListView = GuiCtrlGetHandle($g_idListView)
_GUICtrlListView_AddArray($g_idListview, ...) ; populating listview (fastest way)
_SendMessage($g_hListView, $LVM_DELETEALLITEMS) ; fast deletion + no id's leaks

So, if you are sure that your listview items are not native, then you don't need to call _GUICtrlListView_DeleteAllItems() . Just Send the Message $LVM_DELETEALLITEMS and avoid the loop found in _GUICtrlListView_DeleteAllItems() which would inspect each item to know if it's native or not (you know they are not !)

A worse idea would be to call _SendMessage($g_hListView, $LVM_DELETEALLITEMS) when all your items are natively created with GUICtrlCreateListViewItem() . Why is that ?

Because though deletion appears fast on your monitor, AutoIt will not delete any of the thousands of items controls id's and you'll end with plenty of handle leaks. Have a look at BrewManNH's post, which explains the situation.

And even if "we can have up to 64k control ids", imagine a user doing this in the same session :
* 1st file imported in listview, 30.000 items, using GUICtrlCreateListViewItem()
* Delete them all with _SendMessage($g_hListView, $LVM_DELETEALLITEMS), bad way
* Now Autoit would (for example) assign a control id of 30.027 for any new control created
* 2nd file imported in listview, 30.000 items, using GUICtrlCreateListViewItem()
* Delete them all... and you already reached 60.000 id's all "alive" for AutoIt

None of this happens the other way, which is :
* 1st file imported in listview, 30.000 items, using _GUICtrlListView_AddArray()
* Delete them all with _SendMessage($g_hListView, $LVM_DELETEALLITEMS), good way
* Now Autoit would (for example) assign a control id of 27 for any new control created, great
* 2nd file imported in listview, 30.000 items, using _GUICtrlListView_AddArray()
* Delete them all... and your new controls id would still start at 27, no leak.

My conclusion (for now) :
_GUICtrlListView_AddArray() rules for both reasons described above

Edited January 8, 2020 by pixelsearch

Skysnake · February 28, 2023

_GUICtrlListView_BeginUpdate($ListView)

Do not underestimate the value of wrapping ListView populate / delete functions in _GUICtrlListView_BeginUpdate and

_GUICtrlListView_EndUpdate

I had a Listview with 6000+ lines of data, took ~ 60 seconds / 1000 lines of data (very slow). Wrapping both the data add, and the _GUICtrlListView_DeleteAllItems in _GUICtrlListView_BeginUpdate reduced the total time to delete and re-populate to ~90 seconds (acceptable)

Zedna · April 6, 2023

One important general speed optimisation rule:

Use case sensitive string comparing inside loops with string operations when working with special characters like separators/quotes/spaces/numbers/etc (generally all not a-z/A-Z) -> searching/replacing/comparing.

So typically explicitly set casesense parameter to 1 ($STR_CASESENSE) in functions StringInStr() and StringReplace() and use == instead of = in comparing strings in If $var == '...' Then ...

Case sensitive searching/replacing is much faster.

Also in such simple string operations use StringInStr/StrinReplace instead of StringRegExp/StringRegExpReplace, regular expressions are internally very complex so generally slower (for simple search/replace).

Here is example of such string optimization for code from this topic (few posts above this one):

original code - only part with loop:

For $i = 0 To $iRows -1
        For $j = 0 To $iCols -1
            DllStructSetData($tItem, "SubItem", $j)
            GUICtrlSendMsg($idListView, ($bUnicode ? $LVM_GETITEMTEXTW : $LVM_GETITEMTEXTA), $i, $pItem)
            $sText = DllStructGetData($tBuffer, "Text")
            If StringRegExp($sText, $sQuote) Then $sText = $sQuote & StringReplace($sText, $sQuote, $sDoubleQuote) & $sQuote ; 1st If +++
            If StringRegExp($sText, $sPattern) And StringLeft($sText, 1) <> $sQuote Then $sText = $sQuote & $sText & $sQuote ; 2nd If +++
            $sWrite &= $sText & (($j < $iCols - 1) ? $sDelimiter : @CRLF)
        Next
Next

here is original line with not optimized string testing/manipulation inside loop:

If StringRegExp($sText, $sQuote) Then $sText = $sQuote & StringReplace($sText, $sQuote, $sDoubleQuote) & $sQuote ; 1st If +++

here is my optimized string testing/manipulation (casesense=1) inside loop:

If StringInStr($sText, $sQuote, 1) Then $sText = $sQuote & StringReplace($sText, $sQuote, $sDoubleQuote, 0, 1) & $sQuote ; 1st If +++

Edited April 6, 2023 by Zedna

ListView export speed

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members