Virtual listviews for huge number of rows

[LarsJ]

Don't use WM_SETREDRAW messages in a virtual listview. WM_SETREDRAW messages are used in a standard listview to maintain listview storage e.g. to insert a large number of rows into the listview. In a virtual listview, there is no storage and therefore there is no need to use WM_SETREDRAW messages.

The best way to update a virtual listview each time a new row is added is

GUICtrlSendMsg( $gListview, $LVM_SETITEMCOUNT, $iRows, 0 ) ; Note $iRows
GUICtrlSendMsg( $gListview, $LVM_ENSUREVISIBLE, $iRows-1, 0 ) ; Note $iRows-1

 

[VAN0]

Thanks.

The only reason I'm using WM_SETREDRAW is because of column autosize and last column resize to fill up empty space

[Iczer]

I wanted to adapt this example to virtual ListView where thumbnails would be stored in db, but i do not know method to dynamically replace ImageList in ListView and also - is it even good idea to have huge ImageList? It's size would be (5...8) x Rows in virtual ListView. Is it possible to set images to ListView cells without use ImageList?

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <GuiImageList.au3>
#include <GuiListView.au3>
#include <WinAPI.au3>
#include <WindowsConstants.au3>
#include <GDIplus.au3>
#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>

Global $iGUIWidth, $iGUIHeidth, $iGUILeft, $iGUIRight
Global $iEntryWidth, $iEntryHeidth
Global $aJpgList

Local $aJpgList = get_list_jpg()


Example($aJpgList)



Func get_list_jpg()
    ; List all the files and folders in the desktop directory using the default parameters.
    Local $aFileList = _FileListToArrayRec (@ScriptDir & "\Pic\", "*.jpg", $FLTA_FILES, $FLTAR_NORECUR, $FLTAR_SORT, $FLTAR_NOPATH)
    
    If @error = 1 Then
        MsgBox($MB_SYSTEMMODAL, "", "Path was invalid.")
        Exit
    EndIf
    If @error = 4 Then
        MsgBox($MB_SYSTEMMODAL, "", "No file(s) were found.")
        Exit
    EndIf
    ; Display the results returned by _FileListToArray.
   ;_ArrayDisplay($aFileList, "$aFileList")
    Return($aFileList)
EndFunc



Func Example($aJpgList)
    _GDIPlus_Startup()
    Local $idListview, $hImage
    
    $iGUIWidth = 988
    $iGUIHeidth = 1210
    $iGUILeft = -1
    $iGUIRight = -1
    
    $iEntryWidth = 198
    $iEntryHeidth = 288
    
    $iPicWidth = 190
    $iPicHeidth = 285
    
    $iColumnCount = Round($iGUIWidth/$iEntryWidth)
    $iGUIWidth = $iColumnCount * $iEntryWidth + 16 + 14
    
    GUICreate("JpegList", $iGUIWidth, $iGUIHeidth, -1, -1,  $WS_OVERLAPPEDWINDOW)
    $idListview = GUICtrlCreateListView("", 2, 2, $iGUIWidth - 4, $iGUIHeidth - 32, BitOR($LVS_SHOWSELALWAYS, $LVS_NOSORTHEADER, $LVS_REPORT))
    _GUICtrlListView_SetExtendedListViewStyle($idListview, BitOR($LVS_EX_GRIDLINES, $LVS_EX_DOUBLEBUFFER, $LVS_EX_INFOTIP, $LVS_EX_SUBITEMIMAGES))
    GUISetState(@SW_SHOW)

    ; Load images
    $hImage = _GUIImageList_Create($iPicWidth, $iPicHeidth)
    
    For $i = 1 To $iColumnCount
        _GUICtrlListView_AddColumn($idListview, "Image", $iEntryWidth)
    Next
    
    _GUICtrlListView_SetImageList($idListview, $hImage, 1)

    Global $GDIpBmpLarge
    Global $GDIpBmpResized = 0
    Global $GDIbmp
    
    $iIndex = 0
    $iSubItem = 0
    
    $iPic_RatioMax = $iPicWidth/$iPicHeidth
    
    For $r = 1 To UBound($aJpgList)-1
        local $sFileName = @ScriptDir&"\Pic\"&$aJpgList[$r]
        
        $GDIpBmpLarge = _GDIPlus_BitmapCreateFromFile($sFileName) ;GDI+ image!
        $aDim = _GDIPlus_ImageGetDimension($GDIpBmpLarge)
        
        if $aDim[0] <> $iPicWidth Or $aDim[1] > $iPicHeidth Then
            
            $iPic_Ratio = $aDim[0]/$aDim[1]
            
            Select
                Case $iPic_Ratio > $iPic_RatioMax
                    $iPic_WNew = $iPicWidth
                    $iPic_HNew = Round($iPicWidth/$iPic_Ratio,0)

                Case $iPic_Ratio < $iPic_RatioMax
                    $iPic_WNew = Round($iPicHeidth*$iPic_Ratio,0)
                    $iPic_HNew = $iPicHeidth
                    
                Case $iPic_Ratio = $iPic_RatioMax
                    $iPic_WNew = $iPicWidth
                    $iPic_HNew = $iPicHeidth

            EndSelect
            
            $GDIpBmpResized = _GDIPlus_ImageResize($GDIpBmpLarge, 190,$iPic_HNew) ;GDI+ image
            $GDIbmp = _GDIPlus_BitmapCreateHBITMAPFromBitmap($GDIpBmpResized) ;GDI image!
            $imgInd = _GUIImageList_Add($hImage, $GDIbmp)
        Else
            $GDIbmp = _GDIPlus_BitmapCreateHBITMAPFromBitmap($GDIpBmpLarge) ;GDI image!
            $imgInd = _GUIImageList_Add($hImage, $GDIbmp)
        endif
        
        If $iSubItem = 0 Then
            _GUICtrlListView_AddItem($idListview, $aJpgList[$r], $imgInd)
        Else
            _GUICtrlListView_AddSubItem($idListview, $iIndex,  $aJpgList[$r], $iSubItem, $imgInd)
        EndIf
        
        $iSubItem += 1
        
        If $iSubItem > ($iColumnCount-1) Then
            $iIndex += 1
            $iSubItem = 0
        EndIf
        
        ;cleanup resources
        _GDIPlus_BitmapDispose($GDIpBmpLarge)
        If $GDIpBmpResized <> 0 Then
            _GDIPlus_BitmapDispose($GDIpBmpResized)
            $GDIpBmpResized = 0
        EndIf
        _WinAPI_DeleteObject($GDIbmp)
        
    Next
    
    Do
    Until GUIGetMsg() = $GUI_EVENT_CLOSE

    _GDIPlus_Shutdown()
    GUIDelete()
EndFunc   ;==>Example

 

[Lion66]

Hi  LarsJ.

I'm trying to combine in one script incremental search and sorting on all columns.

Is it possible?

In the first variant (my file Search-or-Send.au3), I do not know how to combine both $sitem options in one Case $LVN_GETDISPINFOW.

It works either line 180 or line 182.

In the second variant (SortBadIndex.au3) after using the search, sorting displays incorrect data, because indices have changed.

How to fix the script?

Thank you.

SortBadIndex.au3 Search-or-Send.au3

[pixelsearch]

@Lion66 In case it may help you, here is a script that combines a virtual listview with incremental search & sorting on all columns :

expandcollapse popup

Opt( "MustDeclareVars", 1 )

#include <EditConstants.au3>
#include <GuiListView.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "RandomArray.au3" ; LarsJ

Global $g_iRows = 1000, $g_iCols = 6, $g_aArray, $g_aSubArray, $g_aIndex[$g_iRows], $g_aSubIndex[$g_iRows]
Global $g_hListView, $g_hEdit, $g_idEditSearch, $g_iSortDir, $g_iSearch = $g_iRows

Example()

Func Example()
    ; Generate Array & one index
    _Generate_All($g_aArray, $g_aIndex)
    $g_aSubArray = $g_aArray
    $g_aSubIndex = $g_aIndex

    ; Create GUI
    Local $hGui = GUICreate( "Virtual ListView + Sort + Incremental search (2d)", 630+20, 788+30+20 )

    ; Create Edit control
    GUICtrlCreateLabel( "Search", 10, 10, 50, 20 , BitOr($SS_CENTERIMAGE, $SS_CENTER) )
    Local $idEdit = GUICtrlCreateEdit( "", 70, 10, 170, 20, BitXOR( $GUI_SS_DEFAULT_EDIT, $WS_HSCROLL, $WS_VSCROLL ) )
    $g_hEdit = GUICtrlGetHandle( $idEdit )
    $g_idEditSearch = GUICtrlCreateDummy()

    ; Create ListView
    Local $idListView = GUICtrlCreateListView( "", 10, 40, 630, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE )
    _GUICtrlListView_SetExtendedListViewStyle( $idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT )
    $g_hListView = GUICtrlGetHandle( $idListView )
    Local $hHeader = _GUICtrlListView_GetHeader( $idListView )
    Local $aCols = [ "Strings", "Integers", "Floats", "Dates", "Times", "R/C" ]
    Local $aWidths = [ 230, 60, 120, 70, 60, 60 ]
    For $i = 0 To $g_iCols - 1
        _GUICtrlListView_AddColumn( $idListView, $aCols[$i], $aWidths[$i] )
    Next

    ; Sorting information
    $g_iSortDir = 0x0400 ; $HDF_SORTUP
    Local $iColumn = -1, $iColumnPrev = -1

    GUIRegisterMsg( $WM_COMMAND, "WM_COMMAND" )
    GUIRegisterMsg( $WM_NOTIFY, "WM_NOTIFY" )

    GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iRows, 0 )

    ; Show GUI
    GUISetState( @SW_SHOW )

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                ExitLoop

            Case $g_idEditSearch
                Local $sSearch = GUICtrlRead( $idEdit )
                If $sSearch = "" Then
                    ; Empty search string, display all rows
                    $g_aSubArray = $g_aArray
                    $g_aSubIndex = $g_aIndex
                    $g_iSearch = $g_iRows
                Else
                    ; Find rows matching the search string
                    $g_iSearch = 0
                    For $i = 0 To $g_iRows - 1
                        ; If StringInStr( $g_aArray[$i][0], $sSearch ) Then ; Normal search
                        If StringRegExp( $g_aArray[$i][0], $sSearch ) Then ; Reg. exp. search
                            For $j = 0 To $g_iCols - 1
                                $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
                            Next
                            $g_aSubIndex[$g_iSearch] = $g_iSearch
                            $g_iSearch += 1
                        EndIf
                    Next
                EndIf
                ConsoleWrite( StringFormat( "%4d", $g_iSearch ) & " rows matching """ & $sSearch & """" & @CRLF )
                GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0 )

            Case $idListView ; Sort
                $iColumn = GUICtrlGetState( $idListView )
                If $iColumn <> $iColumnPrev Then
                    _GUICtrlHeader_SetItemFormat( $hHeader, $iColumnPrev, $HDF_STRING )
                EndIf
                Local $tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
                For $i = 0 To $g_iSearch - 1
                    $g_aSubIndex[$i] = DllStructGetData($tIndex, 1, $i + 1)
                Next
                $g_iSortDir = (($iColumn = $iColumnPrev) ? ($g_iSortDir = $HDF_SORTUP ? $HDF_SORTDOWN : $HDF_SORTUP) : ($HDF_SORTUP))
                _GUICtrlHeader_SetItemFormat( $hHeader, $iColumn, $HDF_STRING + $g_iSortDir )
                GUICtrlSendMsg( $idListView, $LVM_SETSELECTEDCOLUMN, $iColumn, 0 )
                GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0 )
                $iColumnPrev = $iColumn
        EndSwitch
    WEnd

    ; Cleanup
    GUIDelete( $hGui )
EndFunc

;========================================================================
Func WM_NOTIFY( $hWnd, $iMsg, $wParam, $lParam )

    Local $tNMHDR = DllStructCreate( $tagNMHDR, $lParam )
    Switch HWnd( DllStructGetData( $tNMHDR, "hWndFrom" ) )
        Case $g_hListView
            Switch DllStructGetData( $tNMHDR, "Code" )
                Case $LVN_GETDISPINFOW
                    Local $tNMLVDISPINFO = DllStructCreate( $tagNMLVDISPINFO, $lParam )
                    Local Static $tText = DllStructCreate( "wchar[50]" ), $pText = DllStructGetPtr( $tText )
                    If $g_iSortDir = 0x0400 Then ; 0x0400 = $HDF_SORTUP
                        DllStructSetData($tText, 1, $g_aSubArray[$g_aSubIndex[$tNMLVDISPINFO.Item]][$tNMLVDISPINFO.SubItem])
                    Else
                        DllStructSetData($tText, 1, $g_aSubArray[$g_aSubIndex[$g_iSearch -1 -$tNMLVDISPINFO.Item]][$tNMLVDISPINFO.SubItem])
                    EndIf
                    DllStructSetData( $tNMLVDISPINFO, "Text", $pText )
            EndSwitch
    EndSwitch
    Return $GUI_RUNDEFMSG
EndFunc

;========================================================================
Func WM_COMMAND( $hWnd, $iMsg, $wParam, $lParam )

    Local $hWndFrom = $lParam
    Local $iCode = BitShift( $wParam, 16 ) ; High word
    Switch $hWndFrom
        Case $g_hEdit
            Switch $iCode
                Case $EN_CHANGE
                    GUICtrlSendToDummy( $g_idEditSearch )
            EndSwitch
    EndSwitch
    Return $GUI_RUNDEFMSG
EndFunc

;========================================================================
Func _Generate_All(ByRef $g_aArray, ByRef $g_aIndex)

    ConsoleWrite( "$g_iRows = " & $g_iRows & "    $g_iCols = " & $g_iCols & @CRLF )
    Local $hTimer = TimerInit()
    $g_aArray = FAS_Random2DArrayAu3( $g_iRows, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
    For $i = 0 To $g_iRows - 1
        $g_aIndex[$i] = $i
    Next
    ConsoleWrite( "Generating array & one index = " & TimerDiff( $hTimer ) & @CRLF & @CRLF)
EndFunc

;========================================================================
Func _SortArrayStruct(Const ByRef $aArray, $iCol, $iRows)
    Local $tIndex = DllStructCreate("uint[" & $iRows & "]")
    Local $pIndex = DllStructGetPtr($tIndex)
    Local Static $hDll = DllOpen("kernel32.dll")
    Local Static $hDllComp = DllOpen("shlwapi.dll")

    Local $lo, $hi, $mi, $r

    ; Sorting by one column
    For $i = 1 To $iRows - 1
        $lo = 0
        $hi = $i - 1
        Do
            $mi = Int(($lo + $hi) / 2)
            $r = DllCall($hDllComp, 'int', 'StrCmpLogicalW', 'wstr', $aArray[$i][$iCol], 'wstr', $aArray[DllStructGetData($tIndex, 1, $mi + 1)][$iCol])[0]
            Switch $r
                Case -1
                    $hi = $mi - 1
                Case 1
                    $lo = $mi + 1
                Case 0
                    ExitLoop
            EndSwitch
        Until $lo > $hi
        DllCall($hDll, "none", "RtlMoveMemory", "struct*", $pIndex + ($mi + 1) * 4, "struct*", $pIndex + $mi * 4, "ulong_ptr", ($i - $mi) * 4)
        DllStructSetData($tIndex, 1, $i, $mi + 1 + ($lo = $mi + 1))
    Next

    Return $tIndex
EndFunc

Thanks to @LarsJ and @jpm for their scripts that helped me a lot to combine it all. The script requires "RandomArray.au3" from LarsJ. You can find this file into one of his attachment, named "ArrayDisplay.7z" and found at this link

Good luck

Edited May 1, 2021 by pixelsearch

[Lion66]

Hi pixelsearch.

Delicious!

This exactly corresponds to what I wanted to do.

And your answer is very complete and essentially.

Thank you very much. Big Like!

[pixelsearch]

Hi Lion66
Glad it worked

I would like to ask LarsJ a question : would a GUI subclassing technique be preferable or not ?

Here is the preceding script, rewritten to use the subclassing technique (instead of 2 functions WM_NOTIFY and WM_COMMAND)

expandcollapse popup

Opt( "MustDeclareVars", 1 )

#include <EditConstants.au3>
#include <GuiListView.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "RandomArray.au3" ; LarsJ

Global $g_iRows = 1000, $g_iCols = 6, $g_aArray, $g_aSubArray, $g_aIndex[$g_iRows], $g_aSubIndex[$g_iRows]
Global $g_hListView, $g_hEdit, $g_idEditSearch, $g_iSortDir, $g_iSearch = $g_iRows

Example()

Func Example()
    ; Generate Array & one index
    _Generate_All($g_aArray, $g_aIndex)
    $g_aSubArray = $g_aArray
    $g_aSubIndex = $g_aIndex

    ; Create GUI
    Local $hGui = GUICreate( "Virtual ListView + Sort + Incremental search (2e)", 630+20, 788+30+20 )

    ; Create Edit control
    GUICtrlCreateLabel( "Search", 10, 10, 50, 20 , BitOr($SS_CENTERIMAGE, $SS_CENTER) )
    Local $idEdit = GUICtrlCreateEdit( "", 70, 10, 170, 20, BitXOR( $GUI_SS_DEFAULT_EDIT, $WS_HSCROLL, $WS_VSCROLL ) )
    $g_hEdit = GUICtrlGetHandle( $idEdit )
    $g_idEditSearch = GUICtrlCreateDummy()

    ; Create ListView
    Local $idListView = GUICtrlCreateListView( "", 10, 40, 630, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE )
    _GUICtrlListView_SetExtendedListViewStyle( $idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT )
    $g_hListView = GUICtrlGetHandle( $idListView )
    Local $hHeader = _GUICtrlListView_GetHeader( $idListView )
    Local $aCols = [ "Strings", "Integers", "Floats", "Dates", "Times", "R/C" ]
    Local $aWidths = [ 230, 60, 120, 70, 60, 60 ]
    For $i = 0 To $g_iCols - 1
        _GUICtrlListView_AddColumn( $idListView, $aCols[$i], $aWidths[$i] )
    Next

    ; Sorting information
    $g_iSortDir = 0x0400 ; $HDF_SORTUP
    Local $iColumn = -1, $iColumnPrev = -1

    ; Register WM_NOTIFY & WM_COMMAND message handler through subclassing
    Local $pMessageHandler = DllCallbackGetPtr( DllCallbackRegister( "MessageHandler", "lresult", "hwnd;uint;wparam;lparam;uint_ptr;dword_ptr" ) )
    DllCall( "comctl32.dll", "bool", "SetWindowSubclass", "hwnd", $hGui, "ptr", $pMessageHandler, "uint_ptr", 0, "dword_ptr", 0 ) ; $iSubclassId = 0, $pData = 0

    GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iRows, 0 )

    ; Show GUI
    GUISetState( @SW_SHOW )

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                ExitLoop

            Case $g_idEditSearch
                Local $sSearch = GUICtrlRead( $idEdit )
                If $sSearch = "" Then
                    ; Empty search string, display all rows
                    $g_aSubArray = $g_aArray
                    $g_aSubIndex = $g_aIndex
                    $g_iSearch = $g_iRows
                Else
                    ; Find rows matching the search string
                    $g_iSearch = 0
                    For $i = 0 To $g_iRows - 1
                        ; If StringInStr( $g_aArray[$i][0], $sSearch ) Then ; Normal search
                        If StringRegExp( $g_aArray[$i][0], $sSearch ) Then ; Reg. exp. search
                            For $j = 0 To $g_iCols - 1
                                $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
                            Next
                            $g_aSubIndex[$g_iSearch] = $g_iSearch
                            $g_iSearch += 1
                        EndIf
                    Next
                EndIf
                ConsoleWrite( StringFormat( "%4d", $g_iSearch ) & " rows matching """ & $sSearch & """" & @CRLF )
                GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0 )

            Case $idListView ; Sort
                $iColumn = GUICtrlGetState( $idListView )
                If $iColumn <> $iColumnPrev Then
                    _GUICtrlHeader_SetItemFormat( $hHeader, $iColumnPrev, $HDF_STRING )
                EndIf
                Local $tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
                For $i = 0 To $g_iSearch - 1
                    $g_aSubIndex[$i] = DllStructGetData($tIndex, 1, $i + 1)
                Next
                $g_iSortDir = (($iColumn = $iColumnPrev) ? ($g_iSortDir = $HDF_SORTUP ? $HDF_SORTDOWN : $HDF_SORTUP) : ($HDF_SORTUP))
                _GUICtrlHeader_SetItemFormat( $hHeader, $iColumn, $HDF_STRING + $g_iSortDir )
                GUICtrlSendMsg( $idListView, $LVM_SETSELECTEDCOLUMN, $iColumn, 0 )
                GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0 )
                $iColumnPrev = $iColumn
        EndSwitch
    WEnd

    ; Cleanup
    DllCall( "comctl32.dll", "bool", "RemoveWindowSubclass", "hwnd", $hGui, "ptr", $pMessageHandler, "uint_ptr", 0 ) ; $iSubclassId = 0
    GUIDelete( $hGui )
EndFunc

;========================================================================
Func MessageHandler( $hWnd, $iMsg, $wParam, $lParam, $iSubclassId, $pData )

    Switch $iMsg
        Case 0x004E ; 0x004E = $WM_NOTIFY
            Local $tNMHDR = DllStructCreate( $tagNMHDR, $lParam )
            Switch HWnd( DllStructGetData( $tNMHDR, "hWndFrom" ) )
                Case $g_hListView
                    Switch DllStructGetData( $tNMHDR, "Code" )
                        Case $LVN_GETDISPINFOW
                            Local $tNMLVDISPINFO = DllStructCreate( $tagNMLVDISPINFO, $lParam )
                            Local Static $tText = DllStructCreate( "wchar[50]" ), $pText = DllStructGetPtr( $tText )
                            If $g_iSortDir = 0x0400 Then ; 0x0400 = $HDF_SORTUP
                                DllStructSetData($tText, 1, $g_aSubArray[$g_aSubIndex[$tNMLVDISPINFO.Item]][$tNMLVDISPINFO.SubItem])
                            Else
                                DllStructSetData($tText, 1, $g_aSubArray[$g_aSubIndex[$g_iSearch -1 -$tNMLVDISPINFO.Item]][$tNMLVDISPINFO.SubItem])
                            EndIf
                            DllStructSetData( $tNMLVDISPINFO, "Text", $pText )
                            Return
                    EndSwitch
            EndSwitch

        Case 0x0111 ; 0x0111 = $WM_COMMAND
            Local $hWndFrom = $lParam
            Local $iCode = BitShift( $wParam, 16 ) ; High word
            Switch $hWndFrom
                Case $g_hEdit
                    Switch $iCode
                        Case $EN_CHANGE
                            GUICtrlSendToDummy( $g_idEditSearch )
                            Return
                    EndSwitch
            EndSwitch
    EndSwitch

    Return DllCall( "comctl32.dll", "lresult", "DefSubclassProc", "hwnd", $hWnd, "uint", $iMsg, "wparam", $wParam, "lparam", $lParam )[0]
    #forceref $iSubclassId, $pData
EndFunc

;========================================================================
Func _Generate_All(ByRef $g_aArray, ByRef $g_aIndex)

    ConsoleWrite( "$g_iRows = " & $g_iRows & "    $g_iCols = " & $g_iCols & @CRLF )
    Local $hTimer = TimerInit()
    $g_aArray = FAS_Random2DArrayAu3( $g_iRows, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
    For $i = 0 To $g_iRows - 1
        $g_aIndex[$i] = $i
    Next
    ConsoleWrite( "Generating array & one index = " & TimerDiff( $hTimer ) & @CRLF & @CRLF)
EndFunc

;========================================================================
Func _SortArrayStruct(Const ByRef $aArray, $iCol, $iRows)

    Local $tIndex = DllStructCreate("uint[" & $iRows & "]")
    Local $pIndex = DllStructGetPtr($tIndex)
    Local Static $hDll = DllOpen("kernel32.dll")
    Local Static $hDllComp = DllOpen("shlwapi.dll")

    Local $lo, $hi, $mi, $r

    ; Sorting by one column
    For $i = 1 To $iRows - 1
        $lo = 0
        $hi = $i - 1
        Do
            $mi = Int(($lo + $hi) / 2)
            $r = DllCall($hDllComp, 'int', 'StrCmpLogicalW', 'wstr', $aArray[$i][$iCol], 'wstr', $aArray[DllStructGetData($tIndex, 1, $mi + 1)][$iCol])[0]
            Switch $r
                Case -1
                    $hi = $mi - 1
                Case 1
                    $lo = $mi + 1
                Case 0
                    ExitLoop
            EndSwitch
        Until $lo > $hi
        DllCall($hDll, "none", "RtlMoveMemory", "struct*", $pIndex + ($mi + 1) * 4, "struct*", $pIndex + $mi * 4, "ulong_ptr", ($i - $mi) * 4)
        DllStructSetData($tIndex, 1, $i, $mi + 1 + ($lo = $mi + 1))
    Next

    Return $tIndex
EndFunc

Let's hope both "Return" instructions are correct after Case $LVN_GETDISPINFOW and Case $EN_CHANGE, preventing the "call of the next handler in a window's subclass chain"

Edited May 2, 2021 by pixelsearch

[Nine]

Hi @pixelsearch

Hope you don't mind, but your script interpellated me.  I was wondering if we could use struct for Sort index instead of array saving the time to transfer.  Also if we could keep the struct for further reference instead of recreating it every time.  Here what I ended up with (let me know what you think) :

expandcollapse popup

#include <Constants.au3>
#include <EditConstants.au3>
#include <GuiListView.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "RandomArray.au3" ; LarsJ

Opt("MustDeclareVars", 1)

Global $g_iRows = 10000, $g_iCols = 6, $g_aArray, $g_aSubArray, $g_aIndex[$g_iRows], $g_tIndex = DllStructCreate("uint arr[" & $g_iRows & "]")
Global $g_hListView, $g_hEdit, $g_idEditSearch, $g_iSortDir, $g_iSearch = $g_iRows, $g_aIndex[$g_iCols], $g_tDefaultIndex

Example()

Func Example()
  ; Generate Array & one index
  _Generate_All($g_aArray)
  $g_aSubArray = $g_aArray
  $g_tDefaultIndex = $g_tIndex
  ; Create GUI
  Local $hGui = GUICreate("Virtual ListView + Sort + Incremental search (2d)", 630 + 20, 788 + 30 + 20)

  ; Create Edit control
  GUICtrlCreateLabel("Search", 10, 10, 50, 20, BitOR($SS_CENTERIMAGE, $SS_CENTER))
  Local $idEdit = GUICtrlCreateEdit("", 70, 10, 170, 20, BitXOR($GUI_SS_DEFAULT_EDIT, $WS_HSCROLL, $WS_VSCROLL))
  $g_hEdit = GUICtrlGetHandle($idEdit)
  $g_idEditSearch = GUICtrlCreateDummy()

  ; Create ListView
  Local $idListView = GUICtrlCreateListView("", 10, 40, 630, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE)
  _GUICtrlListView_SetExtendedListViewStyle($idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT)
  $g_hListView = GUICtrlGetHandle($idListView)
  Local $hHeader = _GUICtrlListView_GetHeader($idListView)
  Local $aCols = ["Strings", "Integers", "Floats", "Dates", "Times", "R/C"]
  Local $aWidths = [230, 60, 120, 70, 60, 60]
  For $i = 0 To $g_iCols - 1
    _GUICtrlListView_AddColumn($idListView, $aCols[$i], $aWidths[$i])
  Next

  ; Sorting information
  $g_iSortDir = 0x0400   ; $HDF_SORTUP
  Local $iColumn = -1, $iColumnPrev = -1

  GUIRegisterMsg($WM_COMMAND, "WM_COMMAND")
  GUIRegisterMsg($WM_NOTIFY, "WM_NOTIFY")

  GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iRows, 0)

  GUISetState(@SW_SHOW)

  While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
        ExitLoop

      Case $g_idEditSearch
        _GUICtrlHeader_SetItemFormat($hHeader, $iColumn, $HDF_STRING)
        Local $sSearch = GUICtrlRead($idEdit)
        $g_tIndex = $g_tDefaultIndex
        If $sSearch = "" Then
          ; Empty search string, display all rows
          $g_aSubArray = $g_aArray
          $g_iSearch = $g_iRows
        Else
          ; Find rows matching the search string
          $g_iSearch = 0
          For $i = 0 To $g_iRows - 1
            ; If StringInStr( $g_aArray[$i][0], $sSearch ) Then ; Normal search
            If StringRegExp($g_aArray[$i][0], $sSearch) Then               ; Reg. exp. search
              For $j = 0 To $g_iCols - 1
                $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
              Next
              $g_iSearch += 1
            EndIf
          Next
        EndIf
        ConsoleWrite(StringFormat("%4d", $g_iSearch) & " rows matching """ & $sSearch & """" & @CRLF)
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)

      Case $idListView       ; Sort
        $iColumn = GUICtrlGetState($idListView)
        If $iColumn <> $iColumnPrev Then
          _GUICtrlHeader_SetItemFormat($hHeader, $iColumnPrev, $HDF_STRING)
        EndIf
        If GUICtrlRead($idEdit) Then
          $g_tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
        ElseIf VarGetType($g_aIndex[$iColumn]) = "DLLStruct" Then
          $g_tIndex = $g_aIndex[$iColumn]
        Else
          $g_tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
          $g_aIndex[$iColumn] = $g_tIndex
        EndIf
        $g_iSortDir = (($iColumn = $iColumnPrev) ? ($g_iSortDir = $HDF_SORTUP ? $HDF_SORTDOWN : $HDF_SORTUP) : ($HDF_SORTUP))
        _GUICtrlHeader_SetItemFormat($hHeader, $iColumn, $HDF_STRING + $g_iSortDir)
        GUICtrlSendMsg($idListView, $LVM_SETSELECTEDCOLUMN, $iColumn, 0)
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)
        $iColumnPrev = $iColumn
    EndSwitch
  WEnd

  ; Cleanup
  GUIDelete($hGui)
EndFunc   ;==>Example

;========================================================================
Func WM_NOTIFY($hWnd, $iMsg, $wParam, $lParam)

  Local $tNMHDR = DllStructCreate($tagNMHDR, $lParam)
  Switch HWnd(DllStructGetData($tNMHDR, "hWndFrom"))
    Case $g_hListView
      Switch DllStructGetData($tNMHDR, "Code")
        Case $LVN_GETDISPINFOW
          Local $tNMLVDISPINFO = DllStructCreate($tagNMLVDISPINFO, $lParam)
          Local Static $tText = DllStructCreate("wchar[50]"), $pText = DllStructGetPtr($tText)
          If $g_iSortDir = 0x0400 Then           ; 0x0400 = $HDF_SORTUP
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($tNMLVDISPINFO.Item + 1)][$tNMLVDISPINFO.SubItem])
          Else
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($g_iSearch - $tNMLVDISPINFO.Item)][$tNMLVDISPINFO.SubItem])
          EndIf
          DllStructSetData($tNMLVDISPINFO, "Text", $pText)
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_NOTIFY

;========================================================================
Func WM_COMMAND($hWnd, $iMsg, $wParam, $lParam)

  Local $hWndFrom = $lParam
  Local $iCode = BitShift($wParam, 16)     ; High word
  Switch $hWndFrom
    Case $g_hEdit
      Switch $iCode
        Case $EN_CHANGE
          GUICtrlSendToDummy($g_idEditSearch)
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_COMMAND

;========================================================================
Func _Generate_All(ByRef $g_aArray)

  ConsoleWrite("$g_iRows = " & $g_iRows & "    $g_iCols = " & $g_iCols & @CRLF)
  Local $hTimer = TimerInit()
  $g_aArray = FAS_Random2DArrayAu3($g_iRows, "sifdtr", "abcdefghijklmnopqrstuvwxyz")
  For $i = 0 To $g_iRows - 1
    $g_tIndex.arr($i + 1) = $i
  Next
  ConsoleWrite("Generating array & one index = " & TimerDiff($hTimer) & @CRLF & @CRLF)
EndFunc   ;==>_Generate_All

;========================================================================
Func _SortArrayStruct(Const ByRef $aArray, $iCol, $iRows)
  Local $tIndex = DllStructCreate("uint arr[" & $iRows & "]")
  Local $pIndex = DllStructGetPtr($tIndex)
  Local Static $hDll = DllOpen("kernel32.dll")
  Local Static $hDllComp = DllOpen("shlwapi.dll")

  Local $lo, $hi, $mi, $r

  ; Sorting by one column
  For $i = 1 To $iRows - 1
    $lo = 0
    $hi = $i - 1
    Do
      $mi = Int(($lo + $hi) / 2)
      $r = DllCall($hDllComp, 'int', 'StrCmpLogicalW', 'wstr', $aArray[$i][$iCol], 'wstr', $aArray[DllStructGetData($tIndex, 1, $mi + 1)][$iCol])[0]
      Switch $r
        Case -1
          $hi = $mi - 1
        Case 1
          $lo = $mi + 1
        Case 0
          ExitLoop
      EndSwitch
    Until $lo > $hi
    DllCall($hDll, "none", "RtlMoveMemory", "struct*", $pIndex + ($mi + 1) * 4, "struct*", $pIndex + $mi * 4, "ulong_ptr", ($i - $mi) * 4)
    DllStructSetData($tIndex, 1, $i, $mi + 1 + ($lo = $mi + 1))
  Next

  Return $tIndex
EndFunc   ;==>_SortArrayStruct

 

[pixelsearch]

4 hours ago, Nine said:

Hope you don't mind,

Not at all, on the contrary You always got good ideas, like replacing $g_aSubArray[$g_aSubIndex[... with $g_aSubArray[$g_tIndex.arr(...

Also, as you scripted it, keeping the indexes for each column (when all rows are displayed) is really great, especially if there are multi thousands rows. Then, when all rows are displayed again (i.e when edit field is empty again) a sort on any column (asc or desc) will be immediate.

Apart from that great improvement, I didn't notice a time difference between our scripts (mine being named "2d" in GUI title, i.e. the script with WM_NOTIFY, not the "2e" which got the subclassing, and yours that I named "2f" in its title so I'm sure which one was launched)

This is because I only did tests based on 10.000 rows but if there were more rows then a difference would surely slowly appear. We'd probably have a difference of 1s if 100.000 rows . Now if there were 1 million rows...

To illustrate that point, if you add a timer to this part of my script (inside Case $idListView) then you'll notice the ridiculous time taken to re-fill the index based on 10.000 rows (for example by clicking on any Col header immediately after the script is launched) :

Local $hTimer = TimerInit()
For $i = 0 To $g_iSearch - 1
    $g_aSubIndex[$i] = DllStructGetData($tIndex, 1, $i + 1)
Next
ConsoleWrite( "Case $idListView : $g_iSearch = " & $g_iSearch & "   recreating index = " & TimerDiff( $hTimer ) & @CRLF)

Only 2 changes I made to your script are :
* Deleted $g_aIndex[$g_iRows] from Global line code (no problem as it's redeclared $g_aIndex[$g_iCols] on next line)
* Added (2f) in GUI title so we know which script is launched, in case of screen copies etc...

If I may, another great improvement in your script could be this : when the edit field is not empty (user just entered an "a" for example) then clicking on a column header triggers this part of your script (inside Case $idListView)

If GUICtrlRead($idEdit) Then
    $g_tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
ElseIf ...

What about keeping this temporary subindex somewhere, in case the user immediately clicks on the same column header again to display the reverse sort ?

I'm sure it's doable. It could be useful for example if the subarray displays a few thousands rows, then the opposite click sort will be immediate (it could also be scripted for all columns). Meanwhile, a flag could indicate if the user altered the edit field in any way, to quickly forget all these temporary subindexes etc...

I was telling Jpm that, in my script above, I choosed a "dangerous" way (as you surely noticed) because $g_aSubArray[][] and $g_aSubIndex[] are never deleted and recreated (unless all rows are displayed again). Only their first rows are constantly overwritten and $g_iSearch does the job to know which is the last row to display ("the final frontier") . Better not to look at all rows inside these 2 arrays

Edited May 2, 2021 by pixelsearch

[pixelsearch]

On 5/2/2021 at 8:54 PM, pixelsearch said:

What about keeping this temporary subindex somewhere, in case the user immediately clicks on the same column header again to display the reverse sort ?

Hi @Nine
Based on your improvement, I added some code in your script to create temporary subindexes which will be useful to quickly sort partial results (asc then desc). Each subindex is created only when the user clicks a column and they all will be deleted when another partial result is requested. Additionnaly, a label field indicates how many matches have been found. Version is labeled "2g"

Please report in case you find a bug or another improvement can be done.
Thanks !

expandcollapse popup

#include <EditConstants.au3>
#include <GuiListView.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "RandomArray.au3" ; LarsJ

Opt("MustDeclareVars", 1)

Global $g_iRows = 10000, $g_iCols = 6, $g_hListView, $g_iSortDir, $g_hEdit, $g_idEditSearch,  $g_iSearch = $g_iRows
Global $g_aArray, $g_aSubArray, $g_tDefaultIndex, $g_tIndex = DllStructCreate("uint arr[" & $g_iRows & "]")
Global $g_aIndex[$g_iCols], $g_aIndexTemp[$g_iCols] ; VarGetType's : $g_aIndex => "Array", $g_aIndex[0] => "String"

Example()

Func Example()
  ; Generate main array & one index
  _Generate_All($g_aArray)
  $g_aSubArray = $g_aArray
  $g_tDefaultIndex = $g_tIndex

  ; Create GUI
  Local $hGui = GUICreate("Virtual ListView + Sort + Incremental search (2g)", 630 + 20, 788 + 30 + 20)

  ; Create Edit control
  GUICtrlCreateLabel("Search", 10, 10, 50, 20, BitOR($SS_CENTERIMAGE, $SS_CENTER))
  Local $idEdit = GUICtrlCreateEdit("", 70, 10, 170, 20, BitXOR($GUI_SS_DEFAULT_EDIT, $WS_HSCROLL, $WS_VSCROLL))
  $g_hEdit = GUICtrlGetHandle($idEdit)
  $g_idEditSearch = GUICtrlCreateDummy()

  ; Create Label control (number of matching results)
  Local $idResult = GUICtrlCreateLabel("", 260, 10, 350, 20, BitOR($SS_CENTERIMAGE, $SS_SUNKEN, $SS_NOPREFIX)) ; last one for "&"

  ; Create ListView
  Local $idListView = GUICtrlCreateListView("", 10, 40, 630, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE)
  _GUICtrlListView_SetExtendedListViewStyle($idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT)
  $g_hListView = GUICtrlGetHandle($idListView)
  Local $hHeader = _GUICtrlListView_GetHeader($idListView)
  Local $aCols = ["Strings", "Integers", "Floats", "Dates", "Times", "R/C"]
  Local $aWidths = [230, 60, 120, 70, 60, 60]
  For $i = 0 To $g_iCols - 1
    _GUICtrlListView_AddColumn($idListView, $aCols[$i], $aWidths[$i])
  Next

  ; Sorting information
  $g_iSortDir = 0x0400   ; $HDF_SORTUP
  Local $iColumn = -1, $iColumnPrev = -1

  GUIRegisterMsg($WM_COMMAND, "WM_COMMAND")
  GUIRegisterMsg($WM_NOTIFY, "WM_NOTIFY")

  GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iRows, 0)

  GUISetState(@SW_SHOW)

  While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
        ExitLoop

      Case $g_idEditSearch
        _GUICtrlHeader_SetItemFormat($hHeader, $iColumn, $HDF_STRING)
        Local $sSearch = GUICtrlRead($idEdit)
        $g_tIndex = $g_tDefaultIndex
        If $sSearch = "" Then
          ; Empty search string, display all rows
          $g_aSubArray = $g_aArray
          $g_iSearch = $g_iRows
        Else
          ; Find rows matching the search string
          $g_iSearch = 0
          For $i = 0 To $g_iRows - 1
            ; If StringInStr( $g_aArray[$i][0], $sSearch ) Then ; Normal search
            If StringRegExp($g_aArray[$i][0], $sSearch) Then    ; Reg. exp. search
              For $j = 0 To $g_iCols - 1
                $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
              Next
              $g_iSearch += 1
            EndIf
          Next
          ; Delete eventual temporary subindexes
          For $i = 0 To $g_iCols - 1
            If VarGetType($g_aIndexTemp[$i]) = "DLLStruct" Then $g_aIndexTemp[$i] = "" ; "String"
          Next
        EndIf
        ; ConsoleWrite(StringFormat("%4d", $g_iSearch) & " rows matching """ & $sSearch & """" & @CRLF)
        GUICtrlSetData($idResult, ' ' & $g_iSearch & ' rows matching "' & $sSearch & '"')
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)

      Case $idListView ; Sort
        $iColumn = GUICtrlGetState($idListView)
        If $iColumn <> $iColumnPrev Then
          _GUICtrlHeader_SetItemFormat($hHeader, $iColumnPrev, $HDF_STRING)
        EndIf
        ; Set $g_tIndex + eventual update of $g_aIndexTemp[$iColumn] OR $g_aIndex[$iColumn]
        If GUICtrlRead($idEdit) Then
          _UpdateIndex($g_aIndexTemp, $iColumn)
        Else
          _UpdateIndex($g_aIndex, $iColumn)
        EndIf
        $g_iSortDir = (($iColumn = $iColumnPrev) ? ($g_iSortDir = $HDF_SORTUP ? $HDF_SORTDOWN : $HDF_SORTUP) : ($HDF_SORTUP))
        _GUICtrlHeader_SetItemFormat($hHeader, $iColumn, $HDF_STRING + $g_iSortDir)
        GUICtrlSendMsg($idListView, $LVM_SETSELECTEDCOLUMN, $iColumn, 0)
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)
        $iColumnPrev = $iColumn
    EndSwitch
  WEnd

  ; Cleanup
  GUIDelete($hGui)
EndFunc   ;==>Example

;========================================================================
Func WM_NOTIFY($hWnd, $iMsg, $wParam, $lParam)

  Local $tNMHDR = DllStructCreate($tagNMHDR, $lParam)
  Switch HWnd(DllStructGetData($tNMHDR, "hWndFrom"))
    Case $g_hListView
      Switch DllStructGetData($tNMHDR, "Code")
        Case $LVN_GETDISPINFOW
          Local $tNMLVDISPINFO = DllStructCreate($tagNMLVDISPINFO, $lParam)
          Local Static $tText = DllStructCreate("wchar[50]"), $pText = DllStructGetPtr($tText)
          If $g_iSortDir = 0x0400 Then ; 0x0400 = $HDF_SORTUP
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($tNMLVDISPINFO.Item + 1)][$tNMLVDISPINFO.SubItem])
          Else
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($g_iSearch - $tNMLVDISPINFO.Item)][$tNMLVDISPINFO.SubItem])
          EndIf
          DllStructSetData($tNMLVDISPINFO, "Text", $pText)
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_NOTIFY

;========================================================================
Func WM_COMMAND($hWnd, $iMsg, $wParam, $lParam)

  Local $hWndFrom = $lParam
  Local $iCode = BitShift($wParam, 16) ; High word
  Switch $hWndFrom
    Case $g_hEdit
      Switch $iCode
        Case $EN_CHANGE
          GUICtrlSendToDummy($g_idEditSearch)
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_COMMAND

;========================================================================
Func _Generate_All(ByRef $g_aArray)

  ConsoleWrite("$g_iRows = " & $g_iRows & "    $g_iCols = " & $g_iCols & @CRLF)
  Local $hTimer = TimerInit()
  $g_aArray = FAS_Random2DArrayAu3($g_iRows, "sifdtr", "abcdefghijklmnopqrstuvwxyz")
  For $i = 0 To $g_iRows - 1
    $g_tIndex.arr($i + 1) = $i
  Next
  ConsoleWrite("Generating main array & one index = " & TimerDiff($hTimer) & @CRLF & @CRLF)
EndFunc   ;==>_Generate_All

;========================================================================
Func _SortArrayStruct(Const ByRef $aArray, $iCol, $iRows)
  Local $tIndex = DllStructCreate("uint arr[" & $iRows & "]")
  Local $pIndex = DllStructGetPtr($tIndex)
  Local Static $hDll = DllOpen("kernel32.dll")
  Local Static $hDllComp = DllOpen("shlwapi.dll")

  Local $lo, $hi, $mi, $r

  ; Sorting by one column
  For $i = 1 To $iRows - 1
    $lo = 0
    $hi = $i - 1
    Do
      $mi = Int(($lo + $hi) / 2)
      $r = DllCall($hDllComp, 'int', 'StrCmpLogicalW', 'wstr', $aArray[$i][$iCol], 'wstr', $aArray[DllStructGetData($tIndex, 1, $mi + 1)][$iCol])[0]
      Switch $r
        Case -1
          $hi = $mi - 1
        Case 1
          $lo = $mi + 1
        Case 0
          ExitLoop
      EndSwitch
    Until $lo > $hi
    DllCall($hDll, "none", "RtlMoveMemory", "struct*", $pIndex + ($mi + 1) * 4, "struct*", $pIndex + $mi * 4, "ulong_ptr", ($i - $mi) * 4)
    DllStructSetData($tIndex, 1, $i, $mi + 1 + ($lo = $mi + 1))
  Next

  Return $tIndex
EndFunc   ;==>_SortArrayStruct

;========================================================================
Func _UpdateIndex(ByRef $aIndex, $iColumn)

  If VarGetType($aIndex[$iColumn]) = "DLLStruct" Then
    $g_tIndex = $aIndex[$iColumn]
  Else
    $g_tIndex = _SortArrayStruct($g_aSubArray, $iColumn, $g_iSearch)
    $aIndex[$iColumn] = $g_tIndex ; "DLLStruct" (or "Int32" when no match found +++)
  EndIf
EndFunc   ;==>_UpdateIndex

Edit: Jpm's & Melba23's new _ArrayDisplay will indicate {Array} or {Map} when required in the display. To avoid a blank display, should {DLLStruct} be indicated too when needed ?

Not sure, as all these additionals tests on {Array} , {Map} , {DLLStruct} before displaying each element in _ArrayDisplay may slow down a bit the whole process in case of huge arrays. Anyway... the question is asked.

Edited May 4, 2021 by pixelsearch

[pixelsearch]

Hi @FrancescoDiMuro
Glad you enjoyed the precedent version "2g" of virtual listview + sort + incremental search
Below will be my last version "2h" concerning this matter. Though these scripts are all related to Virtual listviews, they may take too much place in this thread (sorry LarsJ) . Also, the GUI is now completely filled !

This new version allows the user to search inside any column (using the combobox). An orange marker is displayed above the searched column header. The searched column is also constantly selected (until the user clicks any header, for sorting)

expandcollapse popup

#include <ComboConstants.au3>
#include <EditConstants.au3>
#include <GuiListView.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "RandomArray.au3" ; LarsJ

Opt("MustDeclareVars", 1)

Global $g_iRows = 10000, $g_iCols = 6, $g_iLeftLV = 10, $g_iTopLV = 40, $g_hGui, $g_hListView, $g_hHeader, $g_hEdit
Global $g_iSortDir, $g_idMarker, $g_idComboCol, $g_idComboColDummy, $g_idEditDummy, $g_iSearch = $g_iRows
Global $g_aArray, $g_aSubArray, $g_tDefaultIndex, $g_tIndex = DllStructCreate("uint arr[" & $g_iRows & "]")
Global $g_aIndex[$g_iCols], $g_aIndexTemp[$g_iCols] ; VarGetType's : $g_aIndex => "Array", $g_aIndex[0] => "String"

Example()

Func Example()
  ; Generate array & one index
  _Generate_All($g_aArray)
  $g_aSubArray = $g_aArray
  $g_tDefaultIndex = $g_tIndex

  ; Create GUI
  $g_hGui = GUICreate("Virtual ListView + Sort + Incremental search (2h)", 630 + 20, 788 + 30 + 20)

  ; Create Edit control (search) + dummy control
  Local $idEdit = GUICtrlCreateEdit("", 120, 10, 305, 20, BitXOR($GUI_SS_DEFAULT_EDIT, $WS_HSCROLL, $WS_VSCROLL))
  $g_hEdit = GUICtrlGetHandle($idEdit)
  $g_idEditDummy = GUICtrlCreateDummy()

  ; Create ComboBox control (how to search : RegEx or Normal ?)
  Local $idSearchHow = GUICtrlCreateCombo("RegEx search", 11, 9, 100, 20, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_DROPDOWNLIST))
  Local $sSearchHow = "RegEx search", $sSearchHowPrev = $sSearchHow ; default way of searching (changeable)
  GUICtrlSetData($idSearchHow, "Normal search", $sSearchHow)

  ; Create ComboBox control (column where to search) + dummy control
  GUICtrlCreateLabel("Col", 429, 10, 20, 20, BitOR($SS_CENTERIMAGE, $SS_CENTER))
  $g_idComboCol = GUICtrlCreateCombo("0", 452, 9, 41, 20, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_DROPDOWNLIST))
  Local $iSearchCol = 0, $iSearchColPrev = $iSearchCol ; default column where to search (changeable)
  For $i = 1 To $g_iCols - 1
    GUICtrlSetData($g_idComboCol, $i & "|", $iSearchCol)
  Next
  $g_idComboColDummy = GUICtrlCreateDummy()

  ; Create Label control (number of matching results)
  Local $idResult = GUICtrlCreateLabel(" " & $g_iRows & " rows (no pattern)", 501, 10, 135, 20, BitOR($SS_CENTERIMAGE, $SS_SUNKEN))

  ; Create ListView
  Local $idListView = GUICtrlCreateListView("", $g_iLeftLV, $g_iTopLV, 630, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE)
  _GUICtrlListView_SetExtendedListViewStyle($idListView, BitOr($LVS_EX_DOUBLEBUFFER, $LVS_EX_FULLROWSELECT))
  $g_hListView = GUICtrlGetHandle($idListView)
  $g_hHeader = _GUICtrlListView_GetHeader($idListView)
  Local $aCols = ["Strings", "Integers", "Floats", "Dates", "Times", "R/C"]
  Local $aWidths = [230, 61, 124, 70, 60, 60]
  For $i = 0 To $g_iCols - 1
    _GUICtrlListView_AddColumn($idListView, $aCols[$i], $aWidths[$i])
  Next

  ; Create Marker (an orange line placed above the header of the column being searched)
  $g_idMarker = GUICtrlCreateLabel("", 0, 0)
  GUICtrlSetBkColor(-1, 0xFFA060) ; orange
  _MoveMarker($iSearchCol)
  _GUICtrlListView_SetSelectedColumn($g_hListView, $iSearchCol)

  ; Sorting information
  $g_iSortDir = 0x0400   ; $HDF_SORTUP
  Local $iSortCol = -1, $iSortColPrev = -1

  GUIRegisterMsg($WM_COMMAND, "WM_COMMAND")
  GUIRegisterMsg($WM_NOTIFY, "WM_NOTIFY")

  GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iRows, 0)

  GUISetState(@SW_SHOW)

  While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
        ExitLoop

      Case $g_idComboCol, $g_idComboColDummy, $idSearchHow
        $iSearchCol = GUICtrlRead($g_idComboCol)
        $sSearchHow = GUICtrlRead($idSearchHow)
        Select
          Case $iSearchCol <> $iSearchColPrev
            _MoveMarker($iSearchCol) ; Search column will be selected below, after ContinueCase
            $iSearchColPrev = $iSearchCol
          Case $sSearchHow <> $sSearchHowPrev
            $sSearchHowPrev = $sSearchHow
          Case Else ; no change in both Combo controls (same search column, same search way)
            ContinueLoop
        EndSelect
        ContinueCase

      Case $g_idEditDummy
        _GUICtrlHeader_SetItemFormat($g_hHeader, $iSortCol, $HDF_STRING)
        Local $sSearch = GUICtrlRead($idEdit)
        $g_tIndex = $g_tDefaultIndex
        If $sSearch = "" Then
          ; Empty search string, display all rows
          $g_aSubArray = $g_aArray
          $g_iSearch = $g_iRows
        Else
          ; Find rows matching the search string
          $g_iSearch = 0
          If $sSearchHow = "RegEx search" Then
            For $i = 0 To $g_iRows - 1 ; duplicated For... Next for speed
              If StringRegExp($g_aArray[$i][$iSearchCol], $sSearch) Then
                For $j = 0 To $g_iCols - 1
                  $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
                Next
                $g_iSearch += 1
              EndIf
            Next
          Else ; "Normal search"
            For $i = 0 To $g_iRows - 1 ; duplicated For... Next for speed
              If StringInStr($g_aArray[$i][$iSearchCol], $sSearch) Then
                For $j = 0 To $g_iCols - 1
                  $g_aSubArray[$g_iSearch][$j] = $g_aArray[$i][$j]
                Next
                $g_iSearch += 1
              EndIf
            Next
          EndIf
          ; Delete eventual temporary subindexes
          For $i = 0 To $g_iCols - 1
            If VarGetType($g_aIndexTemp[$i]) = "DLLStruct" Then $g_aIndexTemp[$i] = "" ; "String"
          Next
        EndIf
        GUICtrlSetData($idResult, " " & $g_iSearch & ($sSearch = "" ? " rows (no pattern)" : " matching rows"))
        ;------
        _GUICtrlListView_SetSelectedColumn($g_hListView, $iSearchCol) ; works fine
        ; _GUICtrlListView_SetSelectedColumn($idListView, $iSearchCol) ; strange behavior here
        ; GUICtrlSendMsg($idListView, $LVM_SETSELECTEDCOLUMN, $iSearchCol, 0) ; strange behavior here
        ;------
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)

      Case $idListView ; Sort
        $iSortCol = GUICtrlGetState($idListView)
        If $iSortCol <> $iSortColPrev Then
          _GUICtrlHeader_SetItemFormat($g_hHeader, $iSortColPrev, $HDF_STRING)
        EndIf
        ; Set $g_tIndex + eventual update of $g_aIndexTemp[$iSortCol] OR $g_aIndex[$iSortCol]
        If GUICtrlRead($idEdit) Then
          _UpdateIndex($g_aIndexTemp, $iSortCol)
        Else
          _UpdateIndex($g_aIndex, $iSortCol)
        EndIf
        $g_iSortDir = (($iSortCol = $iSortColPrev) ? ($g_iSortDir = $HDF_SORTUP ? $HDF_SORTDOWN : $HDF_SORTUP) : ($HDF_SORTUP))
        _GUICtrlHeader_SetItemFormat($g_hHeader, $iSortCol, $HDF_STRING + $g_iSortDir)
        GUICtrlSendMsg($idListView, $LVM_SETSELECTEDCOLUMN, $iSortCol, 0)
        GUICtrlSendMsg($idListView, $LVM_SETITEMCOUNT, $g_iSearch, 0)
        $iSortColPrev = $iSortCol

      Case $GUI_EVENT_RESTORE ; needed, or Marker goes back in 0, 0 after Restore (why ?)
        _MoveMarker($iSearchCol)
    EndSwitch
  WEnd

  ; Cleanup
  GUIDelete($g_hGui)
EndFunc   ;==>Example

;========================================================================
Func WM_NOTIFY($hWnd, $iMsg, $wParam, $lParam)

  Local $tNMHDR = DllStructCreate($tagNMHDR, $lParam)
  Switch HWnd(DllStructGetData($tNMHDR, "hWndFrom"))
    Case $g_hListView
      Switch DllStructGetData($tNMHDR, "Code")
        Case $LVN_GETDISPINFOW
          Local $tNMLVDISPINFO = DllStructCreate($tagNMLVDISPINFO, $lParam)
          Local Static $tText = DllStructCreate("wchar[50]"), $pText = DllStructGetPtr($tText)
          If $g_iSortDir = 0x0400 Then ; 0x0400 = $HDF_SORTUP
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($tNMLVDISPINFO.Item + 1)][$tNMLVDISPINFO.SubItem])
          Else
            DllStructSetData($tText, 1, $g_aSubArray[$g_tIndex.arr($g_iSearch - $tNMLVDISPINFO.Item)][$tNMLVDISPINFO.SubItem])
          EndIf
          DllStructSetData($tNMLVDISPINFO, "Text", $pText)
      EndSwitch

    Case $g_hHeader
      Switch DllStructGetData($tNMHDR, "Code")
        Case $HDN_ENDTRACKW, $HDN_DIVIDERDBLCLICKW ; let's forget $HDN_TRACKW...
          _MoveMarker(GUICtrlRead($g_idComboCol))
        Case $NM_RCLICK
          Local $aHit = _GUICtrlListView_SubItemHitTest($g_hListView)
          ; $aHit[1] : 0-based index of the LV subitem... i.e. the column in our case (may be -1 if right click on empty part of header)
          If $aHit[1] > - 1 Then ; valid column
            GUICtrlSetData($g_idComboCol, $aHit[1])
            GUICtrlSendToDummy($g_idComboColDummy)
          EndIf
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_NOTIFY

;========================================================================
Func WM_COMMAND($hWnd, $iMsg, $wParam, $lParam)

  Local $hWndFrom = $lParam
  Local $iCode = BitShift($wParam, 16) ; High word
  Switch $hWndFrom
    Case $g_hEdit
      Switch $iCode
        Case $EN_CHANGE
          GUICtrlSendToDummy($g_idEditDummy)
      EndSwitch
  EndSwitch
  Return $GUI_RUNDEFMSG
EndFunc   ;==>WM_COMMAND

;========================================================================
Func _Generate_All(ByRef $g_aArray)

  ConsoleWrite("$g_iRows = " & $g_iRows & "    $g_iCols = " & $g_iCols & @CRLF)
  Local $hTimer = TimerInit()
  $g_aArray = FAS_Random2DArrayAu3($g_iRows, "sifdtr", "abcdefghijklmnopqrstuvwxyz")
  For $i = 0 To $g_iRows - 1
    $g_tIndex.arr($i + 1) = $i
  Next
  ConsoleWrite("Generating array & one index = " & TimerDiff($hTimer) & @CRLF & @CRLF)
EndFunc   ;==>_Generate_All

;========================================================================
Func _SortArrayStruct(Const ByRef $aArray, $iCol, $iRows)
  Local $tIndex = DllStructCreate("uint arr[" & $iRows & "]")
  Local $pIndex = DllStructGetPtr($tIndex)
  Local Static $hDll = DllOpen("kernel32.dll")
  Local Static $hDllComp = DllOpen("shlwapi.dll")

  Local $lo, $hi, $mi, $r

  ; Sorting by one column
  For $i = 1 To $iRows - 1
    $lo = 0
    $hi = $i - 1
    Do
      $mi = Int(($lo + $hi) / 2)
      $r = DllCall($hDllComp, 'int', 'StrCmpLogicalW', 'wstr', $aArray[$i][$iCol], 'wstr', $aArray[DllStructGetData($tIndex, 1, $mi + 1)][$iCol])[0]
      Switch $r
        Case -1
          $hi = $mi - 1
        Case 1
          $lo = $mi + 1
        Case 0
          ExitLoop
      EndSwitch
    Until $lo > $hi
    DllCall($hDll, "none", "RtlMoveMemory", "struct*", $pIndex + ($mi + 1) * 4, "struct*", $pIndex + $mi * 4, "ulong_ptr", ($i - $mi) * 4)
    DllStructSetData($tIndex, 1, $i, $mi + 1 + ($lo = $mi + 1))
  Next

  Return $tIndex
EndFunc   ;==>_SortArrayStruct

;========================================================================
Func _UpdateIndex(ByRef $aIndex, $iCol)

  If VarGetType($aIndex[$iCol]) = "DLLStruct" Then
    $g_tIndex = $aIndex[$iCol]
  Else
    $g_tIndex = _SortArrayStruct($g_aSubArray, $iCol, $g_iSearch)
    $aIndex[$iCol] = $g_tIndex ; "DLLStruct" (or "Int32" when no match found +++)
  EndIf
EndFunc   ;==>_UpdateIndex

;========================================================================
Func _MoveMarker($iCol)

  Local $aRect = _GUICtrlHeader_GetItemRect($g_hHeader, $iCol)
  ControlMove($g_hGui, "", $g_idMarker, $g_iLeftLV + $aRect[0], $g_iTopLV - 3, $aRect[2] - $aRect[0] + 1, 3)
EndFunc   ;==>_MoveMarker

Special thanks to LarsJ, Jpm, Melba23, Nine, mikell and many others, you were all inspiring for the evolution of this script.

Update : May 06, 2021
* Added a 2nd way to quickly switch to a new search column : right click on its header !
* Added a GUI control allowing to switch from default RegEx search to a "normal" search (pic above)
* Enlarged search field (interesting for long RegEx patterns)
* LarsJ  RandomArray.au3 + Random2DArray.txt  attached below

RandomArray.au3Random2DArray.txt

Edited May 6, 2021 by pixelsearch

[mLipok]

Where is RandomArray.au3 ?

[pixelsearch]

Hi mLipok
I just attached "RandomArray.au3" at the end of the precedent post, also LarsJ's text file "Random2DArray.txt" with his own explanations.

A new update was done today, with an enlarged search field (pic in previous post). It could be interesting in case of long RegEx patterns.

Edited May 6, 2021 by pixelsearch

[Lion66]

Hi pixelsearch.

Good examples.
I admire how you understand this issue and modify your scripts.
Inspired by your examples, I thought it was possible to connect one of your versions, including sorting, with marking the resulting yellow color, as in this thread:

https://www.autoitscript.com/forum/topic/179112-incremental-search-in-owner-drawn-listview/#comments
But my programming level is not sufficient for such a task and it did not work.
If you find it possible to please us with the solution of this task, it would be interesting to see such a modification.
Thank you.

 

[LarsJ]

On 5/2/2021 at 4:30 AM, pixelsearch said:

would a GUI subclassing technique be preferable or not ?

In your code above, there is no immediate benefit to the subclassing technique. However, if you change the code to a UDF version, there is an advantage to the subclassing technique as you can avoid conflicts between WM_COMMAND and WM_NOTIFY message handles in both the UDF code and the user code.

 

On 5/4/2021 at 3:08 AM, pixelsearch said:

Not sure, as all these additionals tests on {Array} , {Map} , {DLLStruct} before displaying each element in _ArrayDisplay may slow down a bit the whole process in case of huge arrays. Anyway... the question is asked.

The performance impact of these tests will be the same for both a large and a small array because only visible rows are updated. If the listview contains 40 visible rows and 10 columns, then 400 items/subitems will be updated when you e.g. press PageUp/Down. It'll be the same for both a large and a small array. 400 simple tests to refresh an entire page will hardly be a problem.

 

In a WM_NOTIFY message handler with a very large number of messages, one must definitely pay attention to the performance of the code. Implementing custom draw code in a virtual listview will generate both a very large number of $LVN_GETDISPINFOW and $NM_CUSTOMDRAW notifications. As demonstrated in the "Too much code" section here, you can easily execute more code than there is time for with a subsequent crash as a consequence. If one insists on keeping all the code, the only resort is probably to use compiled code.

Therefore, it's advantageous to use arrays as indexes in $LVN_GETDISPINFOW and $NM_CUSTOMDRAW code instead of DllStructs because arrays are faster than DllStructs. Fortunately, there is a very fast method to convert a DllStruct of integers into a 1d array. The method is based on the techniques in Accessing AutoIt Variables. Instead of a pure AutoIt loop, a loop is executed in the internal AutoIt code in relation to these COM conversions. And this loop runs very fast as true compiled code:

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include <Array.au3>
#include "AccVars.au3"

Global $tIndex = DllStructCreate( "int[2000000]" )
For $i = 1 To 2000000
  DllStructSetData( $tIndex, 1, $i-1, $i )
Next

Global $iRows

$iRows =  100000
Runtimes()
$iRows =  250000
Runtimes()
$iRows =  500000
Runtimes()
$iRows =  750000
Runtimes()
$iRows = 1000000
Runtimes()
$iRows = 2000000
Runtimes()

Func Runtimes()
  Local $s = StringRegExpReplace( $iRows, "(\d{1,3}(?=(\d{3})+\z)|\d{3}(?=\d))", "\1,")
  ConsoleWrite( StringFormat( "%-30s", $s & " rows" ) & @CRLF )
  ConsoleWrite( StringFormat( "%-30s", "Copy $tIndex to $aArray   " ) & @CRLF )
  ConsoleWrite( StringFormat( "%-30s", "--------------------------" ) & @CRLF )

  ; AutoIt loop
  Local $hTimer = TimerInit()
  Local $aArray1[$iRows]
  For $i = 0 To $iRows - 1
    $aArray1[$i] = DllStructGetData( $tIndex, 1, $i+1 )
  Next
  ConsoleWrite( StringFormat( "%-30s", "AutoIt loop:     " & StringFormat( "%9.4f", TimerDiff( $hTimer ) ) ) & @CRLF )

  ; COM conversions
  $hTimer = TimerInit()
  Local $aArray2 = IntStructToArray()
  ConsoleWrite( StringFormat( "%-30s", "COM conversions: " & StringFormat( "%9.4f", TimerDiff( $hTimer ) ) ) & @CRLF & @CRLF )
  If $iRows = 100000 Then _ArrayDisplay( $aArray2 )

  #forceref $aArray1
EndFunc

Func IntStructToArray()
  Local $aArray
  AccessVariables01( IntStructToArrayMtd, $aArray )
  Return $aArray
EndFunc

; DllStruct of ints --> 1D array
Func IntStructToArrayMtd( $pvArray )
  ; Create SafeArray of integers
  Local $tSafeArrayBound = DllStructCreate( $tagSAFEARRAYBOUND )
  DllStructSetData( $tSafeArrayBound, "cElements", $iRows )
  Local $pSafeArray = SafeArrayCreate( $VT_I4, 1, $tSafeArrayBound )

  ; Copy data from DllStruct to SafeArray
  Local $pSafeArrayData
  SafeArrayAccessData( $pSafeArray, $pSafeArrayData )
  DllCall( "kernel32.dll", "none", "RtlMoveMemory", "struct*", $pSafeArrayData, "struct*", $tIndex, "ulong_ptr", $iRows*4 )
  SafeArrayUnaccessData( $pSafeArray )

  ; Store SafeArray in $pvArray (pointer to variant)
  DllStructSetData( DllStructCreate( "word", $pvArray ), 1, $VT_ARRAY + $VT_I4 )
  DllStructSetData( DllStructCreate( "ptr", $pvArray + 8 ), 1, $pSafeArray )

  ; On function exit internal AutoIt COM conversions will convert the safearray
  ; of integers in $pvArray to a safearray of variants ie. a true AutoIt array.
EndFunc

Runtimes:

expandcollapse popup

64 bit code on Windows 7      
==========================    

100,000 rows                  
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:      114.6483    
COM conversions:    4.4370    

250,000 rows                  
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:      284.5634    
COM conversions:   11.1708    

500,000 rows                  
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:      561.3047    
COM conversions:   22.2186    

750,000 rows                  
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:      838.6758    
COM conversions:   32.9380    

1,000,000 rows                
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:     1115.1990    
COM conversions:   43.7237    

2,000,000 rows                
Copy $tIndex to $aArray       
--------------------------    
AutoIt loop:     2227.6890    
COM conversions:   87.8566

IntStructToArray.7z

Edited May 8, 2021 by LarsJ

[pixelsearch]

Great explanations. Glad to have you back after 2 long weeks

[RAMzor]

Hi,

There is some way to  to redraw LV background lines colors as in LvVirtArrayColors.au3 example from 1ts post?

It is posible to fire $NM_CUSTOMDRAW inside WM_NOTIFY?

The reason is after deleting some rows (sending message $LVM_DELETEITEM) the line background colors are misordered. Sending $LVM_SETITEMCOUNT revert to view full data.

 

 

[dmob]

From a distance, sounds like you delete an item in LV but did not delete the item from the array you use to fill the LV?

[RAMzor]

Exactly. On second thought this is not a good idea. I must manage 2 arrays (second for bacup data and ability to restore data). Thanx in any way!

 

Now second question ☺️,

How to mark specific row with owerlayed painting (yellow for example) 

[Nine]

1 hour ago, RAMzor said:

How to mark specific row with owerlayed painting (yellow for example) 

Search for larsj topic on colors in LV...