ListViews with many columns

[LarsJ]

The main problem with a normal and also a virtual listview with many columns is that performance gets worse and worse as the number of columns increases. And the performance decrease starts at a relatively low number of columns and is significant.

Examples\Reference.au3 in the zip-file in bottom of post is a reference example. It's a virtual listview with 1,000 rows and 100 columns.

An owner drawn listview with the LVS_OWNERDRAWFIXED style is the listview that supports many columns. An owner drawn listview is updated through WM_DRAWITEM messages. One WM_DRAWITEM message is generated per visible listview row. No additional messages are generated for the listview columns.

If $iColFirst and $iColLast are the first and last visible listview column, then a listview row with index $itemID can be updated with this code:

; Draw visible columns/subitems
For $i = $iColFirst To $iColLast
  $sSubItmText = $itemID & " / " & $i
  DllStructSetData( $tRect, 2, $i ) ; Top
  DllStructSetData( $tRect, 1, $LVIR_BOUNDS ) ; Left
  GUICtrlSendMsg( $idListView, $LVM_GETSUBITEMRECT, $itemID, $pRect )
  If Not $i Then DllStructSetData( $tRect, 3, DllStructGetData( $tRect, 1 ) + GUICtrlSendMsg( $idListView, $LVM_GETCOLUMNWIDTH, 0, 0 ) )
  DllStructSetData( $tRect, 1, DllStructGetData( $tRect, 1 ) + 6 ) ; Left margin
  DllStructSetData( $tRect, 2, DllStructGetData( $tRect, 2 ) + 2 ) ; Top  margin
  DllCall( "user32.dll", "int", "DrawTextW", "handle", $hDC, "wstr", $sSubItmText, "int", StringLen( $sSubItmText ), "ptr", $pRect, "int", 0 ) ; _WinAPI_DrawText
Next

In an owner drawn listview you have to handle the subitem rectangles yourself and update the subitem texts with GDI functions like DrawText. But eg. grid lines are drawn automatically through default code.

$iColFirst and $iColLast can be calculated in a WM_NOTIFY message handler through LVN_ENDSCROLL notifications when listview columns are scrolled horizontally:

Case $LVN_ENDSCROLL
  If Not DllStructGetData( DllStructCreate( $tagNMLVSCROLL, $lParam ), "DX" ) Then Return $GUI_RUNDEFMSG ; Horizontal scrollbar only
  ; First and last visible column in ListView
  GUICtrlSendMsg( $idListView, $LVM_SUBITEMHITTEST, 0, $pHitTestFirst )
  $iColFirst = DllStructGetData( $tHitTestFirst, "SubItem" )
  GUICtrlSendMsg( $idListView, $LVM_SUBITEMHITTEST, 0, $pHitTestLast )
  $iColLast = DllStructGetData( $tHitTestLast, "SubItem" )
  ; Redraw ListView
  GUICtrlSendMsg( $idListView, $LVM_REDRAWITEMS, 0, $iRows - 1 ) ; Redraw ListView items, Forces $WM_DRAWITEM messages to update ListView control

In the examples the owner drawn style is combined with the virtual style, LVS_OWNERDATA, to be able to dynamically feed the listview with data directly from a data source.

 

Medium number of columns
If the number of columns is limited to a few hundred, the built-in listview header can support the columns.

The examples are stored in Examples\Medium\ in the zip-file. Run the examples in SciTE with F5.

FirstTest.au3 demonstrates the code snippets above.

TooManyCols.au3 shows what happens if the listview contains more columns than the built-in header is able to support. Drag the thumb in the horizontal scrollbar all the way to the right to see that the header items are simply missing.

In SelectedRows1.au3 you can select one or more rows. In SelectedRows2.au3 attempts have been made to reduce the flicker that occurs when the thumb in the horizontal scrollbar is swiftly dragged back and forth. More about flicker below.

Checkboxes.au3 and IconImages.au3 shows how to add checkboxes and icons. By drawing everything yourself, you can arbitrarily decide which columns should be provided with checkboxes, icons or both.

FillFromArray1.au3 and FillFromArray2.au3 demonstrates how to fill a listview from an array.

 

Large number of columns
For a large number of columns, 100,000 columns in most examples, you need to create the header control yourself with the functions in GuiHeader.au3. The header control isn't built into the listview but is an independent child window. The idea is to only show header items for visible listview columns. In ReSizeColumns.au3 listview columns can be resized to a minimum of 50 pixels. This makes a maximum of approx. 20 visible columns in the listview and therefore also a maximum of 20 header items. The first header item is always the header item with index 0. The second header item is always the header item with index 1. And so on. Header item texts (and header item widths if needed) are dynamically updated when listview columns are scrolled horizontally.

The examples are stored in Examples\Large\ in the zip-file. There are three subfolders named UserDrawn, OwnerDrawn and CustomDrawn. The names refer to the technique used to draw the header control. The listview is still owner drawn.

In UserDrawn\FirstTest.au3 the header control is updated directly in the LVN_ENDSCROLL section of the WM_NOTIFY message handler. It's simple code, but it generates a lot of flicker in the header control when the thumb in the horizontal scrollbar is dragged quickly back and forth. To avoid that much flicker, you can use either an owner or custom drawn header.

OwnerDrawn\FirstTest.au3 shows an owner drawn header control. Because an owner drawn header doesn't directly support themes and because the code in the WM_DRAWITEM message handler becomes a bit more complicated when there are two owner drawn controls, a custom drawn header is the preferred technique.
 

Custom drawn header
The basic code is shown in CustomDrawn\FirstTest.au3.

SelectedRows1.au3 implements selected items. If you select 10 items in a row so that a completely blue rectangle is created, and scrolls very quickly back and forth in the horizontal scrollbar, a lot of flicker occurs in the listview, especially in the blue rectangle. The flicker looks less violent if there are no selected rows.

In SelectedRows2.au3 attempts have been made to reduce flicker. If you keep an eye on the header control when scrolling quickly in the horizontal scrollbar, the header is updated much fewer times than the listview. This generates much less flicker in the header.

The reason why the header is updated fewer times is because the header isn't built into the listview but is an independent child window. The LVN_ENDSCROLL code updates the header and listview equally many times. But the header has to wait for WM_PAINT messages before it's updated. And a WM_PAINT message isn't generated until there is a delay in the number of WM_DRAWITEM messages. Ie. when there is a slight pause in the movement of the thumb in the horizontal scrollbar.

The idea to reduce flicker in the listview is only to update the subitem texts most times, and to update visual features and effects eg. selected rows that is the cause of the most flicker very few times.

This also improves performance during fast scrolls in the horizontal scrollbar. It doesn't take a lot of code just to update the subitem texts. Updating visual features and effects requires a lot more code.

To get an overview of the Windows messages and notifications and the order of the messages, a message monitor is indispensable.

ReSizeColumns.au3 demonstrates how to resize listview columns. Header item widths and texts are dynamically updated through NM_CUSTOMDRAW notifications when listview columns are scrolled horizontally. When a header item width is updated, it'll automatically generate new NM_CUSTOMDRAW notifications to update the text of the current header item and all header items to the right of the current one. The critical code is to avoid an endless loop of NM_CUSTOMDRAW notifications. Here too, a message monitor is indispensable.

GoToRowColumn.au3 implements code to go to a spicific row/column in the listview. Right click ListView, update row/column numbers in GUI and press Enter.

OneMillionColumns.au3 is an example with 1,000,000 columns. This example should be run on Windows 10 as 64 bit code only.

ShowRowNumbers.au3. If row numbers are needed in a standard few-column listview, the first column is usually used for row numbers. But this doesn't work well in a listview with many columns. Here it's better to use an extra listview to the left for row numbers.

When scrolling with the vertical scrollbar, the row numbers are updated accordingly. Note that the row numbers are updated discretely in the same way as the header when scrolling in the horizontal scroll bar.

There is no keyboard support in this example. If you press End to go to the last row, the row numbers are not updated. I'll add a new example with keyboard support.

KeyboardNavigation1.au3 (as well as 2 and 3) has keyboard support to navigate the listview with Home, End, PageUp and PageDown keys. Press Shift and one of the keys for horizontal navigation. As there is no selected row or cell in the examples, there is no support for the arrow keys.

Especially Shift+Pageup/Down generates a lot of flicker in KeyboardNavigation1.au3. KeyboardNavigation2.au3 and KeyboardNavigation3.au3 are attempts to reduce flicker. In KeyboardNavigation2.au3, auto repeat of the PageUp/Down keys is disabled. In KeyboardNavigation3.au3, only the top/left row/column is updated when PageUp/Down is kept pressed constantly.

Note that in these listviews with both many rows and many columns, performance and flicker appear to be two sides of the same coin. The better performance the less flicker and vice versa.

MarkCurrentCell1.au3. Click to mark a cell. Click again to unmark the cell. No keyboard support. It looks like this:

MarkCurrentCell2-4.au3 has keyboard support through the arrow keys as well as Home, End, PageUp and PageDown. Use Shift+Home/End/PageUp/PageDown for horizontal navigation. MarkCurrentCell3-4.au3 is an attempt to reduce flicker when PageUp/Down is kept pressed constantly.

EditMarkedCell.au3 is based on MarkCurrentCell2.au3 and can be used to edit the text in the marked cell. Double-click or press Enter to edit the cell text. Press Esc or Enter in the edit box to cancel or accept the new cell text.

ResizeGui.au3 is based on ShowRowNumbers.au3 and demonstrates a resizable GUI.

 

Usage of many-column listviews
A many-column listview with built-in header that supports a medium number of columns can be used as a regular listview. An ordinary listview performs poorly when there are many columns. This many-column listview based on owner drawn and virtual listview styles performs much better with the ability to optimize performance in specific situations.

It may be difficult to imagine a usage for many-column listviews with a custom header that supports a large number of columns. May be such a listview can be better regarded as a kind of grid control, that eg. can be used for visual editing of an array or SQLite table. You can imagine a 2d-array with 1000 rows and 1000 columns. The data source in EditMarkedCell.au3 is exactly such an array.

 

Zip-file
The zip-file contains source files for examples and includes.

You need AutoIt 3.3.12 or later. Tested on Windows 7 and Windows 10.

The examples should run on all Windows versions. If necessary, reduce the number of rows and especially columns on older versions.

Comments are welcome. Let me know if there are any issues.

ManyColumns.7z

Edited November 10, 2019 by LarsJ
Updates

[supersonic]

This is awesome! The dream of fast scrolling (and even larger) ListViews could come true…

IMHO – The only drawback this isn’t a set of UDFs. Any chance to build an UDF set based on your code? As a *complete* replacement to GUICtrl* and _GUICtrlListView_*?

[LarsJ]

supersonic, I don't think there is such a need for listviews with a large number of columns that it can justify to spend a lot of time making a complete UDF.

In your own post here it should be possible to implement such a listview based on the medium examples.

There is one important thing to consider when using an owner drawn listview: While the code in the WM_DRAWITEM message handler is running, it must not be interrupted by other Windows messages. If listview data (label texts) is stored directly in the listview (which is the case for normal listviews) and data is extracted from the listview in the WM_DRAWITEM code with _GUICtrlListView_GetItemText, it will generate WM_NOTIFY messages. These messages can cause a lot of problems as discussed in this thread. Therefore, it's convenient to combine an owner drawn listview with a virtual listview where data isn't stored directly in the listview but in an external data source eg. an array. Note that the problems in the thread are not an expression of an actual error in the AutoIt code. The problems are an indication that AutoIt as an interpreted language isn't fast enough to handle a large number of WM_DRAWITEM messages while at the same time handling an even larger number of WM_NOTIFY messages. In the code in first post in the thread there is both a WM_DRAWITEM and a WM_NOTIFY message handler. Exactly as in my examples here.

 

I have added two new examples and updated most other examples to make the code more uniform and correct some errors. New zip-file at bottom of first post.

[dmob]

19 hours ago, LarsJ said:

There is one important thing to consider when using an owner drawn listview:

@LarsJ Thank you for your tutorials. I have learned tons about listviews from your posts

[supersonic]

LarsJ, many thanks for going into detail and your new/updated examples. I will give it a try as soon as I have more leisure time.

 

[LarsJ]

The size of the cell rectangle given by column index $i and row index $itemID can be determined with this code:

DllStructSetData( $tRect, 2, $i ) ; Top
DllStructSetData( $tRect, 1, $LVIR_BOUNDS ) ; Left
GUICtrlSendMsg( $idListView, $LVM_GETSUBITEMRECT, $itemID, $pRect )

The first column when $i = 0 is the column representing the entire listview item. When $i = 0, the code above will result in a column width corresponding to the width of the entire listview item. If there are 100 columns each 90 pixels wide, it will result in a column width for column 0 of 9,000 pixels.

To calculate the real width of the cell rectangle for column 0, an extra line must be added:

DllStructSetData( $tRect, 2, $i ) ; Top
DllStructSetData( $tRect, 1, $LVIR_BOUNDS ) ; Left
GUICtrlSendMsg( $idListView, $LVM_GETSUBITEMRECT, $itemID, $pRect )
If Not $i Then DllStructSetData( $tRect, 3, DllStructGetData( $tRect, 1 ) + GUICtrlSendMsg( $idListView, $LVM_GETCOLUMNWIDTH, 0, 0 ) )

The cell rectangle calculation is used in WM_DRAWITEM code. This means that all examples are updated to add this line.

The code box at top of first post is corrected accordingly.

 

A number of new examples marked with red have been added in first post.

 

So far, these are all examples I've planned for the topic of many-column listviews.

New zip file at bottom of first post.

[BJR]

Hello LarsJ,

Your ListView controls Functions/Files/Codes are just great.

Thank you for sharing these files/codes.

If you've ListView groups functions, Can you Please also share those functions ?

ListView Group Functions:-

Set/GetLVGroupFont (Size,Color,Style,....) of any ListView
Set/GetLVGroupHandle
Set/GetLVGroupPosition
Set/GetLVGroupEnable/Disable
Set/GetLVGroupAdd/RemoveFromArray
Set/GetLVGroupAdd/Remove
Set/GetLVGroupNameHyperlink
Set/GetLVGroupCollapse/ExpandState

Set/GetLVGroupItemCount

Set/GetLVGroupSort
.....

 

Edited January 24, 2020 by BJR
Have missing

[LarsJ]

If you search carefully, then it is possible to find a few code examples that are about listview groups.

I don't use listview groups quite often, so I don't know the code completely. I'm not quite sure if all the functions can be implemented. But an example regarding listview groups is certainly not unattractive. I've made a note on the subject. But don't expect any code already this coming weekend.

[BJR]

15 hours ago, LarsJ said:

If you search carefully, then it is possible to find a few code examples that are about listview groups.

I don't use listview groups quite often, so I don't know the code completely. I'm not quite sure if all the functions can be implemented. But an example regarding listview groups is certainly not unattractive. I've made a note on the subject. But don't expect any code already this coming weekend.

 

Thank You.