UIAutomation - Can't get the value of a Text control in a ListItem

[FrancescoDiMuro]

Good afternoon everyone
Third day into learning UIAutomation and I am struggling a bit to read the value of a Text control which is contained by a ListItem control.
I am trying to learn UIAutomation following @LarsJ's UIAutomation tutorial, so I'd be very happy to get some advices without mixing this approach with the @junkew's one.
Below, there are the ListItem and the Text control inspected with UIASpy.

ListItem control:

Spoiler

Text control:

Spoiler

This is the code I'm using to read the various Text controls:

Spoiler

expandcollapse popup

#include "CUIAutomation2.au3"
#include <GuiListView.au3>

Opt("MustDeclareVars", 1)

Test()

Func Test()

    Local $oUIAutomation, _
          $pDesktop, _
          $oDesktop, _
          $pWindow, _
          $oWindow, _
          $pCondition, _
          $pPane1, _
          $oPane1, _
          $pPane2, _
          $oPane2, _
          $pListView, _
          $oListView, _
          $pListItems, _
          $oListItems, _
          $iListItems, _
          $pListItem, _
          $oListItem, _
          $pTextControls, _
          $oTextControls, _
          $iTextControls, _
          $pText, _
          $oText, _
          $sText, _
          $oValue, _
          $pValue

    ConsoleWrite(@CRLF & "----- UIAutomation Object -----" & @CRLF)

    ; Create UI Automation object
    $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
    ConsoleWrite("$oUIAutomation OK" & @CRLF)

    ConsoleWrite("----- Desktop -----" & @CRLF)

    ; Get Desktop element
    Global $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement($pDesktop)
    $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
    ConsoleWrite("$oDesktop OK" & @CRLF)

    ConsoleWrite("----- Main Window -----" & @CRLF)

    $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "WindowName", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oDesktop.FindFirst($TreeScope_Descendants, $pCondition, $pWindow)
    $oWindow = ObjCreateInterface($pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oWindow) Then Return ConsoleWrite("$oWindow ERR" & @CRLF)
    ConsoleWrite("$oWindow OK" & @CRLF)

    ConsoleWrite("----- First Pane -----" & @CRLF)

    ; Why these two instructions are not working?
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "AfxControlBar140s", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "59422", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_PaneControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oWindow.FindFirst($TreeScope_Descendants, $pCondition, $pPane1)
    $oPane1 = ObjCreateInterface($pPane1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oPane1) Then Return ConsoleWrite("$oPane1 ERR" & @CRLF)
    ConsoleWrite("$oPane1 OK" & @CRLF)

    ConsoleWrite("----- Second Pane -----" & @CRLF)

    ; Same question of Pane1
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "Afx:00790000:b:00010003:00000000:00000000", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "81", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_PaneControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane1.FindFirst($TreeScope_Descendants, $pCondition, $pPane2)
    $oPane2 = ObjCreateInterface($pPane2, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oPane2) Then Return ConsoleWrite("$oPane2 ERR" & @CRLF)
    ConsoleWrite("$oPane2 OK" & @CRLF)

    ConsoleWrite("----- ListView -----" & @CRLF)

    #cs

    ; The ListView, even if has two parents pane, is not found is the second one, but in the first one

    ; Same question of Pane1
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "SysListView32", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "33217", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_ListControlTypeId, $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_NativeWindowHandlePropertyId, "0x0005091A", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_RuntimeIdPropertyId, "330010", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane2.FindFirst($TreeScope_Children, $pCondition, $pListView)
    $oListView = ObjCreateInterface($pListView, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oListView) Then Return ConsoleWrite("$oListView ERR" & @CRLF)
    ConsoleWrite("$oListView OK" & @CRLF)

    #ce

    $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "SysListView32", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane1.FindFirst($TreeScope_Children, $pCondition, $pListView)
    $oListView = ObjCreateInterface($pListView, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oListView) Then Return ConsoleWrite("$oListView ERR" & @CRLF)
    ConsoleWrite("$oListView OK" & @CRLF)

    ConsoleWrite("----- List Items -----" & @CRLF)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_ListItemControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oListView.FindAll($TreeScope_Children, $pCondition, $pListItems)
    $oListItems = ObjCreateInterface($pListItems, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    $oListItems.Length($iListItems)
    If Not $iListItems Then Return ConsoleWrite("$oListItems ERR" & @CRLF)
    ConsoleWrite("$oListItems OK" & @CRLF)

    For $i = 0 To $iListItems - 1

        $oListItems.GetElement($i, $pListItem)
        $oListItem = ObjCreateInterface($pListItem, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

        $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_TextControlTypeId, $pCondition)
        If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
        ConsoleWrite("$pCondition OK" & @CRLF)

        ConsoleWrite("----- Text Controls -----" & @CRLF)

        $oListItem.FindAll($TreeScope_Descendants, $pCondition, $pTextControls)
        $oTextControls = ObjCreateInterface($pTextControls, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
        $oTextControls.Length($iTextControls)
        If Not $iTextControls Then Return ConsoleWrite("$oTextControls ERR" & @CRLF)
        ConsoleWrite("$oTextControls OK" & @CRLF)

        For $j = 0 To $iTextControls - 1

            $oTextControls.GetElement($j, $pText)
            $oText = ObjCreateInterface($pText, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

            ; None of these methods are working
            ; $oText.GetCurrentPropertyValue($UIA_NamePropertyId, $sText)
            ; $oText.GetCurrentPropertyValue($UIA_ValueValuePropertyId, $sText)

            $oText.GetCurrentPattern($UIA_LegacyIAccessiblePatternId, $pValue)
            $oValue = ObjCreateInterface($pValue, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern)
            If Not IsObj($oValue) Then Return ConsoleWrite("$oValue ERR" & @CRLF)
            ConsoleWrite("$oValue OK" & @CRLF)

            ; $oValue.CurrentName($sText)
            $oValue.CurrentValue($sText)

            ConsoleWrite("Value of [" & $i & "," & $j & "] item: " & $sText & @CRLF)
        Next

    Next

ConsoleWrite(@CRLF)

EndFunc

 

This is the output from the SciTE console:

Spoiler

expandcollapse popup

----- UIAutomation Object -----
$oUIAutomation OK
----- Desktop -----
$oDesktop OK
----- Main Window -----
$pCondition OK
$oWindow OK
----- First Pane -----
$pCondition OK
$oPane1 OK
----- Second Pane -----
$pCondition OK
$oPane2 OK
----- ListView -----
$pCondition OK
$oListView OK
----- List Items -----
$pCondition OK
$oListItems OK
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [0,0] item: 
$oValue OK
Value of [0,1] item: 
$oValue OK
Value of [0,2] item: 
$oValue OK
Value of [0,3] item: 
$oValue OK
Value of [0,4] item: 
$oValue OK
Value of [0,5] item: 
$oValue OK
Value of [0,6] item: 
$oValue OK
Value of [0,7] item: 
$oValue OK
Value of [0,8] item: 
$oValue OK
Value of [0,9] item: 
$oValue OK
Value of [0,10] item: 
$oValue OK
Value of [0,11] item: 
$oValue OK
Value of [0,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [1,0] item: 
$oValue OK
Value of [1,1] item: 
$oValue OK
Value of [1,2] item: 
$oValue OK
Value of [1,3] item: 
$oValue OK
Value of [1,4] item: 
$oValue OK
Value of [1,5] item: 
$oValue OK
Value of [1,6] item: 
$oValue OK
Value of [1,7] item: 
$oValue OK
Value of [1,8] item: 
$oValue OK
Value of [1,9] item: 
$oValue OK
Value of [1,10] item: 
$oValue OK
Value of [1,11] item: 
$oValue OK
Value of [1,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [2,0] item: 
$oValue OK
Value of [2,1] item: 
$oValue OK
Value of [2,2] item: 
$oValue OK
Value of [2,3] item: 
$oValue OK
Value of [2,4] item: 
$oValue OK
Value of [2,5] item: 
$oValue OK
Value of [2,6] item: 
$oValue OK
Value of [2,7] item: 
$oValue OK
Value of [2,8] item: 
$oValue OK
Value of [2,9] item: 
$oValue OK
Value of [2,10] item: 
$oValue OK
Value of [2,11] item: 
$oValue OK
Value of [2,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [3,0] item: 
$oValue OK
Value of [3,1] item: 
$oValue OK
Value of [3,2] item: 
$oValue OK
Value of [3,3] item: 
$oValue OK
Value of [3,4] item: 
$oValue OK
Value of [3,5] item: 
$oValue OK
Value of [3,6] item: 
$oValue OK
Value of [3,7] item: 
$oValue OK
Value of [3,8] item: 
$oValue OK
Value of [3,9] item: 
$oValue OK
Value of [3,10] item: 
$oValue OK
Value of [3,11] item: 
$oValue OK
Value of [3,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [4,0] item: 
$oValue OK
Value of [4,1] item: 
$oValue OK
Value of [4,2] item: 
$oValue OK
Value of [4,3] item: 
$oValue OK
Value of [4,4] item: 
$oValue OK
Value of [4,5] item: 
$oValue OK
Value of [4,6] item: 
$oValue OK
Value of [4,7] item: 
$oValue OK
Value of [4,8] item: 
$oValue OK
Value of [4,9] item: 
$oValue OK
Value of [4,10] item: 
$oValue OK
Value of [4,11] item: 
$oValue OK
Value of [4,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [5,0] item: 
$oValue OK
Value of [5,1] item: 
$oValue OK
Value of [5,2] item: 
$oValue OK
Value of [5,3] item: 
$oValue OK
Value of [5,4] item: 
$oValue OK
Value of [5,5] item: 
$oValue OK
Value of [5,6] item: 
$oValue OK
Value of [5,7] item: 
$oValue OK
Value of [5,8] item: 
$oValue OK
Value of [5,9] item: 
$oValue OK
Value of [5,10] item: 
$oValue OK
Value of [5,11] item: 
$oValue OK
Value of [5,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [6,0] item: 
$oValue OK
Value of [6,1] item: 
$oValue OK
Value of [6,2] item: 
$oValue OK
Value of [6,3] item: 
$oValue OK
Value of [6,4] item: 
$oValue OK
Value of [6,5] item: 
$oValue OK
Value of [6,6] item: 
$oValue OK
Value of [6,7] item: 
$oValue OK
Value of [6,8] item: 
$oValue OK
Value of [6,9] item: 
$oValue OK
Value of [6,10] item: 
$oValue OK
Value of [6,11] item: 
$oValue OK
Value of [6,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [7,0] item: 
$oValue OK
Value of [7,1] item: 
$oValue OK
Value of [7,2] item: 
$oValue OK
Value of [7,3] item: 
$oValue OK
Value of [7,4] item: 
$oValue OK
Value of [7,5] item: 
$oValue OK
Value of [7,6] item: 
$oValue OK
Value of [7,7] item: 
$oValue OK
Value of [7,8] item: 
$oValue OK
Value of [7,9] item: 
$oValue OK
Value of [7,10] item: 
$oValue OK
Value of [7,11] item: 
$oValue OK
Value of [7,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [8,0] item: 
$oValue OK
Value of [8,1] item: 
$oValue OK
Value of [8,2] item: 
$oValue OK
Value of [8,3] item: 
$oValue OK
Value of [8,4] item: 
$oValue OK
Value of [8,5] item: 
$oValue OK
Value of [8,6] item: 
$oValue OK
Value of [8,7] item: 
$oValue OK
Value of [8,8] item: 
$oValue OK
Value of [8,9] item: 
$oValue OK
Value of [8,10] item: 
$oValue OK
Value of [8,11] item: 
$oValue OK
Value of [8,12] item:

 

As you can see, I posted some questions in the code itself because I tried different methods to do such task.
I already tried with _GUICtrlListView_GetItemText* and ControlListView() functions once "arrived" to the ListView control, but none of them worked.
Could you please direct me to the right approach to help me achieve this?
Thank you in advance.

Cheers

[LarsJ]

Good to see that someone is at least trying.

When you cannot see the text value in UIASpy, I don't think you can create code to read the value yourself. This is probably because the control doesn't have adequate UI Automation support.

And since your own experiments with classic code do not work either, it seems that the control completely lacks automation code.

You can try with MSAA code in Inspect.exe. Switch to MSAA mode through the dropdown control in the upper left corner.

If that does not work either, OCR techniques are probably the only solution.

Edited January 6, 2021 by LarsJ
OCR techniques

[FrancescoDiMuro]

@LarsJ
Thanks for bumping in!
I had to download and install the Windows SDK because the link provided was broken; so, I seize the opportunity to upload it here.
I tried inspecting the Control using MSAA mode, and these are the results:

ListView control:

Spoiler

ListItem control:

Spoiler

I was looking at the ListItem control (guessing that "Voce di elenco" is a ListItem and "Elenco" is a ListView), the Value is null, and so, I don't know if I can retrieve any value from this ListView (probably not).
Just for curiosity, I inspected the same controls with UIAutomation mode, and these are the results:

ListView control:

Spoiler

ListItem control:

Spoiler

Text control:

Spoiler

I'd like to thank you once again for the amazing tool you made, which is very simple to use and it gives you a lot of information and creates code out of the box.

Cheers

[FrancescoDiMuro]

Well.
The only obtainable thing is the header from the ListView, but not the ListItem texts themselves.
Seems that this problem is unsolvable.
Thanks anyone for the help.

[LarsJ]

You should still be able to use OCR techniques (there are many examples of using Tesseract) to obtain the value of the text control.

Implement the following functionality in a script:

• Get the bounding rectangle of the text control with UIA code
• Make a screenshot of this rectangle and save it as an image
• Use Tesseract to read the text in the image
• Validate the output text from Tesseract

I myself have used this technique successfully a few times.

[FrancescoDiMuro]

@LarsJ

Thank you for you help once again.

After some days of searching, I l've seen that the software has its own scripting language to interact with the data displayed in the controls, and so, I'm gonna study it since it's C++, and I always wanted to re-use it since I have studied it in high school.

Thanks