UIA doesn't allow to click on a specific button [Solved]

[ChessMate]

Hello,
I have currently a problem with one website.
By browsing the forum I have managed to create a script working perfectly to find any button on a webpage, wait for the refreshing and doing the needed operation.
However, there is one button, the final one, "SAUVEGARDER" = "SAVE" which I can't activate, nor with invoke, nor with leftclick, nor with any other idea I had.

A mouseclick on its position works perfectly, but it is resolution sensitive so I would prefer avoid it.
 

Could you please either share me a way to get the position of the UIA (it could be enough as parameters for MouseClick to solve the problem), or any other idea to solve this.
I am stuck with this for weeks and the forum does not seem to have a similar issue resolved.

It is a script inspired by the example 6 & 7 of Junkew : 

 

Edited January 14, 2022 by ChessMate

[ChessMate]

I forgot to mention. The two other buttons next to the problematic one works perfectly and base on Inspect are of the exact same class so I have absolutely no idea of the reason behind this issue :

Here is the Inspect log for this button : 

[junkew]

Give some more information based on other spying tools either simplespy or the UIASpy on what they reveal. (and try the code they can generate for you)
Certainly on the issupported event actions you can see if invoke/click is supported

• Can you highlight it?
• Can you setfocus to it?

If above 2 are yes then its interesting to see from the spy tools what their output is on the supported actions/events (as it then finds the correct object its just analysing the properties from UIA of that element on what you can/cant achieve)

• Make sure the button's are enabled
• I am not sure what your issue is on "A mouseclick on its position works perfectly, but it is resolution sensitive so I would prefer avoid it"
  If you can find it then the coordinates returned are mouseclickable for that resolution you are working on.
  It could be you have to be implementing some things for DPI awareness.

 

 

 

[ChessMate]

21 hours ago, junkew said:

Give some more information based on other spying tools either simplespy or the UIASpy on what they reveal. (and try the code they can generate for you)

The code generated by UIASPy has had no effect.

Certainly on the issupported event actions you can see if invoke/click is supported

• Can you highlight it?
• Can you setfocus to it?

If above 2 are yes then its interesting to see from the spy tools what their output is on the supported actions/events (as it then finds the correct object its just analysing the properties from UIA of that element on what you can/cant achieve)

• Make sure the button's are enabled
  
  
  Invoke seems supported and the button is activated. I don't see any inormation about highlight and setfocus but I have tried my code without them witout any difference.

 

• I am not sure what your issue is on "A mouseclick on its position works perfectly, but it is resolution sensitive so I would prefer avoid it"
  If you can find it then the coordinates returned are mouseclickable for that resolution you are working on.
  It could be you have to be implementing some things for DPI awareness.
  
  This one was just to explain that I tried a mouseclick without any UIA feature with the script on the coordinates on the button and it works.
  It was to check if it was some kind of scriptproof feature which would prevent the interaction. It happens it is not the case.
  

 

 

 

 

[junkew]

Hard to help.

Look at the 3 different buttons with spy tool if you see a difference.

Build 3 pieces of code for each button the same to find them and print the information for name, classname, x,y,h,w to see if you really are finding the correct element. Look for spaces before and after the name.

If you find the element and have xyhw do a mouseclick instead of invoke or try defaultaction method.

[ChessMate]

Hello Junkew,

I just realized my tests of the sample code was wrong, because I have an unknown variable error.


It seems like the variables defined locally are considered as a problem.
They are positionned as return variable so they should not create any problem.

The UIA_constants.au3 variables are not the problem as I can replace them with their absolute values without any change.

If I can get either the hyperlink or the GetProperties working, I should manage.


By the way, the "RÉINITIALISER" button and the "SAUVEGARDER" one have the exact same caracteristics in the spy tool and one react with the invoke while the other one don't.

[junkew]

Is oparent an object? Can you highlight oparent?

[ChessMate]

Parents are Groups with "Click on Ancestor Option" until the highlighted one here which just click.

Each of them is unfocusable :

 

[junkew]

I mean more from you coding. For debugging finding element you have to make sure you found the oParent and can highlight that from there you can then do the findfirst.

With findall you can try if its found properly 

As said without access to the website its hard to say why it does not work.

[ChessMate]

I am sure I can get access to the first and second button.

These three buttons have the same ancestor and I know the "SAUVEGARDER" button is the last child (number 2) so to be sure of getting it I could try something like that :

 

$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)

    If Not IsObj($oDocument) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    Else
        $t = StringSplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId), ";")
        _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

    EndIf
    Sleep(500)

     $oSavePlease = _UIA_getObjectByFindAll($oDocument, "name:=RÉINITIALISER", $treescope_subtree)
    If Not IsObj($oSavePlease) Then
        _UIA_DumpThemAll($oDocument, $treescope_subtree)
        ConsoleWrite("Test")
     Else

$UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    Local $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(0, "UI automation treewalker failed", "UI Automation failed failed")
    EndIf

    ;;~  at least 1 assumed (assuming we are not spying the desktop)
    Local $oParentHandle
    $oTW.GetParentElement($oSavePlease, $oParentHandle)

    $oSVP=_UIA_getFirstObjectOfElement(§oParentHandle,"index:=2", $treescope_subtree)
    If Not IsObj($oSVP) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    Else

    _UIA_action($oSVP, "focus")
    _UIA_action($oSavePlease, "click")


However it is not working due to the handle format.




Is there any way to directly get the coordinates from an object created with _UIA_getObjectByFindAll ?
It could prove me if I am selecting the right button and if it is the case, I just have to put these coordinates in a mouseclick and the problem is solved.


        

 

[junkew]

As Inspect sees your elements they should be reachable and if you reach it yes you can invoke or mouseclick on the coordinates

You should try with something like this

findThemAll($oDocument, $treescope_subtree)

be aware if you have a big html document with thousands of elements its better to start in a lower part of the tree

• Most likely there are other elements with same name/title/text
• Classname differs
• Spaces before or after
• ....

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

[ChessMate]

Hello, I have tried it both with $Treescope = 1 and with a $Treescope_subtree.

Both return 

So the script seems to don't get any classnamepropertyid from the UI_element, but the UI_element exist and work as we get the name correctly.
The classnamepropertyid value is 30012 as intended too so the problem doesn't come from it.

I have tried it too with the RÉINITIALISER button which I can invoke and I don't get the class either :
 


Here is the tried code. 

$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)

    If Not IsObj($oDocument) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    Else
        $t = StringSplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId), ";")
        _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

    EndIf
    Sleep(500)

     $oSavePlease = _UIA_getObjectByFindAll($oDocument, "name:=SAUVEGARDER", $treescope_subtree)
   If Not IsObj($oSavePlease) Then
        _UIA_DumpThemAll($oDocument, $treescope_subtree)
        ConsoleWrite("Test")
     Else
        findThemAll($oSavePlease, $treescope_subtree)

[junkew]

Please put the code in coding tags to keep it readable
Just run a lengthy dump first from your whole page and then see if the name is not there multiple times

findThemAll($oChrome, $treescope_subtree)

findThemAll is just a function helpfull for debugging so you have to extend/customize to your own need. It looks like it finds the element so you could add in the for/next loop

if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)= "SAUVEGARDER" then
   $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
        _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
endif

Is there any way to directly get the coordinates from an object created with _UIA_getObjectByFindAll ?

yes, see above how to get the bounding rectangle so it becomes something like below

$oUIElement=_UIA_getObjectByFindAll(.....
$t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
_UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

 

[ChessMate]

Sorry, I didn't recognize the coding tags before.

I found a second occurrence of the "SAUVEGARDER" name thanks to the findThemAll function.
As there is no other occurence of the two other buttons I guess it is indeed the issue with the "SAUVEGARDER" one.


I manage to find the other occurence in SpyTool and it is of a different controltype so I put a condition to forcefully avoiding targeting the wrong one.
 

$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)

    If Not IsObj($oDocument) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    Else
        $t = StringSplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId), ";")
        _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

    EndIf
    Sleep(500)

     $oSavePlease = _UIA_getObjectByFindAll($oDocument, "name:=SAUVEGARDER,controltype:="& $UIA_HyperlinkControlTypeId, $treescope_subtree)

   If Not IsObj($oSavePlease) Then
        _UIA_DumpThemAll($oDocument, $treescope_subtree)
        ConsoleWrite("Test")
     Else
      if _UIA_getPropertyValue($oSavePlease, $UIA_NamePropertyId)= "SAUVEGARDER" then
         $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
         _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
      Endif
      ConsoleWrite($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

        _UIA_action($oSavePlease, "focus")
        _UIA_action($oSavePlease, "click")
        _UIA_action($oSavePlease, "invoke")

It still does not work so I guess I should use another criteria to separate the objects.
I guess the _UIA_getPropertyValue  is unnecessary if I can pinpoint he right element, however right now it proves I am still targeting the wrong one as the $t is not returning anything. If controlType does not work, what else would you recommend to use ?

Here is the right "SAUVEGARDER" : 


And the wrong one :


 
 
  

[junkew]

Be aware there is a class property and a controltype property. Seems you are messing up with those 2 properties

[junkew]

"name:=SAUVEGARDER; index:=0"
"name:=SAUVEGARDER; index:=1"

The UIA Wrappers are mainly made to make syntax like above possible with index and indexrelative and all based on findall of UIA iterating the array and filtering out with a more human friendly syntax instead of creating all the conditions and anding them together. But as I sofar never had time to completely finish it there are still potential bugs in it.

Edited January 7, 2022 by junkew
changed comma to semicolon

[ChessMate]

I don't see any class information so I try to use what I can find.

If I put : 

"name:=SAUVEGARDER,controltype:="& $UIA_HyperlinkControlTypeId

or

"name:=SAUVEGARDER, index:=1"

or even index:=0,

the object is not found.

The only class I can found is in the dev tool : 
and it does not work as comparison too.
 

"name:=SAUVEGARDER,class:=btn success large ladda-button"

 

[junkew]

Sorry syntax is with a semicolon

;~ _UIA_setVar("MACH3.mnuFile","name:=((File)|(Bestand));index:=2")

First get it debugged with findthemall modification's like

if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="SAUVEGARDER" then
            consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)
        endif

Integrated like below

 

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        
        if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="SAUVEGARDER" then
            $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
            _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
            consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)           
        endif
        
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

Edited January 7, 2022 by junkew

[ChessMate]

Thank you Junkew

It worked with the controltype criteria !
 

$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)

    If Not IsObj($oDocument) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    Else
        $t = StringSplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId), ";")
        _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

    EndIf
    Sleep(500)

     $oSavePlease = _UIA_getObjectByFindAll($oDocument, "name:=SAUVEGARDER;controltype:="& $UIA_HyperlinkControlTypeId, $treescope_subtree)

   If Not IsObj($oSavePlease) Then
        _UIA_DumpThemAll($oDocument, $treescope_subtree)
        ConsoleWrite("Test")
     Else
        _UIA_action($oSavePlease, "invoke")
   EndIf

So the summary :

The problem was due to two occurences of the same name in the treescope.
It was discovered thanks to the "FindThemAll" function.

The script was always calling the first one, which was the wrong one.

By using the Spytool, the difference between the two objects became apparent.
The two object had for example a different controltype.

Adding the controltype criteria has thus allowed to target the right object.

"name:=SAUVEGARDER;controltype:="& $UIA_HyperlinkControlTypeId

The button is now correctly targeted and can be invoked.

Thank you very much Junkew !