IUIAutomation MS framework automate chrome, FF, IE, ....

[hnakamur]

Hi, junkew

 

Thanks for your great scripts and examples! 

I would like to ask you under what license can I use these scripts?

 

Thanks.

[marko001]

I tested using mainly this func:

Func oIUIAutomationEventHandler_HandleAutomationEvent( $pSelf, $pSender, $iEventId ) ; Ret: long  Par: ptr;int
  ConsoleWrite( "oIUIAutomationEventHandler_HandleAutomationEvent: " & $iEventId & @CRLF )
  If $iEventId <> $UIA_Window_WindowClosedEventId Then
    Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    $oSender.AddRef()
    if _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) = "Shiny.Chat.ViewModels.PrivateChatSessionViewModel" Then
        ConsoleWrite(_UIA_getAllPropertyValues($oSender) & @CRLF)
        MsgBox(0,"","");
    EndIf
    ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                  "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                  "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                  "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                  "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                  "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  EndIf
  Return $S_OK
EndFunc

 

In the nested IF i check if the pop-up is what i'm looking for (and I suppose I need to check with its $UIA_NamePropertyId).

Once I get it, I need to close it.

I looked trough all the _UIA_getAllPropertyValues($oSender) but couldn't find the absolute position of the pop-up neither the close button coords.

These datas are available if che pop-up is clicked (as shown in simplespy.au3).

How can I retrieve this information and mouseclick it? (or how can I close the popup trough code?)

Thanks a lot,

Marco

[junkew]

Also different solutions are possible

a. Combine it with example 1 a treewalker to see the components of the window itself

use properties like UIA_BoundingRectanglePropertyId on those components to find the area for clicking

b. easiest I think is to retrieve the native hwnd: UIA_NativeWindowHandlePropertyId and with other autoit commands just send a WinClose(hWnd)

[marko001]

got the handle but, probably since it isn't a standard window, winclose(_UIA_getPropertyValue($oSender, $UIA_NativeWindowHandlePropertyId)) doesn't close it.

I worked around the ex1 and added controls for position:

Func sampleWinList()
    Local $var = WinList()
    $pos = WinGetPos("Network Messages")
    For $i = 1 To $var[0][0]
        ; Only display visble windows
        If $var[$i][0] <> "" And IsVisible($var[$i][1]) Then
            If $var[$i][0] = "Network Messages" Then
                ConsoleWrite("Title=" & $var[$i][0] & @TAB & "Handle=" & $var[$i][1] & @TAB & "Pos: " &$pos[0] & "-" & $pos[1] & "-" & $pos[2] & "-" & $pos[3] & @CRLF)
            Else
                ConsoleWrite("Title=" & $var[$i][0] & @TAB & "Handle=" & $var[$i][1] & @CRLF)
            EndIf
        EndIf
    Next
EndFunc   ;==>sampleWinList

 

in this case, simple winlist()  the popup (name is "Network Messages") appears in list with its coords)

Func sampleTW($t)
    ConsoleWrite("initializing tw " & $t & @CRLF)
;~ ' Lets show all the items of the desktop with a treewalker
    If $t = 1 Then $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    If $t = 2 Then $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)
    If $t = 3 Then $UIA_oUIAutomation.ContentViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

    While IsObj($oUIElement) = True
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Handle=" & Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB & "Rect: " & _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId) & @CRLF)
        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    WEnd
EndFunc   ;==>sampleTW

 

in this case the window doesn't appear in list except in "All childs of taskbar". 

Probably because this is child of the main program ?

Edited July 2, 2014 by marko001

[junkew]

no winlist ignores the hierarchy

whereas a treewalker gives you options to walk thru the tree

with getparent, getfirstchild, getnextsibling

as in the example you only walk thru the childs of the desktop but not the grandchildren

[marko001]

and how do I get, for example, the grandchildren?

I mean, winlist() give me the 2 windows needed:

Medical Center (main of MedicalCenter.exe)

Network Messages (child of MedicalCenter.exe)

and _UIA_getPropertyValue() gives me only the first

[EDIT]:

I used

findThemAll($UIA_oDesktop, $treescope_subtree)
...
...
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        If StringInStr(_UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId),"Network Message") <> 0 Then
            ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB & "Rect: " & _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId) & @CRLF)
        EndIf

    Next
 

 

This way is a bit slow, if I put no conditions I can see hundreds results.

Edited July 2, 2014 by marko001

[junkew]

The logic in a treewalker in a tree of depth 2 is

A ---> B    ---> D

|         +-------->E

+----->C ------>F

          +------->G

So from A you getFirstChild which is B and if you want to go to C you ask for nextSibling if you want to go to D you ask for firstChild (and as its a tree it can be done more recursively deep in the tree)

 

This is a combination of example 19 and example 1

It checks for the event and sees if its class #32770 and then will print the items of that dialogbox by making a treewalker on the #32770 popupped window

 

expandcollapse popup

;~ Example 19a events
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

Opt( "MustDeclareVars", 1 )

;~ Global Const $S_OK = 0x00000000
;~ Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationEventHandler, $oIUIAutomationEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationEventHandler = ObjectFromTag( "oIUIAutomationEventHandler_", $dtagIUIAutomationEventHandler, $tIUIAutomationEventHandler, True )
  If Not IsObj( $oIUIAutomationEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  ; Window open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  ; Menu open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationEventHandler )
  Exit
EndFunc



;~ Func _UIA_getPropertyValue( $obj, $id )
;~   Local $tVal
;~   $obj.GetCurrentPropertyValue( $id, $tVal )
;~   Return $tVal
;~ EndFunc

Func oIUIAutomationEventHandler_HandleAutomationEvent( $pSelf, $pSender, $iEventId ) ; Ret: long  Par: ptr;int
    dim $t
;~   ConsoleWrite( "oIUIAutomationEventHandler_HandleAutomationEvent: " & $iEventId & @CRLF )
  If $iEventId <> $UIA_Window_WindowClosedEventId Then
    Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    $oSender.AddRef()

        ;~catch popup
        if _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId )= "#32770" then

        ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                      "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                      "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                      "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                      "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                      "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )

    ;~  javascript:alert("hello");

        ConsoleWrite("initializing tw " & $t & @CRLF)
    ;~ ' Lets show all the items of the desktop with a treewalker
    ;~  If $t = 1 Then $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    ;~  If $t = 2 Then $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)

    $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)

    ;~  If $t = 3 Then $UIA_oUIAutomation.ContentViewWalker($UIA_pTW)

        $UIA_oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
        If IsObj($UIA_oTW) = 0 Then
            MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
        EndIf

        $UIA_oTW.GetFirstChildElement($oSender, $UIA_pUIElement)
        $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

        While IsObj($UIA_oUIElement) = True
            ConsoleWrite("Title is: " & _UIA_getPropertyValue($UIA_oUIElement, $UIA_NamePropertyId) & @TAB & "Handle=" & Hex(_UIA_getPropertyValue($UIA_oUIElement, $UIA_NativeWindowHandlePropertyId)) & @TAB & "Class=" & _UIA_getPropertyValue($UIA_oUIElement, $uia_classnamepropertyid) & @CRLF)
            $UIA_oTW.GetNextSiblingElement($UIA_oUIElement, $UIA_pUIElement)
            $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        WEnd

      EndIf
  EndIf

  Return $S_OK
EndFunc

Func oIUIAutomationEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
;~     ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationEventHandler Then
;~     ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUIAutomationEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
;~     ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationEventHandler_AddRef( $pSelf ) ; Ret: ulong
;~   ConsoleWrite( "oIUIAutomationEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationEventHandler_Release( $pSelf ) ; Ret: ulong
;~   ConsoleWrite( "oIUIAutomationEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
;~             ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
;~         ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

I test this with

1. run script

2. go to IE and drop bookmarklet in address bar

javascript:alert("hello");

Edited July 2, 2014 by junkew

[marko001]

I got it and using that example I found the popup:

I changed the IF to _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId) = "50032" to fullfil my request

If _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId) = "50032" Then



        ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _

                      "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _

                      "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _

                      "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _

                      "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _

                      "Pos       = " & _UIA_getPropertyValue( $oSender, $UIA_BoundingRectanglePropertyId ) & @CRLF & _

                      "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )

        ConsoleWrite("**** Desktop windows ****" & @CRLF)

        ConsoleWrite(_UIA_getAllPropertyValues($oSender) & @CRLF)

For the pop-up now I can get:

Title     = Shiny.Chat.ViewModels.PrivateChatSessionViewModel

Pos         = 454;311;400;550
Handle    = 00950CE8

This is interesting since I can calculate position of the close button.

Anyway Winclose(00950CE8) doesn't work.

How can I now get all the contents of the pop-up? (close button, name,...)

This could be interesting since I could ctrlclick it or at least having exact coords of close button to close (if I use simplespy.au3 clicking on the pop-up items I have all these infos but I'd like to have them in the program itself)

Thanks for you patience,

Marco

[marko001]

Well I got it now using the following:

findThemAll($oSender, $TreeScope_subtree)
...
 
Func findThemAll($oElementStart, $TreeScope)
;~  Get result with findall function alternative could be the treewalker
    Local $oCondition, $oAutomationElementArray,$oUIElement
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Pos: " & _UIA_getPropertyValue( $oUIElement, $UIA_BoundingRectanglePropertyId ) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next

EndFunc   ;==>findThemAll
 

 

In this way I have everything I need:

Title is: Close    Pos: 828;312;24;24    Class=Button

Which is the correct command to parse to close now the window without using the usual mouseclick("left",828+12,312-2,1,0)?

Thanks again,

M.

[LarsJ]

You can click the button with code like this:

Local $pInvokeButton, $oInvokeButton
$oUIButton.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )
$oInvokeButton = ObjCreateInterface( $pInvokeButton, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
$oInvokeButton.Invoke()

[marko001]

Before reading your message I tried with

        if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Close" Then
            ConsoleWrite("Closing now" & @CRLF)
            Sleep(5000)
            dim $t2
            ;_UIA_action($oUIElement,"leftclick")
            $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
            MouseClick("left",$t2[1],$t2[2],1,0)
        EndIf

_UIA_action($oUIElement,"leftclick") bring mouse to 0,0

and the loop you see doesn't work because he iterate the for 2 times, 1st one brings mouse to 0,0 and clicks (losing focus to popup) and 2nd time goes to correct location and clicks (but popup is now under another window)

GOing to check your solution now.

M.

[EDIT]

ERROR: $oUIButton: undeclared global variable.

        if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Close" Then
            ConsoleWrite("chiudo!!!" & @CRLF)
            Sleep(5000)
            Local $pInvokeButton, $oInvokeButton
            $oUIElement.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )
            $oInvokeButton = ObjCreateInterface( $pInvokeButton, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
            $oInvokeButton.Invoke()
;~             dim $t2
;~             $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
;~             MouseClick("left",$t2[1],$t2[2],1,0)
        EndIf

it doesn't click and bring main window at front.

Edited July 3, 2014 by marko001

[LarsJ]

$oUIButton is the $oUIElement which matches the title and class in post 229.

[junkew]

 

this works for me (its an OK button on my example and not CLOSE) no issue with x,y beeing 0,0

 

within the loop

if _UIA_getPropertyValue($UIA_oUIElement, $UIA_NamePropertyId)="OK" Then
                _UIA_Action($UIA_oUIElement,"leftclick")
            EndIf

uncomment this in UIWrappers.au3

 

;~  _UIA_DEBUG("Title is: <" &  _UIA_getPropertyValue($UIA_oUIElement,$UIA_NamePropertyId) &  ">" & $clickcount & ":" & $clickaction & ":" & $x & ":" & $y & ":" & @CRLF, $UIA_Log_Wrapper)

then you can see the details in log.txt (within the same folder if you did not make a UIA.CFG configuration file)

and this

 

$oUIButton.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )
$oInvokeButton = ObjCreateInterface( $pInvokeButton, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
$oInvokeButton.Invoke()

can also be written with the UIAWrappers.au3 to

_UIA_Action($UIA_oUIElement,"invoke") 

offcourse in the library it does the same as LarsJ has written out Edited July 3, 2014 by junkew

[marko001]

infact I changed it in $oUIElement.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )

Button is just the [X] close button upright near minimize,maximize. There's no other button and Title is "Close".

MsgBox(0,"Coords: ",_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),5)

works fine, I could just use that.

[marko001]

This way works fine:

    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Pos: " & _UIA_getPropertyValue( $oUIElement, $UIA_BoundingRectanglePropertyId ) & @TAB & "Center =" & _UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Close" Then
             MsgBox(0,"Coords: ",_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),2)
            ConsoleWrite("Closing Now" & @CRLF)
            Sleep(5000)
            dim $t2
            $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
            MouseClick("left",$t2[1],$t2[2],1,0)
            ExitLoop
            ;_UIA_action($oUIElement,"leftclick")
;~             Local $pInvokeButton, $oInvokeButton
;~             $oUIElement.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )
;~             $oInvokeButton = ObjCreateInterface( $pInvokeButton, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
;~             $oInvokeButton.Invoke()
;~             dim $t2
;~             $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
;~             MouseClick("left",$t2[1],$t2[2],1,0)
        EndIf

    Next

 

I see, pausing the code, that after closing the popup it reiterate the loop another time, look here:

Closing Now

Title     = Shiny.Chat.ViewModels.PrivateChatSessionViewModel
Class     = ToolWindow
Ctrl type = 50032
Ctrl name = window
Value     =
Pos       = 249;423;250;550
Handle    = 002B0472

Title is: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Centro =374;698    Class=ToolWindow
Title is: Minimize    Pos: 425;424;24;24    Center =437;436    Class=Button
Title is: Maximize    Pos: 449;424;24;24    Center =461;436    Class=Button
Title is: Close    Pos: 473;424;24;24    Center =485;436    Class=Button

AND HERE IT'S CLOSED

?? It loops again??

Closing Now
Title     =
Class     = ToolWindow
Ctrl type = 50032
Ctrl name = finestra
Value     =
Pos       = 249;423;250;550
Handle    = 002B0472

Title is:     Pos: 249;423;250;550    Center =374;698    Class=ToolWindow
Title is: Minimize    Pos: 0;0;0;0    Center =-2147483648;-2147483648    Class=Button
Title is: Maximize    Pos: 0;0;0;0    Center =-2147483648;-2147483648    Class=Button
Title is: Close    Pos: 0;0;0;0    Center =-2147483648;-2147483648    Class=Button
Closing Now
 

Any idea?

[marko001]

Got it. It just need some time to reset. If I put a sleep(5000) after the mouseclick() it works perfectly.

This is the final code:

Func findThemAll($oElementStart, $TreeScope)
    Local $oCondition, $oAutomationElementArray,$oUIElement
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength
    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)
    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Close" Then
            ConsoleWrite("Closing Now" & @CRLF)
            Sleep(500)
            dim $t2
            $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
            MouseClick("left",$t2[1],$t2[2],1,0)
            Sleep(5000)
            ExitLoop
        EndIf
    Next
EndFunc   ;==>findThemAll

If you want to try to explain me where and how to use:

Local $pInvokeButton, $oInvokeButton
$oUIButton.GetCurrentPattern( $UIA_InvokePatternId, $pInvokeButton )
$oInvokeButton = ObjCreateInterface( $pInvokeButton, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
$oInvokeButton.Invoke()

would be really appreciated, in any case I can use it now as shown. Really thanks anyway for the great help.

And the last question:

which is the best way to include in the program these functions? trough an

AdlibRegister("_checkpopups", 5000) ;

or it's enough to run the function before the while cycle?

Marco

Edited July 3, 2014 by marko001

[junkew]

There is an issue on your code as you seem to be falling thru to the 2nd window closing and as such

_UIA_Action($UIA_oUIElement,"leftclick")

is not working

 

You should start reading here

http://msdn.microsoft.com/en-us/library/windows/desktop/ee684009(v=vs.85).aspx

UIAutomation framework is huge and I only abstracted so far a small subset

Controls you can detect in multiple ways in order of complexity but the more complex the more you can do

1. based on mouseposition

2. by iterating thru a tree with a treewalker

3. act on events that happen like opening/closing of windows or changing properties

You do not have to use adlibregister as event functions are automatically called.

In your example you can see that actually 2 windows are closing the 2nd doesn't have a name. You should have an if statement to get the correct one.

When you have an IUIElement you can do depending on the class different things if defined. Each thing you can do is hidden in a pattern and 1 control can have n patterns with m methods/functions

LarsJ gave you the details on getting a pattern

Here you have all the patterns

;module UIA_PatternIds
Global Const $UIA_InvokePatternId=10000
Global Const $UIA_SelectionPatternId=10001
Global Const $UIA_ValuePatternId=10002
Global Const $UIA_RangeValuePatternId=10003
Global Const $UIA_ScrollPatternId=10004
Global Const $UIA_ExpandCollapsePatternId=10005
Global Const $UIA_GridPatternId=10006
Global Const $UIA_GridItemPatternId=10007
Global Const $UIA_MultipleViewPatternId=10008
Global Const $UIA_WindowPatternId=10009
Global Const $UIA_SelectionItemPatternId=10010
Global Const $UIA_DockPatternId=10011
Global Const $UIA_TablePatternId=10012
Global Const $UIA_TableItemPatternId=10013
Global Const $UIA_TextPatternId=10014
Global Const $UIA_TogglePatternId=10015
Global Const $UIA_TransformPatternId=10016
Global Const $UIA_ScrollItemPatternId=10017
Global Const $UIA_LegacyIAccessiblePatternId=10018
Global Const $UIA_ItemContainerPatternId=10019
Global Const $UIA_VirtualizedItemPatternId=10020
Global Const $UIA_SynchronizedInputPatternId=10021

so if you have an element

you can first check the patternproperties like

$UIA_IsWindowPatternAvailablePropertyId

 

if its available you can thru this pattern call methods like

 

Global Const $sIID_IUIAutomationWindowPattern="{0FAEF453-9208-43EF-BBB2-3B485177864F}"
Global $dtagIUIAutomationWindowPattern = "Close hresult();" & _
"WaitForInputIdle hresult(int;long*);" & _
"SetWindowVisualState hresult(long);" & _
"CurrentCanMaximize hresult(long*);" & _
"CurrentCanMinimize hresult(long*);" & _
"CurrentIsModal hresult(long*);" & _
"CurrentIsTopmost hresult(long*);" & _
"CurrentWindowVisualState hresult(long*);" & _
"CurrentWindowInteractionState hresult(long*);" & _
"CachedCanMaximize hresult(long*);" & _
"CachedCanMinimize hresult(long*);" & _
"CachedIsModal hresult(long*);" & _
"CachedIsTopmost hresult(long*);" & _
"CachedWindowVisualState hresult(long*);" & _
"CachedWindowInteractionState hresult(long*);"

getting a pattern is abstracted in 

func _UIA_getPattern($obj,$patternID)

used like this see example 9 for mediaplayer

$oValueP=_UIA_getpattern($oVolumeButton,$UIA_ValuePatternId)
if isobj($oValueP) Then
dim $volume
    $oValuep.currentvalue($volume)
    _UIA_Debug("Volume is " & $volume & @CRLF)
EndIf

[hnakamur]

 

Hi, junkew

 

Thanks for your great scripts and examples! 

I would like to ask you under what license can I use these scripts?

 

Thanks.

 

 

I found the line in UIAWrappers.au3

; License .......: GPL

[junkew]

http://www.autoitscript.com/site/donate/

Just help a great tool like autoit by referencing it. The IUIAutomation and wrappers are just as it says wrappers around UIAUTOMATIONCORE.DLL which is shipped as part of the windows operating system.

you can hire me for EUR 300,-- per hour for onsite dedicated support

[marko001]

I solved most parts. It remains unknown why it calls function 2 times once pop-up is visible:

Func oIUIAutomationEventHandler_HandleAutomationEvent($pSelf, $pSender, $iEventId) ; Ret: long  Par: ptr;int
    Dim $t
    If $iEventId <> $UIA_Window_WindowClosedEventId Then
        Local $oSender = ObjCreateInterface($pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        $oSender.AddRef()
        If _UIA_getPropertyValue($oSender, $UIA_ControlTypePropertyId) = "50032" Then ;50032 popup
            ConsoleWrite("PopUp Visible - Windows Name: " & _UIA_getPropertyValue($oSender, $UIA_NamePropertyId) & @CRLF)
            findThemAll($oSender, $TreeScope_Subtree)
            ; sleep(5000) ; used to avoid 2nd loop
        EndIf
    EndIf
    Return $S_OK
EndFunc   ;==>oIUIAutomationEventHandler_HandleAutomationEvent
 
[...]
 
Func findThemAll($oElementStart, $TreeScope)
    Local $oCondition, $oAutomationElementArray, $oUIElement
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength
    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)
    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Pos: " & _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId) & @TAB & "Center =" & _UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next
EndFunc   ;==>findThemAll
 

 

I tried to put a sleep(5000) but it doesn't work.

I can I stop this reiteration but not the function itself? (I need to be informed about pop-ups but just one time not twice).

Thanks,

M.