Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


junkew
 Share

Recommended Posts

  • Moderators

D4RKON3,

Thanks. :)

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

Greetingz All

First off Thanks to everyone for their contribution.

@ junkew You are brilliant & thanks for sharing your code, knowledge, & time.

I have been working with your FF & IE examples. When I run the code I get a error in the UIAWrapper.

"C:\Program Files (x86)\AutoIt3\Include\UIAWrappers.au3" (363) : ==> Variable must be of type "Object".:
$obj.GetCurrentPropertyValue($Id,$tVal)
$obj^ ERROR

Well I am new to AutoIt & though I have tried to figure this out I don't know how to fix this. Could someone please point me in the right direction.

Thanks & all the best,

d ;)

Link to comment
Share on other sites

In an update I will put this function as a fix

replace the function in UIA_Wrappers with below and script will at least continue (but still not do what you want as the object itself is not found)

func _UIA_getPropertyValue($obj, $id)
    local $tval
    local $tStr

    if not isobj($obj) Then
        seterror(1,0,0)
        return "** NO PROPERTYVALUE DUE TO NONEXISTING OBJECT"
    EndIf
    
    $obj.GetCurrentPropertyValue($Id,$tVal)
    $tStr=""
    if isarray($tVal) Then
        for $i=0 to ubound($tval)-1
            $tStr=$tStr & $tVal[$i]
            if $i <> ubound($tVal)-1 Then
                $tStr=$tStr & ";"
            endif
        Next
        return $tStr
    endIf
    return $tVal
EndFunc
Link to comment
Share on other sites

Okay thanks.

In my quest to use the JAB I came across this post. The answer seems logical from my limited programing knowledge. I am not sure if you are doing anything like this in your code. Or know/don't know, I just wanted to put it out there in my effort to further help the development of the project.

Thanks again. Now I am off to try & put this wonderful code you wrote to work.

All the best,

dirty :P

Link to comment
Share on other sites

Strange thing: When I run the script with admin rights, I get error "The requested action with this object has failed". But when I run it in normal mode- everything works fine. What can be the reason for that? Thanx and best regards, and Happy New Year!!!

Link to comment
Share on other sites

Hi!

1. Possible bug: I found that in some cases ^a - simply won't work (for example I have multi language keyboard layout- and I was having something like a

avarvarvar instead of varvarvar, so I made a small change

case "setvalue using keys"
            $obj.setfocus()
            ;send("^a")
          send($p1)
          sleep($UIA_DefaultWaitTime)

Now works ok -perhaps will be usefull for anyone-  but again it can be different on different machines of course

2.  Is it possible to use Controlsend() function instead of send () -  so that I could work with application in background

case "setvalue using keys"
            $obj.setfocus()
            ;send("^a")
           Controlsend($p1)
            sleep($UIA_DefaultWaitTime)

If the answer "yes" how can I find controlID?

ControlSend ( "title", "text", controlID, "string" [, flag = 0] )

Thanx in advance!!!!!!

Link to comment
Share on other sites

not sure what in multi language the shortcut is for select all (Ctrl+A keycombination)

you could try to replace with 

a. send("^(a)")

b. send("^{a}")

c. make a small testapplication against notepad and see how it behaves with setvalue actions

try to use the action setvalue instead of setvalue using keys. Setvalue using keys is already a workaround for those textboxes that do not deal well with setvalue action itself
 

controlsend is the native AutoIT way of doing it so check within the helpfile of autoit the controlsend syntax

in general the simple spy automationid will be equal to the ID property of the AU3INF.EXE result(s) but not all controls are identifiable by a controlid.

Use the AU3INF.exe files to see your result(s)

This could be an alternate viewer for controls '?do=embed' frameborder='0' data-embedContent>>

Link to comment
Share on other sites

Thank you very much for your help and support!!!! 

if IsObj($IE) Then
   $sText="search"; indexrelative:=1"    ;Is equal to
   $oButton=_UIA_getFirstObjectOfElement($IE,"name:=" & $sText, $treescope_subtree)
EndIf

Is there any method to calculate the x,y position of found element on the window or on the screen?

 

Thanx in advance!!!

Link to comment
Share on other sites

I have made some tests with event handler interfaces. For a start just to verify whether I could get some events at all.

I succeeded to get events for IUIAutomationEventHandler, IUIAutomationFocusChangedEventHandler and UIAutomationStructureChangedEventHandler but not for IUIAutomationPropertyChangedEventHandler.

Update 2014-01-13

Added the proper function to create custom interface methods to all 4 examples. See post #115. This means that all examples are working including the example with IUIAutomationPropertyChangedEventHandler.

IUIAutomationEventHandler

Run this code in Scite and you'll see window and menu open/close events in the console when you open/close a program e.g. Calculator and when you open/close a menu.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationEventHandler, $oIUIAutomationEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationEventHandler = ObjectFromTag( "oIUIAutomationEventHandler_", $dtagIUIAutomationEventHandler, $tIUIAutomationEventHandler, True )
  If Not IsObj( $oIUIAutomationEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  ; Window open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  ; Menu open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationEventHandler_HandleAutomationEvent( $pSelf, $pSender, $iEventId ) ; Ret: long  Par: ptr;int
  ConsoleWrite( "oIUIAutomationEventHandler_HandleAutomationEvent: " & $iEventId & @CRLF )
  If $iEventId <> $UIA_Window_WindowClosedEventId Then
    Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    $oSender.AddRef()
    ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                  "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                  "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                  "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                  "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                  "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  EndIf
  Return $S_OK
EndFunc

Func oIUIAutomationEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationEventHandler Then
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUIAutomationEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationFocusChangedEventHandler

If you test with Calculator you'll get events when you click a number button and the focus changes between the button and the edit box.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationFocusChangedEventHandler, $oIUIAutomationFocusChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationFocusChangedEventHandler = ObjectFromTag( "oIUIAutomationFocusChangedEventHandler_", $dtagIUIAutomationFocusChangedEventHandler, $tIUIAutomationFocusChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationFocusChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  If $oUIAutomation.AddFocusChangedEventHandler( 0, $oIUIAutomationFocusChangedEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationFocusChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationFocusChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent( $pSelf, $pSender ) ; Ret: long  Par: ptr
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent: " & $pSender & @CRLF )
  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef()
  ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  Return $S_OK
EndFunc

Func oIUIAutomationFocusChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationFocusChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: IUIAutomationFocusChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationFocusChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationStructureChangedEventHandler

Switch between Standard and Scientific view in Calculator or switch folder in Windows Explorer and you'll get lots of events.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationStructureChangedEventHandler, $oIUIAutomationStructureChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationStructureChangedEventHandler = ObjectFromTag( "oIUIAutomationStructureChangedEventHandler_", $dtagIUIAutomationStructureChangedEventHandler, $tIUIAutomationStructureChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationStructureChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  If $oUIAutomation.AddStructureChangedEventHandler( $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationStructureChangedEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationStructureChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationStructureChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationStructureChangedEventHandler_HandleStructureChangedEvent( $pSelf, $pSender, $iChangeType, $pRuntimeId ) ; Ret: long  Par: ptr;long;ptr
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_HandleStructureChangedEvent: " & $pSender & " " & $iChangeType & @CRLF )
  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef()
  ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  Return $S_OK
EndFunc

Func oIUIAutomationStructureChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationStructureChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: IUIAutomationStructureChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationStructureChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationPropertyChangedEventHandler

I have tested the code below with Internet Explorer and I have expected to get $UIA_ValueValuePropertyId change events when I switch between different URLs. But I get no events. I have tried with several other programs but I simply get no events.

Is there anyone who has managed to record these events? In that case, I'd like to see the code.

To test the code you must change the description for AddPropertyChangedEventHandlerNativeArray in CUIAutomation2.au3 to this:

"AddPropertyChangedEventHandlerNativeArray hresult(ptr;long;ptr;ptr;struct*;int);"
#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationPropertyChangedEventHandler, $oIUIAutomationPropertyChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationPropertyChangedEventHandler = ObjectFromTag( "oIUIAutomationPropertyChangedEventHandler_", $dtagIUIAutomationPropertyChangedEventHandler, $tIUIAutomationPropertyChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationPropertyChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  Local $tPropertyArray = DllStructCreate( "int[2]" )
  DllStructSetData( $tPropertyArray, 1, $UIA_ValueValuePropertyId, 1 )
  DllStructSetData( $tPropertyArray, 1, $UIA_ToggleToggleStatePropertyId, 2 )
  $oUIAutomation.AddPropertyChangedEventHandlerNativeArray( $pUIElement, $TreeScope_Descendants, 0, $oIUIAutomationPropertyChangedEventHandler, $tPropertyArray, 2 )

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(100)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationPropertyChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationPropertyChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $vVal
  $obj.GetCurrentPropertyValue( $id, $vVal )
  Return $vVal
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent( $pSelf, $pSender, $iPropertyId, $newValue ) ; Ret: long  Par: ptr;int;variant
  ConsoleWrite( @CRLF & "oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = " & $iPropertyId & ", $newValue = " & $newValue & @CRLF )

  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef(  )

  ConsoleWrite( "Handle    " & _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) & @CRLF )
  ConsoleWrite( "Name      " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF )
  ConsoleWrite( "Class     " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF )
  ConsoleWrite( "Ctrl type " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF )
  ConsoleWrite( "Ctrl name " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF )
  ConsoleWrite( "Value     " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF )

  Return $S_OK
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationPropertyChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

Update 2014-01-13

Added the proper function to create custom interface methods to all 4 examples. See post #115. This means that all examples are working including the example with IUIAutomationPropertyChangedEventHandler.

Edited by LarsJ
Link to comment
Share on other sites

I didn't succeed to get events for the IUIAutomationPropertyChangedEventHandler in the post above. As a quick test I have put the code in a DLL. And now I get events (printed in a text file).

This is the code in the DLL:

#define WIN32_LEAN_AND_MEAN

#include <stdio.h>
#include <windows.h>
#include <oleauto.h>
#include <UIAutomation.h>

class CEventHandler : public IUIAutomationPropertyChangedEventHandler
{
  public: 

    CEventHandler();
    ~CEventHandler();

    HRESULT RegisterEventHandler();

    IFACEMETHODIMP QueryInterface(__in REFIID riid, __out PVOID *ppvObj);
    IFACEMETHODIMP_(ULONG) AddRef();
    IFACEMETHODIMP_(ULONG) Release();

    IFACEMETHODIMP HandlePropertyChangedEvent(IUIAutomationElement *sender, PROPERTYID propertyId, VARIANT newValue);

  private:

    void  UIAPrintProperty( IUIAutomationElement *pUIAutomationElement, PROPERTYID propertyId, char *text );
    ULONG _cRef;
    FILE  *f;
};

CEventHandler  *pEventHandler = NULL;


extern "C" __declspec(dllexport) int __cdecl TstFunc()
{
  pEventHandler = new CEventHandler();
  if ( pEventHandler == NULL ) return 1;

  HRESULT hr = pEventHandler->RegisterEventHandler();
  if ( SUCCEEDED(hr) < 0 ) return 1;

  return 0;
}


CEventHandler::CEventHandler() : _cRef(1)
{
  CoInitialize( NULL );
  f = fopen( "UIAEventHandler.txt", "w" );
}

CEventHandler::~CEventHandler()
{
  fclose(f);
  CoUninitialize();
}

HRESULT CEventHandler::RegisterEventHandler()
{
  HRESULT              hr = E_FAIL;
  IUIAutomation        *pUIAutomation;
  IUIAutomationElement *pUIAutomationElement;
  PROPERTYID           properties[] = {UIA_ValueValuePropertyId,
                                       UIA_ToggleToggleStatePropertyId};

  hr = CoCreateInstance(CLSID_CUIAutomation, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pUIAutomation));
  if (SUCCEEDED(hr))
  {
    hr = pUIAutomation->GetRootElement( &pUIAutomationElement );
    if (SUCCEEDED(hr))
    {
      hr = pUIAutomation->AddPropertyChangedEventHandlerNativeArray( pUIAutomationElement, TreeScope_Descendants, NULL, this, properties, sizeof(properties)/sizeof(properties[0]) );
      if (SUCCEEDED(hr)) {
        fprintf( f, "AddPropertyChangedEventHandlerNativeArray\n" );
      }
    }
  }

  return hr;
}

IFACEMETHODIMP CEventHandler::HandlePropertyChangedEvent(IUIAutomationElement *sender, PROPERTYID propertyId, VARIANT newValue)
{
  fprintf( f, "\nHandlePropertyChangedEvent\n" );
  UIAPrintProperty( sender, UIA_NativeWindowHandlePropertyId,     "Handle   " );
  UIAPrintProperty( sender, UIA_NamePropertyId,                   "Name     " );
  UIAPrintProperty( sender, UIA_ClassNamePropertyId,              "Class    " );
  UIAPrintProperty( sender, UIA_ControlTypePropertyId,            "Ctrl type" );
  UIAPrintProperty( sender, UIA_LocalizedControlTypePropertyId,   "Ctrl name" );
  UIAPrintProperty( sender, UIA_LegacyIAccessibleValuePropertyId, "Value    " );
  return S_OK;
}

IFACEMETHODIMP CEventHandler::QueryInterface(__in REFIID riid, __out PVOID *ppvObj)
{
  if (ppvObj == NULL)
  {
    return E_POINTER;
  }

  *ppvObj = NULL;

  HRESULT hr = E_NOINTERFACE;

  if ( riid == __uuidof(IUnknown) )
  {
    fprintf( f, "QueryInterface: IUnknown\n" );
    *ppvObj = static_cast<IUIAutomationPropertyChangedEventHandler*>(this);
    AddRef();
    hr = S_OK;
  }
  else if ( riid == __uuidof(IUIAutomationPropertyChangedEventHandler) )
  {
    fprintf( f, "QueryInterface: IUIAutomationPropertyChangedEventHandler\n" );
    *ppvObj = static_cast<IUIAutomationPropertyChangedEventHandler*>(this);
    AddRef();
    hr = S_OK;
  }
  else
  {
    LPOLESTR lpolestr;
    StringFromCLSID( riid, &lpolestr );
    fwprintf( f, L"QueryInterface: %s\n", lpolestr );
    CoTaskMemFree( lpolestr );
  }

  return hr;
}

IFACEMETHODIMP_(ULONG) CEventHandler::AddRef()
{
  return InterlockedIncrement(&_cRef);
}

IFACEMETHODIMP_(ULONG) CEventHandler::Release()
{
  ULONG cRef = InterlockedDecrement(&_cRef);
  if (cRef == 0)
  {
    delete this;
  }

  return cRef;
}

void CEventHandler::UIAPrintProperty( IUIAutomationElement *pUIAutomationElement, PROPERTYID propertyId, char *text )
{
  VARIANT v;
  pUIAutomationElement->GetCurrentPropertyValue( propertyId, &v );
  if ( v.vt == VT_I4 )
    if ( strcmp( text, "Handle   " ) == 0 )
      fprintf( f, "%s: 0x%p\n", text, v.lVal);
    else
      fprintf( f, "%s: %d\n", text, v.lVal);
  else if ( v.vt == VT_BSTR ) {
    fprintf( f, "%s: ", text );
    fwprintf( f, L"%s\n", v.bstrVal ); }
}

The AutoIt GUI:

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>

Opt( "MustDeclareVars", 1 )

MainFunc()

Func MainFunc()
  Local $dllTest, $aRet
  If @AutoItX64 Then
    $dllTest = DllOpen( "UIAEventHandler-x64.dll" )
  Else
    $dllTest = DllOpen( "UIAEventHandler.dll" )
  EndIf
  GUICreate( "UIAEventHandler", 200, 100, 200, 100 )
  GUISetState( @SW_SHOW )
  $aRet = DllCall( $dllTest, "int:cdecl", "TstFunc" )
  ;ConsoleWrite( "@error = " & @error & @CRLF )
  ;ConsoleWrite( "TstFunc = " & $aRet[0] & @CRLF )
  While GUIGetMsg() <> $GUI_EVENT_CLOSE
  WEnd
  GUIDelete()
  DllClose( $dllTest )
  Exit
EndFunc

And this is the events when I open Internet Explorer, change the URL to AutoIt home, open a text file with Wordpad and types "Test":

QueryInterface: {0000001B-0000-0000-C000-000000000046}
QueryInterface: {00000003-0000-0000-C000-000000000046}
AddPropertyChangedEventHandlerNativeArray
QueryInterface: {0000001B-0000-0000-C000-000000000046}
QueryInterface: IUnknown
QueryInterface: {00000018-0000-0000-C000-000000000046}
QueryInterface: {00000019-0000-0000-C000-000000000046}
QueryInterface: {4C1E39E1-E3E3-4296-AA86-EC938D896E92}
QueryInterface: IUIAutomationPropertyChangedEventHandler
QueryInterface: {1C733A30-2A1C-11CE-ADE5-00AA0044773D}
QueryInterface: IUIAutomationPropertyChangedEventHandler

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : about:blank

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : about:blank

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/site/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/site/

HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : T 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Te 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Tes 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Test

DLL files:

DLLfiles.7z

Link to comment
Share on other sites

@topten:

if IsObj($IE) Then

   $sText="search"; indexrelative:=1"    ;Is equal to

   $oButton=_UIA_getFirstObjectOfElement($IE,"name:=" & $sText, $treescope_subtree)

EndIf

above is functionally equal but technically differently implemented

_UIA_getFirstObjectOfElement will use the MS UIA library straightforward to find on a single property the first object and as such syntax with indexrelative will not work

The UIA_Wrappers add some functionality to walk thru all elements and add enhancements like

a. indexrelative, index, instance properties

b. match on regular expression

c. match easier on multiple properties combined (opposed to the complex logica of and/or conditions the MS UIA has)

 

Is there any method to calculate the x,y position of found element on the window or on the screen?

_UIA_getPropertyValue($oElement, $UIA_BoundingRectanglePropertyId)

 

 

@LarsJ: Regarding eventhandling I will post another example (without dll)

Edited by junkew
Link to comment
Share on other sites

LarsJ, I have rewritten that event handling function (several times LOL), so your non-working example should look more like this:

#include "CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationPropertyChangedEventHandler, $oIUIAutomationPropertyChangedEventHandler

Global $oUIAutomation

MainFunc()


Func MainFunc()

    $oIUIAutomationPropertyChangedEventHandler = ObjectFromTag("oIUIAutomationPropertyChangedEventHandler_", $dtagIUIAutomationPropertyChangedEventHandler, $tIUIAutomationPropertyChangedEventHandler)
    If Not IsObj($oIUIAutomationPropertyChangedEventHandler) Then Return

    $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return

    Local $pUIElement
    $oUIAutomation.GetRootElement($pUIElement) ; Desktop
    If Not $pUIElement Then Return

    Local $tPropertyArray = DllStructCreate("int[2]")
    DllStructSetData($tPropertyArray, 1, $UIA_ValueValuePropertyId, 1)
    DllStructSetData($tPropertyArray, 1, $UIA_ToggleToggleStatePropertyId, 2)
    Local $x = $oUIAutomation.AddPropertyChangedEventHandlerNativeArray($pUIElement, $TreeScope_Descendants, 0, $oIUIAutomationPropertyChangedEventHandler, $tPropertyArray, 2)


    HotKeySet("{ESC}", "Quit")

    While Sleep(100)
    WEnd

EndFunc   ;==>MainFunc

Func Quit()
    Exit
EndFunc   ;==>Quit



Func _UIA_getPropertyValue($obj, $id)
    Local $vVal
    $obj.GetCurrentPropertyValue($id, $vVal)
    Return $vVal
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent($pSelf, $pSender, $iPropertyId, $newValue) ; Ret: long  Par: ptr;int;variant
    ConsoleWrite(@CRLF & "oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = " & $iPropertyId & ", $newValue = " & $newValue & @CRLF)

    Local $oSender = ObjCreateInterface($pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    $oSender.AddRef()

    ConsoleWrite("Handle    " & _UIA_getPropertyValue($oSender, $UIA_NativeWindowHandlePropertyId) & @CRLF)
    ConsoleWrite("Name      " & _UIA_getPropertyValue($oSender, $UIA_NamePropertyId) & @CRLF)
    ConsoleWrite("Class     " & _UIA_getPropertyValue($oSender, $UIA_ClassNamePropertyId) & @CRLF)
    ConsoleWrite("Ctrl type " & _UIA_getPropertyValue($oSender, $UIA_ControlTypePropertyId) & @CRLF)
    ConsoleWrite("Ctrl name " & _UIA_getPropertyValue($oSender, $UIA_LocalizedControlTypePropertyId) & @CRLF)
    ConsoleWrite("Value     " & _UIA_getPropertyValue($oSender, $UIA_LegacyIAccessibleValuePropertyId) & @CRLF)

    Return $S_OK
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_QueryInterface($pSelf, $pRIID, $pObj) ; Ret: long  Par: ptr;ptr*
    Local $sIID = StringFromGUID($pRIID)
    If $sIID = $sIID_IUnknown Then
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown" & @CRLF)
        DllStructSetData(DllStructCreate("ptr", $pObj), 1, $pSelf)
        Return $S_OK
    ElseIf $sIID = $sIID_IUIAutomationPropertyChangedEventHandler Then
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler" & @CRLF)
        DllStructSetData(DllStructCreate("ptr", $pObj), 1, $pSelf)
        Return $S_OK
    Else
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: " & $sIID & @CRLF)
        Return $E_NOINTERFACE
    EndIf
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_AddRef($pSelf) ; Ret: ulong
    ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_AddRef" & @CRLF)
    Return 1
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_Release($pSelf) ; Ret: ulong
    ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_Release" & @CRLF)
    Return 1
EndFunc



Func StringFromGUID($pGUID)
    Local $aResult = DllCall("ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40)
    If @error Then Return SetError(@error, @extended, "")
    Return SetExtended($aResult[0], $aResult[2])
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc
...That's almost exact replica of c++ dll code you posted. Edited by trancexx
Link to comment
Share on other sites

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\Elwin\Documents\UIA\ex12_propertyevent.au3" /UserParams    
+>19:12:27 Starting AutoIt3Wrapper v.2.1.0.33    Environment(Language:0413  Keyboard:00020409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64)
>Running AU3Check (3.3.10.2)  from:C:\Program Files (x86)\AutoIt3
+>19:12:27 AU3Check ended.rc:0
>Running:(3.3.10.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\Elwin\Documents\UIA\ex12_propertyevent.au3"    
--> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
0
0
0
0
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown
oIUIAutomationPropertyChangedEventHandler_Release
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {0000001B-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000003-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {0000001B-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000018-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000019-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {4C1E39E1-E3E3-4296-AA86-EC938D896E92}
oIUIAutomationPropertyChangedEventHandler_Release
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {1C733A30-2A1C-11CE-ADE5-00AA0044773D}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler

oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = 30045, $newValue = 0x001D0000
Handle    1901574
Name      Adres en zoeken met SecureSearch
Class     Edit
Ctrl type 50004
Ctrl name bewerken
Value     http://www.google.nl/ 

On the example of trancexx I only get one time output (and all I can do is press escape and program exits) 

steps i did

1. start example in Scite

2. open Internet Explorer and see above output coming

3. clicking around, closing IE and reopening gives no additonal output (not expected)

4. press escape and program exits (as expected)

 

Link to comment
Share on other sites

So it works for you correctly now?

I also left consolewrite line inside ObjectFromTag() in case there are more types that aren't converted from COM types to callback type correctly because I'm not sure if all the types inside your UDF are covered. It should always print zeroes.

You had some invalid types last time I checked the UDF, but it was really long time ago.

Edited by trancexx
Link to comment
Share on other sites

yes it works now correctly.

It still could be there are invalid types in the UDF but I think most are fixed (only complaints came from LarsJ since november)

  • none definitions changed to correct type
  • LarsJ identified this to be fixed in CUIAutomation2.au3 (lines 532 - 537 in the current version):
    "TreeScope hresult(long*);" & _
    "TreeScope hresult(long);" & _
    "TreeFilter hresult(ptr*);" & _
    "TreeFilter hresult(ptr);" & _
    "AutomationElementMode hresult(long*);" & _
    "AutomationElementMode hresult(long);"

    with these lines:
    "get_TreeScope hresult(long*);" & _
    "put_TreeScope hresult(long);" & _
    "get_TreeFilter hresult(ptr*);" & _
    "put_TreeFilter hresult(ptr);" & _
    "get_AutomationElementMode hresult(long*);" & _
    "put_AutomationElementMode hresult(long);"
     
  • and LarsJ gave now back some issue on SAFEARRAY currently its in the UDF probably an int and should be a struct*
    AddPropertyChangedEventHandlerNativeArray  is an example of where safearray was defined incorrectly
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...