IUIAutomation MS framework automate chrome, FF, IE, ....

junkew · February 9, 2014

@Isakizada: I am more looking in practical terms what the benefits are from either solution

With IAccessbile I never did get AccessibleObjectFromPoint to be working

'?do=embed' frameborder='0' data-embedContent>>

So far on applications I worked IUIautomation was available and I prefer that over the IAccessible

But as IAccessible exist(s) longer probably more applications support it and some new ones have IAccessible2 interface (so far not seen an AutoIT interface definition for that)

LarsJ · February 11, 2014

Microsoft Active Accessibility (MSAA). AccessibleObjectFromPoint.

To run this example you must replace the entire $dtagIAccessible description string in CUIAutomation2.au3 with this description (this is the same description as in post #136):

Global $dtagIAccessible = "GetTypeInfoCount hresult(uint*);" & _ ; IDispatch
"GetTypeInfo hresult(uint;int;ptr*);" & _
"GetIDsOfNames hresult(struct*;wstr;uint;int;int);" & _
"Invoke hresult(int;struct*;int;word;ptr*;ptr*;ptr*;uint*);" & _
"get_accParent hresult(ptr*);" & _                               ; IAccessible
"get_accChildCount hresult(long*);" & _
"get_accChild hresult(variant;idispatch*);" & _
"get_accName hresult(variant;bstr*);" & _
"get_accValue hresult(variant;bstr*);" & _
"get_accDescription hresult(variant;bstr*);" & _
"get_accRole hresult(variant;variant*);" & _
"get_accState hresult(variant;variant*);" & _
"get_accHelp hresult(variant;bstr*);" & _
"get_accHelpTopic hresult(bstr*;variant;long*);" & _
"get_accKeyboardShortcut hresult(variant;bstr*);" & _
"get_accFocus hresult(struct*);" & _
"get_accSelection hresult(variant*);" & _
"get_accDefaultAction hresult(variant;bstr*);" & _
"accSelect hresult(long;variant);" & _
"accLocation hresult(long*;long*;long*;long*;variant);" & _
"accNavigate hresult(long;variant;variant*);" & _
"accHitTest hresult(long;long;variant*);" & _
"accDoDefaultAction hresult(variant);" & _
"put_accName hresult(variant;bstr);" & _
"put_accValue hresult(variant;bstr);"

AccessibleObjectFromPoint. Don't copy this code. Download the UDF.

Func AccessibleObjectFromPoint( $x, $y, ByRef $pAccessible, ByRef $tVarChild )
  Local $tPOINT = DllStructCreate( "long;long" )
  DllStructSetData( $tPOINT, 1, $x )
  DllStructSetData( $tPOINT, 2, $y )
  Local $tPOINT64 = DllStructCreate( "int64", DllStructGetPtr( $tPOINT ) )
  Local $tVARIANT = DllStructCreate( $tagVARIANT )
  VariantInit( $tVARIANT )
  Local $aRet = DllCall( $hdllOleacc, "int", "AccessibleObjectFromPoint", "int64", DllStructGetData( $tPOINT64, 1 ), "ptr*", 0, "struct*", $tVARIANT )
  If @error Then Return SetError(1, 0, $S_FALSE)
  If $aRet[0] Then Return SetError(2, 0, $aRet[0])
  $pAccessible = $aRet[2]
  $tVarChild = $aRet[3]
  Return $S_OK
EndFunc

MSAccessibility.au3

Run this example in Scite. Place the mouse cursor over a window/control and press Ctrl+w. Info in console.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )

MainFunc()


Func MainFunc()

  HotKeySet( "^w", "GetElementInfo" )
  HotKeySet( "{ESC}", "Close" )

  While Sleep(100)
  Wend

EndFunc


Func GetElementInfo()
  Local $aMousePos = MouseGetPos(), $pAccessible, $tVarChild
  If AccessibleObjectFromPoint( $aMousePos[0], $aMousePos[1], $pAccessible, $tVarChild ) = $S_OK Then
    Local $vt = BitAND( DllStructGetData( $tVarChild, 1, 1 ), 0xFFFF )
    Local $iVal = DllStructGetData( $tVarChild, 1, 3 )
    ConsoleWrite( @CRLF )
    ConsoleWrite( "$pAccessible      = " & $pAccessible & @CRLF )
    ConsoleWrite( "$tVarChild: $vt   = " & $vt & @CRLF )
    ConsoleWrite( "$tVarChild: $iVal = " & $iVal & @CRLF )
    If $vt = $VT_I4 Then
      Local $oElement = ObjCreateInterface( $pAccessible, $sIID_IAccessible, $dtagIAccessible )
      If IsObj( $oElement ) Then
        If $iVal = $CHILDID_SELF Then
          ConsoleWrite( "Info for this element:" & @CRLF )
        Else
          ConsoleWrite( "Info for parent element:" & @CRLF )
        EndIf
        PrintElementInfo( $oElement, $CHILDID_SELF, "" )
      Else
        ConsoleWrite( "ObjCreateInterface failed" & @CRLF )
      EndIf
    Else
      ConsoleWrite( "Invalid value for $vt: " & $vt & @CRLF )
    EndIf
  Else
    ConsoleWrite( @CRLF )
    ConsoleWrite( "AccessibleObjectFromPoint failed" & @CRLF )
  EndIf
EndFunc


Func Close()
  Exit
EndFunc

Testet on XP 32 bit and Win 7 64 bit. Should work on Vista and Win 8 too.

junkew, I just quickly skimmed through the thread above.

nitekram, No you can't. This error in line 657 is caused by an invalid $obj-parameter when you call _UIA_getObjectByFindAll. Before you call this function you should check if it's a proper object.

lsakizada, Note that the documentation in your link deals with the managed code in the .NET framework. This is not what junkew is implementing. junkew is implementing the functions in the Windows API. You find the proper link here.

junkew · February 11, 2014

@LarsJ: Whats the logic of adding the idispatch functions? (I understand why you rename the dispids names and some parameter types)

accparent should return an idispatch object and that one should then use

PropertyGet
accParent

--------------------------------------------------------------------------------


object.accParent()
Attributes
FUNCFLAG_FHIDDEN
The function should not be displayed to the user.Applies To
object
Instance of IAccessibleParameters
None
Return Value
accParent returns
VT_DISPATCH (object)

but if the vtable just "includes the idispatch methods" then probably I should follow in the CUIAutomation2 module logic like this (although this can have conflicts if people import AutoITobject package)

;===============================================================================
#interface "IUnknown"
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"
; Definition
Global $dtagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
        "AddRef dword();" & _
        "Release dword();"
; List
Global $ltagIUnknown = "QueryInterface;" & _
        "AddRef;" & _
        "Release;"
;===============================================================================
;===============================================================================
#interface "IDispatch"
Global Const $sIID_IDispatch = "{00020400-0000-0000-C000-000000000046}"
; Definition
Global $dtagIDispatch = $dtagIUnknown & _
        "GetTypeInfoCount hresult(dword*);" & _
        "GetTypeInfo hresult(dword;dword;ptr*);" & _
        "GetIDsOfNames hresult(ptr;ptr;dword;dword;ptr);" & _
        "Invoke hresult(dword;ptr;dword;word;ptr;ptr;ptr;ptr);"
; List
Global $ltagIDispatch = $ltagIUnknown & _
        "GetTypeInfoCount;" & _
        "GetTypeInfo;" & _
        "GetIDsOfNames;" & _
        "Invoke;"
;===============================================================================

and change the dtag to below (although my understanding would then be that get_accParent is not logic to be in there anymore

Global $dtagIAccessible = $dtagIDispatch & _
"get_accParent hresult(ptr*);" & _                               ; IAccessible
...

So I am a little confused and need some references for understanding this in a better way

edit: I think I understand as oleacc.h defines the interface differently compared to when you use a typelib inspector / TLBINF32 information on UIAutomationCore.dll

oleacc.h

Edited February 11, 2014 by junkew

LarsJ · February 12, 2014

When I make interface descriptions I use header files (in this case oleacc.h) from Windows SDK to figure out the v-table order, and I use MSDN documentation to figure out the parameters (in this case IDispatch and IAccessible).

This is the relevant part of oleacc.h:

/* C style interface */

typedef struct IAccessibleVtbl
{
    BEGIN_INTERFACE
    
    HRESULT ( STDMETHODCALLTYPE *QueryInterface )( 
        __RPC__in IAccessible * This,
        /* [in] */ __RPC__in REFIID riid,
        /* [annotation][iid_is][out] */ 
        __RPC__deref_out  void **ppvObject);
    
    ULONG ( STDMETHODCALLTYPE *AddRef )( 
        __RPC__in IAccessible * This);
    
    ULONG ( STDMETHODCALLTYPE *Release )( 
        __RPC__in IAccessible * This);
    
    HRESULT ( STDMETHODCALLTYPE *GetTypeInfoCount )( 
        __RPC__in IAccessible * This,
        /* [out] */ __RPC__out UINT *pctinfo);
    
    HRESULT ( STDMETHODCALLTYPE *GetTypeInfo )( 
        __RPC__in IAccessible * This,
        /* [in] */ UINT iTInfo,
        /* [in] */ LCID lcid,
        /* [out] */ __RPC__deref_out_opt ITypeInfo **ppTInfo);
    
    HRESULT ( STDMETHODCALLTYPE *GetIDsOfNames )( 
        __RPC__in IAccessible * This,
        /* [in] */ __RPC__in REFIID riid,
        /* [size_is][in] */ __RPC__in_ecount_full(cNames) LPOLESTR *rgszNames,
        /* [range][in] */ __RPC__in_range(0,16384) UINT cNames,
        /* [in] */ LCID lcid,
        /* [size_is][out] */ __RPC__out_ecount_full(cNames) DISPID *rgDispId);
    
    /* [local] */ HRESULT ( STDMETHODCALLTYPE *Invoke )( 
        IAccessible * This,
        /* [in] */ DISPID dispIdMember,
        /* [in] */ REFIID riid,
        /* [in] */ LCID lcid,
        /* [in] */ WORD wFlags,
        /* [out][in] */ DISPPARAMS *pDispParams,
        /* [out] */ VARIANT *pVarResult,
        /* [out] */ EXCEPINFO *pExcepInfo,
        /* [out] */ UINT *puArgErr);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accParent )( 
        __RPC__in IAccessible * This,
        /* [retval][out] */ __RPC__deref_out_opt IDispatch **ppdispParent);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accChildCount )( 
        __RPC__in IAccessible * This,
        /* [retval][out] */ __RPC__out long *pcountChildren);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accChild )( 
        __RPC__in IAccessible * This,
        /* [in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt IDispatch **ppdispChild);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accName )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszName);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accValue )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszValue);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accDescription )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszDescription);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accRole )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__out VARIANT *pvarRole);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accState )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__out VARIANT *pvarState);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accHelp )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszHelp);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accHelpTopic )( 
        __RPC__in IAccessible * This,
        /* [out] */ __RPC__deref_out_opt BSTR *pszHelpFile,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__out long *pidTopic);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accKeyboardShortcut )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszKeyboardShortcut);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accFocus )( 
        __RPC__in IAccessible * This,
        /* [retval][out] */ __RPC__out VARIANT *pvarChild);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accSelection )( 
        __RPC__in IAccessible * This,
        /* [retval][out] */ __RPC__out VARIANT *pvarChildren);
    
    /* [id][propget][hidden] */ HRESULT ( STDMETHODCALLTYPE *get_accDefaultAction )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [retval][out] */ __RPC__deref_out_opt BSTR *pszDefaultAction);
    
    /* [id][hidden] */ HRESULT ( STDMETHODCALLTYPE *accSelect )( 
        __RPC__in IAccessible * This,
        /* [in] */ long flagsSelect,
        /* [optional][in] */ VARIANT varChild);
    
    /* [id][hidden] */ HRESULT ( STDMETHODCALLTYPE *accLocation )( 
        __RPC__in IAccessible * This,
        /* [out] */ __RPC__out long *pxLeft,
        /* [out] */ __RPC__out long *pyTop,
        /* [out] */ __RPC__out long *pcxWidth,
        /* [out] */ __RPC__out long *pcyHeight,
        /* [optional][in] */ VARIANT varChild);
    
    /* [id][hidden] */ HRESULT ( STDMETHODCALLTYPE *accNavigate )( 
        __RPC__in IAccessible * This,
        /* [in] */ long navDir,
        /* [optional][in] */ VARIANT varStart,
        /* [retval][out] */ __RPC__out VARIANT *pvarEndUpAt);
    
    /* [id][hidden] */ HRESULT ( STDMETHODCALLTYPE *accHitTest )( 
        __RPC__in IAccessible * This,
        /* [in] */ long xLeft,
        /* [in] */ long yTop,
        /* [retval][out] */ __RPC__out VARIANT *pvarChild);
    
    /* [id][hidden] */ HRESULT ( STDMETHODCALLTYPE *accDoDefaultAction )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild);
    
    /* [id][propput][hidden] */ HRESULT ( STDMETHODCALLTYPE *put_accName )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [in] */ __RPC__in BSTR szName);
    
    /* [id][propput][hidden] */ HRESULT ( STDMETHODCALLTYPE *put_accValue )( 
        __RPC__in IAccessible * This,
        /* [optional][in] */ VARIANT varChild,
        /* [in] */ __RPC__in BSTR szValue);
    
    END_INTERFACE
} IAccessibleVtbl;

The first 3 methods belongs to IUnknown, next 4 to IDispatch and the last 21 to IAccessible. ObjCreateInterface adds automatically the 3 IUnknown methods. It does not add IDispatch methods. This means that the interface description string must contain 25 methods with the order as shown in the header file.

You should use the MicroSoft header files. Then there is nothing to be in doubts about.

If you look at the documentation for get_accParent (the link above), you'll see that it takes one parameter, it's an output parameter, and it's a pointer to a pointer to an IDispatch interface. What we have to specify is the first pointer. It's an output parameter so the specifikation must be: ptr*

I have not gone through all 21 methods in IAccessible. Only the ones I've used. Some of the parameters seems to be wrong.

Method names. The method names in the description string are only used in AutoIt. These names doesn't matter as long as they are different and valid AutoIt function names. For IAccessible you can use the names junkew1 to junkew21. This would work perfectly alright. But it's difficult to find documentation for junkew1. It seems easier to use the same names as are used in the documentation and in the header file.

AutoItObject. You should only use AutoItObject, if you need to create your own objects. For UI Automation you should not use AutoItObject. ObjCreateInterface is much better and much faster. Because you create objects all the time speed is important. With FindAll and an IUIAutomationElementArray you can easily need to create hundreds of objects in a loop.

However, you can use the same description string for ObjCreateInterface and AutoItObject. And you don't need the list descriptions. Not even for AutoItObject. You should delete the lists.

Edited February 12, 2014 by LarsJ

junkew · February 12, 2014

I used the code from threads

'?do=embed' frameborder='0' data-embedContent>>

to generate the CUIAutomation2.au3 (CUIAutomation.au3 is the AIO version) appareantly thats the reason why multiple interfaces are incorrect described.

Will make a new version based on the input given in the thread and check if i can find the vtbl information of UIAutomationCore in an IDL or H file.

And indeed I also make use of the windows SDK when something seems not to be behaving as expected

LarsJ · February 19, 2014

Microsoft Active Accessibility (MSAA). This example shows how to use IServiceProvider to get pointers to IAccessible2 and IAccessibleEx (doku here and here) interfaces from an IAccessible object.

To get pointers to IAccessible2 interfaces you must register a IAccessible2 COM proxy stub DLL, IAccessible2Proxy.dll. Or you can just copy the DLL to System32 (and SysWOW64 on a 64 bit Windows) and reboot. You can download the DLL on the middle of the page in the link. The purpose of the DLL is to make IAccessible2 interfaces known to Windows. If you don't register the DLL, you'll only get IAccessibleEx interfaces.

I have tested IAccessible2 with Mozilla Firefox, Google Chrome and Opera. Note that these browsers are all running as 32 bit programs. Also on a 64 bit Windows. This means that you must run the example as a 32 bit program (the DLL above is also a 32 bit version). IAccessibleEx interfaces are tested with Internet Explorer.

This is the code to get IAccessible2 interface pointers:

; Try to get an IAccessible2 interface pointer for the child object
If $oChildObj.QueryInterface( DllStructGetPtr( $tIID_IServiceProvider ), $pService ) = $S_OK Then
  $oService = ObjCreateInterface( $pService, $sIID_IServiceProvider, $dtagIServiceProvider )
  If $oService.QueryService( $tIID_IAccessible2, $tIID_IAccessible2, $pIAccessible2 ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$pIAccessible2 = " & Ptr( $pIAccessible2 )
    $iInfo += 1
  EndIf
EndIf

The code for IAccessibleEx is similar.

To run the example you must replace the entire $dtagIAccessible description string in CUIAutomation2.au3 with this description (this is the same description as in post #136 and #142):

Global $dtagIAccessible = "GetTypeInfoCount hresult(uint*);" & _ ; IDispatch
"GetTypeInfo hresult(uint;int;ptr*);" & _
"GetIDsOfNames hresult(struct*;wstr;uint;int;int);" & _
"Invoke hresult(int;struct*;int;word;ptr*;ptr*;ptr*;uint*);" & _
"get_accParent hresult(ptr*);" & _                               ; IAccessible
"get_accChildCount hresult(long*);" & _
"get_accChild hresult(variant;idispatch*);" & _
"get_accName hresult(variant;bstr*);" & _
"get_accValue hresult(variant;bstr*);" & _
"get_accDescription hresult(variant;bstr*);" & _
"get_accRole hresult(variant;variant*);" & _
"get_accState hresult(variant;variant*);" & _
"get_accHelp hresult(variant;bstr*);" & _
"get_accHelpTopic hresult(bstr*;variant;long*);" & _
"get_accKeyboardShortcut hresult(variant;bstr*);" & _
"get_accFocus hresult(struct*);" & _
"get_accSelection hresult(variant*);" & _
"get_accDefaultAction hresult(variant;bstr*);" & _
"accSelect hresult(long;variant);" & _
"accLocation hresult(long*;long*;long*;long*;variant);" & _
"accNavigate hresult(long;variant;variant*);" & _
"accHitTest hresult(long;long;variant*);" & _
"accDoDefaultAction hresult(variant);" & _
"put_accName hresult(variant;bstr);" & _
"put_accValue hresult(variant;bstr);"

This is the code for the example. Press Ctrl+w when the mouse is over a window. Results are shown with _ArrayDisplay. Look for lines like "$pIAccessible2 = 0x00905D64" and "$pIAccessibleEx = 0x00905E64".

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Global $aInfo[10000], $iInfo = 0

MainFunc()


Func MainFunc()

  HotKeySet( "^w", "GetWindowInfo" )
  HotKeySet( "{ESC}", "Close" )

  While Sleep(100)
  WEnd

EndFunc


Func GetWindowInfo()
  Local $aMousePos = MouseGetPos(), $pObject, $tVarChild, $vt, $err = 0
  If AccessibleObjectFromPoint( $aMousePos[0], $aMousePos[1], $pObject, $tVarChild ) = $S_OK Then
    $vt = BitAND( DllStructGetData( $tVarChild, 1, 1 ), 0xFFFF )
    If $vt = $VT_I4 Then
      Local $oObject, $hWnd, $sName = ""
      If WindowFromAccessibleObject( $pObject, $hWnd ) = $S_OK Then

        AccessibleObjectFromWindow( $hWnd, $OBJID_CLIENT, $tIID_IAccessible, $pObject )
        $oObject = ObjCreateInterface( $pObject, $sIID_IAccessible, $dtagIAccessible )

        ; Get parent objects to top window
        Local $pParent, $pPrev, $oPrev, $iRole
        Do
          $pPrev = $pParent
          $oPrev = $oObject
          $oObject.get_accParent( $pParent )
          $oObject = ObjCreateInterface( $pParent, $sIID_IAccessible, $dtagIAccessible )
          $oObject.get_accName( $CHILDID_SELF, $sName )
        Until String( $sName ) = "Desktop"

        ; Info for top window (previous to desktop)
        PrintElementInfoToArray( $pPrev, $oPrev, $CHILDID_SELF, "" )

        ; Info for child objects of the top window
        If $oPrev.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
          WindowFromAccessibleObject( $pPrev, $hWnd )
          PrintWindowObjects( $hWnd )
        Else
          $err = 1
        EndIf

      Else
        $err = 1
      EndIf
    Else
      $err = 1
    EndIf
  Else
    $err = 1
  EndIf
  If $err Then _
    MsgBox( 0, "IAccessible2 and IAccessibleEx interfaces", _
               "AccessibleObjectFromPoint failed." & @CRLF & _
               "Try another point e.g. the window title bar." )
EndFunc


Func PrintWindowObjects( $hWindow )

  ; Object from window
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return

  ; Get window objects
  WalkTreeWithAccessibleChildren( $pWindow, 0, 5 )

  ; Print window objects
  ReDim $aInfo[$iInfo]
  _ArrayDisplay( $aInfo, "IAccessible2 and IAccessibleEx interfaces" )
  ReDim $aInfo[10000]
  $iInfo = 0

EndFunc


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return
  $oAcc.AddRef()

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "
  Next

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  Local $vt, $pChildObj, $oChildObj, $iChildElem, $iRole, $hWnd
  Local $pService, $oService, $pIAccessible2, $pIAccessibleEx

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object

      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )

      If IsObj( $oChildObj ) Then

        ; Try to get an IAccessible2 interface pointer for the child object
        If $oChildObj.QueryInterface( DllStructGetPtr( $tIID_IServiceProvider ), $pService ) = $S_OK Then
          $oService = ObjCreateInterface( $pService, $sIID_IServiceProvider, $dtagIServiceProvider )
          If $oService.QueryService( $tIID_IAccessible2, $tIID_IAccessible2, $pIAccessible2 ) = $S_OK Then
            $aInfo[$iInfo] = $sIndent & "$pIAccessible2 = " & Ptr( $pIAccessible2 )
            $iInfo += 1
          EndIf
        EndIf

        PrintElementInfoToArray( $pChildObj, $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )

      EndIf

    Else ; $vt = $VT_I4

      ; Child element

      ; Try to get an IAccessibleEx interface pointer for the parent object
      If $oAcc.QueryInterface( DllStructGetPtr( $tIID_IServiceProvider ), $pService ) = $S_OK Then
        $oService = ObjCreateInterface( $pService, $sIID_IServiceProvider, $dtagIServiceProvider )
        If $oService.QueryService( $tIID_IAccessibleEx, $tIID_IAccessibleEx, $pIAccessibleEx ) = $S_OK Then
          $aInfo[$iInfo] = $sIndent & "$pIAccessibleEx = " & Ptr( $pIAccessibleEx )
          $iInfo += 1
        EndIf
      EndIf

      $iChildElem = DllStructGetData( $tVarChildren, $i, 3 )
      PrintElementInfoToArray( $pAcc, $oAcc, $iChildElem, $sIndent )

    EndIf

  Next

EndFunc


Func PrintElementInfoToArray( $pElement, $oElement, $iChild, $sIndent )
  Local $hWnd, $sName, $iRole, $sRole, $iRoleLen
  Local $iState, $sState, $iStateLen
  Local $sValue, $x, $y, $w, $h
  If $iChild <> $CHILDID_SELF Then
    $aInfo[$iInfo] = $sIndent & "$iChildElem = " & $iChild
    $iInfo += 1
  EndIf
  If $iChild = $CHILDID_SELF And $oElement.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
    WindowFromAccessibleObject( $pElement, $hWnd )
    $aInfo[$iInfo] = $sIndent & "$hWnd   = " & $hWnd
    $iInfo += 1
  EndIf
  $oElement.get_accName( $iChild, $sName )
  $aInfo[$iInfo] = $sIndent & "$sName  = " & $sName
  $iInfo += 1
  If $oElement.get_accRole( $iChild, $iRole ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iRole  = 0x" & Hex( $iRole )
    $iInfo += 1
    $iRoleLen = GetRoleText( $iRole, 0, 0 ) + 1
    $sRole = DllStructCreate( "wchar[" & $iRoleLen & "]" )
    GetRoleText( $iRole, DllStructGetPtr( $sRole ), $iRoleLen )
    $aInfo[$iInfo] = $sIndent & "$sRole  = " & DllStructGetData( $sRole, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accState( $iChild, $iState ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iState = 0x" & Hex( $iState )
    $iInfo += 1
    $iStateLen = GetStateText( $iState, 0, 0 ) + 1
    $sState = DllStructCreate( "wchar[" & $iStateLen & "]" )
    GetStateText( $iState, DllStructGetPtr( $sState ), $iStateLen )
    $aInfo[$iInfo] = $sIndent & "$sState = " & DllStructGetData( $sState, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accValue( $iChild, $sValue ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$sValue = " & $sValue
    $iInfo += 1
  EndIf
  IF $oElement.accLocation( $x, $y, $w, $h, $iChild ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$x, $y, $w, $h = " & $x & ", " & $y & ", " & $w & ", " & $h
    $iInfo += 1
  EndIf
  $aInfo[$iInfo] = ""
  $iInfo += 1
EndFunc


Func Close()
  Exit
EndFunc

You need this UDF: MSAccessibility.au3

To get IAccessible2 and IAccessibleEx objects we only have to create the interface descriptions.

junkew · February 20, 2014

I will add on TO DO list the ISimpleDom interface which will expose more of the DOM in firefox and chrome

https://developer.mozilla.org/en-US/docs/Web/Accessibility/AT-APIs/ImplementationFeatures/MSAA#Additional_DOM_Support

Jordanoise · February 20, 2014

Hello. I do not understand how to automate the left click on a hyperlink on the page in Chrome? I need to find a link with specific text and click on it. Prompt code please.

junkew · February 20, 2014

#include <MsgBoxConstants.au3>

MsgBox($MB_SYSTEMMODAL, "Prompt for Chrome automation", "Read post 1 example 5", 10)

LarsJ · February 20, 2014

Microsoft Active Accessibility (MSAA). WinEvents. To handle MSAA events you must implement SetWinEventHook, AccessibleObjectFromEvent and UnhookWinEvent. SetWinEventHook and UnhookWinEvent are already implemented in WinAPISys.au3 (WinAPIEx.au3 in 3.3.8.1). These two functions are just copied and renamed. We have to implement AccessibleObjectFromEvent.

This is an example with menu and focus events. The script is tested on Calculator.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )

Global $hMenuEventProc, $hMenuEventHook
Global $hFocusEventProc, $hFocusEventHook

MainFunc()


Func MainFunc()

  $hMenuEventProc = DllCallbackRegister( "MenuEventProc", "none", "ptr;dword;hwnd;long;long;dword;dword" )
  $hMenuEventHook = SetWinEventHook( $EVENT_SYSTEM_MENUSTART, $EVENT_SYSTEM_MENUPOPUPEND, DllCallbackGetPtr( $hMenuEventProc ) )

  $hFocusEventProc = DllCallbackRegister( "FocusEventProc", "none", "ptr;dword;hwnd;long;long;dword;dword" )
  $hFocusEventHook = SetWinEventHook( $EVENT_OBJECT_FOCUS, $EVENT_OBJECT_FOCUS, DllCallbackGetPtr( $hFocusEventProc ) )

  HotKeySet( "{ESC}", "Close" )

  While Sleep(100)
  WEnd

EndFunc


Func MenuEventProc( $hMenuEventHook, $iMenuEvent, $hWnd, $iObjectID, $iChildID, $iThreadID, $iEventTime )

  Local $pAccessible, $oAccessible, $tVarChild
  If AccessibleObjectFromEvent( $hWnd, $iObjectID, $iChildID, $pAccessible, $tVarChild ) = $S_OK Then
    $oAccessible = ObjCreateInterface( $pAccessible, $sIID_IAccessible, $dtagIAccessible )

    Switch $iMenuEvent
      Case $EVENT_SYSTEM_MENUSTART
        ConsoleWrite( "Menu event: $EVENT_SYSTEM_MENUSTART" & @CRLF )
      Case $EVENT_SYSTEM_MENUEND
        ConsoleWrite( "Menu event: $EVENT_SYSTEM_MENUEND" & @CRLF )
      Case $EVENT_SYSTEM_MENUPOPUPSTART
        ConsoleWrite( "Menu event: $EVENT_SYSTEM_MENUPOPUPSTART" & @CRLF )
      Case $EVENT_SYSTEM_MENUPOPUPEND
        ConsoleWrite( "Menu event: $EVENT_SYSTEM_MENUPOPUPEND" & @CRLF )
    EndSwitch

    ; Get parent objects to top window
    Local $pParent, $oObject = $oAccessible, $pPrev, $oPrev, $sName
    Do
      $pPrev = $pParent
      $oPrev = $oObject
      $oObject.get_accParent( $pParent )
      $oObject = ObjCreateInterface( $pParent, $sIID_IAccessible, $dtagIAccessible )
      $oObject.get_accName( $CHILDID_SELF, $sName )
    Until String( $sName ) = "Desktop"

    ; Name of top window (previous to desktop)
    $oPrev.get_accName( $CHILDID_SELF, $sName )

    ConsoleWrite( "Program = " & $sName & @CRLF )
    PrintElementInfo( $oAccessible, $iChildID, "" )
  EndIf

EndFunc


Func FocusEventProc( $hFocusEventHook, $iFocusEvent, $hWnd, $iObjectID, $iChildID, $iThreadID, $iEventTime )

  Local $pAccessible, $oAccessible, $tVarChild
  If AccessibleObjectFromEvent( $hWnd, $iObjectID, $iChildID, $pAccessible, $tVarChild ) = $S_OK Then
    $oAccessible = ObjCreateInterface( $pAccessible, $sIID_IAccessible, $dtagIAccessible )

    ; Get parent objects to top window
    Local $pParent, $oObject = $oAccessible, $pPrev, $oPrev, $sName
    Do
      $pPrev = $pParent
      $oPrev = $oObject
      $oObject.get_accParent( $pParent )
      $oObject = ObjCreateInterface( $pParent, $sIID_IAccessible, $dtagIAccessible )
      $oObject.get_accName( $CHILDID_SELF, $sName )
    Until String( $sName ) = "Desktop"

    ; Name of top window (previous to desktop)
    $oPrev.get_accName( $CHILDID_SELF, $sName )

    ConsoleWrite( "Focus event:" & @CRLF )
    ConsoleWrite( "Program = " & $sName & @CRLF )
    PrintElementInfo( $oAccessible, $iChildID, "" )
  EndIf

EndFunc


Func Close()
  UnhookWinEvent( $hMenuEventHook )
  DllCallbackFree( $hMenuEventProc )
  UnhookWinEvent( $hFocusEventHook )
  DllCallbackFree( $hFocusEventProc )
  Exit
EndFunc

You need this UDF: MSAccessibility.au3

To run the example you must replace the entire $dtagIAccessible description string in CUIAutomation2.au3 with this description (this is the same description as in post #136, #142 and #146):

Global $dtagIAccessible = "GetTypeInfoCount hresult(uint*);" & _ ; IDispatch
"GetTypeInfo hresult(uint;int;ptr*);" & _
"GetIDsOfNames hresult(struct*;wstr;uint;int;int);" & _
"Invoke hresult(int;struct*;int;word;ptr*;ptr*;ptr*;uint*);" & _
"get_accParent hresult(ptr*);" & _                               ; IAccessible
"get_accChildCount hresult(long*);" & _
"get_accChild hresult(variant;idispatch*);" & _
"get_accName hresult(variant;bstr*);" & _
"get_accValue hresult(variant;bstr*);" & _
"get_accDescription hresult(variant;bstr*);" & _
"get_accRole hresult(variant;variant*);" & _
"get_accState hresult(variant;variant*);" & _
"get_accHelp hresult(variant;bstr*);" & _
"get_accHelpTopic hresult(bstr*;variant;long*);" & _
"get_accKeyboardShortcut hresult(variant;bstr*);" & _
"get_accFocus hresult(struct*);" & _
"get_accSelection hresult(variant*);" & _
"get_accDefaultAction hresult(variant;bstr*);" & _
"accSelect hresult(long;variant);" & _
"accLocation hresult(long*;long*;long*;long*;variant);" & _
"accNavigate hresult(long;variant;variant*);" & _
"accHitTest hresult(long;long;variant*);" & _
"accDoDefaultAction hresult(variant);" & _
"put_accName hresult(variant;bstr);" & _
"put_accValue hresult(variant;bstr);"

junkew · February 20, 2014

updated first post with new CUIAutomation2.au3 (smallest file is the last one) as fixes of LarsJ

LarsJ · February 21, 2014

IAccessible2. WinEvents. In post #150 there is an example with MSAA events. IAccessible2 is a complement to MSAA and most of the functions that works for MSAA will also work for IAccessible2: The functions that are used to handle MSAA events can also be used to handle IAccessible2 events.

This is an example with the IAccessible2 events $IA2_EVENT_DOCUMENT_LOAD_COMPLETE, $IA2_EVENT_DOCUMENT_LOAD_STOPPED and $IA2_EVENT_DOCUMENT_RELOAD:

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"
#include "IAccessible2.au3"

#include <WindowsConstants.au3>
#include <GuiConstantsEx.au3>
#include <EditConstants.au3>

Opt( "MustDeclareVars", 1 )

Global $hEventProc, $hEventHook

Global $idEdit

MainFunc()


Func MainFunc()

  $hEventProc = DllCallbackRegister( "EventProc", "none", "ptr;dword;hwnd;long;long;dword;dword" )
  $hEventHook = SetWinEventHook( $IA2_EVENT_DOCUMENT_LOAD_COMPLETE, $IA2_EVENT_DOCUMENT_RELOAD, DllCallbackGetPtr( $hEventProc ) )

  ; Create GUI
  Local $iStyle = BitOR( $WS_CLIPCHILDREN, $WS_OVERLAPPEDWINDOW )
  Local $hGui = GUICreate( "IAccessible2 & IAccessibleEx interfaces", 400, 400, 100, 100, $iStyle )

  ; Create Edit control
  $iStyle = BitXOR( $GUI_SS_DEFAULT_EDIT, $WS_HSCROLL )
  $idEdit = GUICtrlCreateEdit( "", 0, 0, 400, 400, $iStyle )
  GUICtrlSetFont( $idEdit, 10, 400, 0, "Courier New" )
  GUICtrlSetBkColor( $idEdit, 0xFFFFFF )

  GuiSetState()

  ; Message loop
  Local $iMsg
  While 1
    $iMsg = GUIGetMsg()
    If $iMsg = 0 Or $iMsg = $GUI_EVENT_MOUSEMOVE Then ContinueLoop
    Switch $iMsg
      Case $GUI_EVENT_CLOSE
        ExitLoop
    EndSwitch
  WEnd

  UnhookWinEvent( $hEventHook )
  DllCallbackFree( $hEventProc )
  GUIDelete( $hGui )
  Exit

EndFunc


Func EventProc( $hEventHook, $iEvent, $hWnd, $iObjectID, $iChildID, $iThreadID, $iEventTime )

  Switch $iEvent
    Case $IA2_EVENT_DOCUMENT_LOAD_COMPLETE
      GUICtrlSetData( $idEdit, "$iEvent = " & $iEvent & " ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)" & @CRLF, 1 )
    Case $IA2_EVENT_DOCUMENT_LOAD_STOPPED
      GUICtrlSetData( $idEdit, "$iEvent = " & $iEvent & " ($IA2_EVENT_DOCUMENT_LOAD_STOPPED)" & @CRLF, 1 )
    Case $IA2_EVENT_DOCUMENT_RELOAD
      GUICtrlSetData( $idEdit, "$iEvent = " & $iEvent & " ($IA2_EVENT_DOCUMENT_RELOAD)" & @CRLF, 1 )
  EndSwitch

  GUICtrlSetData( $idEdit, "$hWnd   = " & $hWnd & "; $iObjectID = " & $iObjectID & "; $iChildID = " & $iChildID & @CRLF, 1 )

  Local $pObject, $oObject
  If AccessibleObjectFromWindow( $hWnd, $iObjectID, $tIID_IAccessible, $pObject ) = $S_OK Then
    $oObject = ObjCreateInterface( $pObject, $sIID_IAccessible, $dtagIAccessible )
    PrintElementInfoToEdit( $oObject, $CHILDID_SELF, "" )
  EndIf

EndFunc


Func PrintElementInfoToEdit( $oElement, $iChild, $sIndent )
  Local $hWnd, $sName, $iRole, $sRole, $iRoleLen
  Local $iState, $sState, $iStateLen
  Local $sValue, $x, $y, $w, $h
  If $iChild <> $CHILDID_SELF Then _
    GUICtrlSetData( $idEdit, $sIndent & "$iChildElem = " & $iChild & @CRLF, 1 )
  $oElement.get_accName( $iChild, $sName )
  GUICtrlSetData( $idEdit, $sIndent & "$sName  = " & $sName & @CRLF, 1 )
  If $oElement.get_accRole( $iChild, $iRole ) = $S_OK Then
    ConsoleWrite( $sIndent & "$iRole  = 0x" & Hex( $iRole ) & @CRLF )
    GUICtrlSetData( $idEdit, $sIndent & "$iRole  = 0x" & Hex( $iRole ) & @CRLF, 1 )
    $iRoleLen = GetRoleText( $iRole, 0, 0 ) + 1
    $sRole = DllStructCreate( "wchar[" & $iRoleLen & "]" )
    GetRoleText( $iRole, DllStructGetPtr( $sRole ), $iRoleLen )
    GUICtrlSetData( $idEdit, $sIndent & "$sRole  = " & DllStructGetData( $sRole, 1 ) & @CRLF, 1 )
  EndIf
  If $oElement.get_accState( $iChild, $iState ) = $S_OK Then
    GUICtrlSetData( $idEdit, $sIndent & "$iState = 0x" & Hex( $iState ) & @CRLF, 1 )
    $iStateLen = GetStateText( $iState, 0, 0 ) + 1
    $sState = DllStructCreate( "wchar[" & $iStateLen & "]" )
    GetStateText( $iState, DllStructGetPtr( $sState ), $iStateLen )
    GUICtrlSetData( $idEdit, $sIndent & "$sState = " & DllStructGetData( $sState, 1 ) & @CRLF, 1 )
  EndIf
  If $oElement.get_accValue( $iChild, $sValue ) = $S_OK Then _
    GUICtrlSetData( $idEdit, $sIndent & "$sValue = " & $sValue & @CRLF, 1 )
  IF $oElement.accLocation( $x, $y, $w, $h, $iChild ) = $S_OK Then _
    GUICtrlSetData( $idEdit, $sIndent & "$x, $y, $w, $h = " & $x & ", " & $y & ", " & $w & ", " & $h & @CRLF, 1 )
  GUICtrlSetData( $idEdit, @CRLF, 1 )
EndFunc

And this is the output in the edit box after a test with Firefox and Chrome:

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x00010718; $iObjectID = -4; $iChildID = -246966144
$sName  = Mozilla Firefox Start Page - Mozilla Firefox
$iRole  = 0x0000000E
$sRole  = application
$iState = 0x00160040
$sState = focusable
$x, $y, $w, $h = 216, 114, 1192, 903

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x00010718; $iObjectID = -4; $iChildID = -246973920
$sName  = AutoItScript - AutoItScript Website - Mozilla Firefox
$iRole  = 0x0000000E
$sRole  = application
$iState = 0x00160040
$sState = focusable
$x, $y, $w, $h = 216, 114, 1192, 903

$iEvent = 263 ($IA2_EVENT_DOCUMENT_RELOAD)
$hWnd   = 0x00010718; $iObjectID = -4; $iChildID = -246973920
$sName  = AutoItScript - AutoItScript Website - Mozilla Firefox
$iRole  = 0x0000000E
$sRole  = application
$iState = 0x00160040
$sState = focusable
$x, $y, $w, $h = 216, 114, 1192, 903

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x00010718; $iObjectID = -4; $iChildID = -163839136
$sName  = AutoItScript - AutoItScript Website - Mozilla Firefox
$iRole  = 0x0000000E
$sRole  = application
$iState = 0x00160040
$sState = focusable
$x, $y, $w, $h = 216, 114, 1192, 903

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x0003060C; $iObjectID = -4; $iChildID = -1013
$sName  = Accessibility
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = chrome://accessibility/
$x, $y, $w, $h = 217, 185, 1190, 830

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x00030622; $iObjectID = -4; $iChildID = -1030
$sName  = New Tab
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = chrome://apps/
$x, $y, $w, $h = 217, 185, 1190, 830

$iEvent = 261 ($IA2_EVENT_DOCUMENT_LOAD_COMPLETE)
$hWnd   = 0x00030624; $iObjectID = -4; $iChildID = -1054
$sName  = AutoItScript - AutoItScript Website
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 217, 185, 1175, 1112

See post #146 for conditions to use IAccessible2. Before you run this example you should run the example in post #146 to verify that you are able to get IAccessible2 pointers. If you don't get any IAccessible2 pointers you will not get any events.

You need MSAccessibility.au3 and IAccessible2.au3.

You also need the new (smallest) version of CUIAutomation2.au3 in first post.

LarsJ · February 28, 2014

IAccessible2. IAccessible2.au3 below contains structures, constants, enumerations and interface descriptions to implement IAccessible2 interfaces (IAccessible2 API Version 1.3 released autumn 2012).

See post #146 for conditions to use IAccessible2.

The main IAccessible2 interface is IAccessible2. You get an IAccessible2 interface pointer by using QueryService from an IAccessible object as described in post #146. This is shown in the example below.

From the IAccessible2 main object, you get pointers to the other interfaces with the usual QueryInterface method. Shown in the example.

An exception is IAccessibleApplication. You can't use QueryInterface in all cases. Then you have to use QueryService. Also shown in the example.

The example recursively walks through all the objects of a top window. For each object it creates IAccessible and IAccessible2 interfaces. For each IAccessible2 main interface it tries to get pointers for the other IAccessible2 interfaces. If it succeeds various information is extracted.

(You would normally not use a recursive procedure in a real automation event. You would try to find the object in a more direct way.)

Press Ctrl+w when the mouse is over a window. Results are shown with _ArrayDisplay. The array can contain several thousands rows. Give it a few seconds to fill the array.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"
#include "IAccessible2.au3"

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Global $aInfo[10000], $iInfo = 0
Global $fAccessibleApplication

MainFunc()


Func MainFunc()

  HotKeySet( "^w", "GetWindowInfo" )
  HotKeySet( "{ESC}", "Close" )

  While Sleep(100)
  WEnd

EndFunc


Func GetWindowInfo()
  Local $aMousePos = MouseGetPos(), $pObject, $tVarChild, $vt, $err = 0
  If AccessibleObjectFromPoint( $aMousePos[0], $aMousePos[1], $pObject, $tVarChild ) = $S_OK Then
    $vt = BitAND( DllStructGetData( $tVarChild, 1, 1 ), 0xFFFF )
    If $vt = $VT_I4 Then
      Local $oObject, $hWnd, $sName = ""
      If WindowFromAccessibleObject( $pObject, $hWnd ) = $S_OK Then

        AccessibleObjectFromWindow( $hWnd, $OBJID_CLIENT, $tIID_IAccessible, $pObject )
        $oObject = ObjCreateInterface( $pObject, $sIID_IAccessible, $dtagIAccessible )

        ; Get parent objects to top window
        Local $pParent, $pPrev, $oPrev, $iRole
        Do
          $pPrev = $pParent
          $oPrev = $oObject
          $oObject.get_accParent( $pParent )
          $oObject = ObjCreateInterface( $pParent, $sIID_IAccessible, $dtagIAccessible )
          $oObject.get_accName( $CHILDID_SELF, $sName )
        Until String( $sName ) = "Desktop"

        ; Info for top window (previous to desktop)
        PrintElementInfoToArray( $pPrev, $oPrev, $CHILDID_SELF, "" )

        ; Info for child objects of the top window
        If $oPrev.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
          WindowFromAccessibleObject( $pPrev, $hWnd )
          PrintWindowObjects( $hWnd )
        Else
          $err = 1
        EndIf

      Else
        $err = 1
      EndIf
    Else
      $err = 1
    EndIf
  Else
    $err = 1
  EndIf
  If $err Then _
    MsgBox( 0, "IAccessible2 interfaces", _
               "AccessibleObjectFromPoint failed." & @CRLF & _
               "Try another point e.g. the window title bar." )
EndFunc


Func PrintWindowObjects( $hWindow )

  ; Object from window
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return

  ; Get window objects
  $fAccessibleApplication = False
  WalkTreeWithAccessibleChildren( $pWindow, 0, 8 )

  ; Print window objects
  ReDim $aInfo[$iInfo]
  _ArrayDisplay( $aInfo, "IAccessible2 interfaces" )
  ReDim $aInfo[10000]
  $iInfo = 0

EndFunc


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return
  $oAcc.AddRef()

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "
  Next

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  Local $vt, $pChildObj, $oChildObj, $iChildElem, $iRole, $hWnd
  Local $pService, $oService, $pAccessible2, $oAccessible2, $iComStatus, $fShowError = False
  Local $iIA2Relations, $pRelation, $oRelation, $sIA2RelationType, $iIA2Role, $sIA2Attributes

  ; Interfaces
  Local $pAccessible2_2, $pAccessibleAction, $pAccessibleApplication, $pAccessibleComponent, $pAccessibleDocument, $pAccessibleEditableText
  Local $pAccessibleText, $pAccessibleText2, $pAccessibleHypertext, $pAccessibleHypertext2, $pAccessibleHyperlink, $pAccessibleImage
  Local $pAccessibleRelation, $pAccessibleTable, $pAccessibleTable2, $pAccessibleTableCell, $pAccessibleValue

  Local $oAccessibleAction, $iIA2Actions, $sIA2ActionName
  Local $oAccessibleApplication, $sIA2AppName, $sIA2AppVersion, $sIA2ToolkitName, $sIA2ToolkitVersion
  Local $oAccessibleHypertext, $iIA2Hyperlinks, $pHyperlink, $oHyperlink, $sIA2HyperlinkName

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object

      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )

      If IsObj( $oChildObj ) Then

        ; Try to get an IAccessible2 interface pointer for the child object
        If $oChildObj.QueryInterface( DllStructGetPtr( $tIID_IServiceProvider ), $pService ) = $S_OK Then
          $oService = ObjCreateInterface( $pService, $sIID_IServiceProvider, $dtagIServiceProvider )
          If $oService.QueryService( $tIID_IAccessible2, $tIID_IAccessible2, $pAccessible2 ) = $S_OK Then
            $aInfo[$iInfo] = $sIndent & "$pAccessible2     = " & Ptr( $pAccessible2 )
            $iInfo += 1
            $oAccessible2 = ObjCreateInterface( $pAccessible2, $sIID_IAccessible2, $dtagIAccessible2 )
            If IsObj( $oAccessible2 ) Then

              ; $oAccessible2.role
              $oAccessible2.role( $iIA2Role )
              If $iIA2Role > 0x400 Then
                $aInfo[$iInfo] = $sIndent & "$iIA2Role         = " & GetIA2Role( $iIA2Role )
              Else
                $aInfo[$iInfo] = $sIndent & "$iIA2Role         = 0x" & Hex( $iIA2Role )
              EndIf
              $iInfo += 1

              ; $oAccessible2.nRelations
              $oAccessible2.nRelations( $iIA2Relations )
              $aInfo[$iInfo] = $sIndent & "$iIA2Relations    = " & $iIA2Relations
              $iInfo += 1
              For $j = 0 To $iIA2Relations - 1
                $oAccessible2.relation( $j, $pRelation )
                If $pRelation Then
                  $oRelation = ObjCreateInterface( $pRelation, $sIID_IAccessibleRelation, $dtagIAccessibleRelation )
                  $oRelation.relationType( $sIA2RelationType )
                  $aInfo[$iInfo] = $sIndent & "$sIA2RelationType = " & $sIA2RelationType
                  $iInfo += 1
                EndIf
              Next

              ; $oAccessible2.attributes
              $oAccessible2.attributes( $sIA2Attributes )
              $aInfo[$iInfo] = $sIndent & "$sIA2Attributes   = " & $sIA2Attributes
              $iInfo += 1

              ; IAccessible2_2 interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessible2_2 ), $pAccessible2_2 )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessible2_2          = " & Ptr( $pAccessible2_2 )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessible2_2          = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleAction interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleAction ), $pAccessibleAction )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleAction       = " & Ptr( $pAccessibleAction )
                $iInfo += 1
                $oAccessibleAction = ObjCreateInterface( $pAccessibleAction, $sIID_IAccessibleAction, $dtagIAccessibleAction )
                $oAccessibleAction.nActions( $iIA2Actions )
                If $iIA2Actions Then
                  $aInfo[$iInfo] = "    " & $sIndent & "$iIA2Actions    = " & $iIA2Actions
                  $iInfo += 1
                  For $j = 0 To $iIA2Actions - 1
                    $oAccessibleAction.name( $j, $sIA2ActionName )
                    $aInfo[$iInfo] = "    " & $sIndent & "$sIA2ActionName = " & $sIA2ActionName
                    $iInfo += 1
                  Next
                EndIf
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleAction       = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleApplication interface (QueryInterface)
              If Not $fAccessibleApplication Then
                $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleApplication ), $pAccessibleApplication )
                If $iComStatus = $S_OK Then
                  $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleApplication  = " & Ptr( $pAccessibleApplication )
                  $iInfo += 1
                  $oAccessibleApplication = ObjCreateInterface( $pAccessibleApplication, $sIID_IAccessibleApplication, $dtagIAccessibleApplication )
                  $oAccessibleApplication.appName( $sIA2AppName )
                  $oAccessibleApplication.appVersion( $sIA2AppVersion )
                  $oAccessibleApplication.toolkitName( $sIA2ToolkitName )
                  $oAccessibleApplication.toolkitVersion( $sIA2ToolkitVersion )
                  $aInfo[$iInfo+0] = "    " & $sIndent & "$sIA2AppName        = " & $sIA2AppName
                  $aInfo[$iInfo+1] = "    " & $sIndent & "$sIA2AppVersion     = " & $sIA2AppVersion
                  $aInfo[$iInfo+2] = "    " & $sIndent & "$sIA2ToolkitName    = " & $sIA2ToolkitName
                  $aInfo[$iInfo+3] = "    " & $sIndent & "$sIA2ToolkitVersion = " & $sIA2ToolkitVersion
                  $iInfo += 4
                  $fAccessibleApplication = True
                ElseIf $fShowError Then
                  $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleApplication  = Error: " & Hex( $iComStatus )
                  $iInfo += 1
                EndIf
                If $iComStatus <> $S_OK Then
                  ; IAccessibleApplication interface (QueryService)
                  $iComStatus = $oService.QueryService( $tIID_IAccessibleApplication, $tIID_IAccessibleApplication, $pAccessibleApplication )
                  If $iComStatus = $S_OK Then
                    $aInfo[$iInfo] = $sIndent & "QueryService   $pAccessibleApplication  = " & Ptr( $pAccessibleApplication )
                    $iInfo += 1
                    $oAccessibleApplication = ObjCreateInterface( $pAccessibleApplication, $sIID_IAccessibleApplication, $dtagIAccessibleApplication )
                    $oAccessibleApplication.appName( $sIA2AppName )
                    $oAccessibleApplication.appVersion( $sIA2AppVersion )
                    $oAccessibleApplication.toolkitName( $sIA2ToolkitName )
                    $oAccessibleApplication.toolkitVersion( $sIA2ToolkitVersion )
                    $aInfo[$iInfo+0] = "    " & $sIndent & "$sIA2AppName        = " & $sIA2AppName
                    $aInfo[$iInfo+1] = "    " & $sIndent & "$sIA2AppVersion     = " & $sIA2AppVersion
                    $aInfo[$iInfo+2] = "    " & $sIndent & "$sIA2ToolkitName    = " & $sIA2ToolkitName
                    $aInfo[$iInfo+3] = "    " & $sIndent & "$sIA2ToolkitVersion = " & $sIA2ToolkitVersion
                    $iInfo += 4
                    $fAccessibleApplication = True
                  ElseIf $fShowError Then
                    $aInfo[$iInfo] = $sIndent & "QueryService   $pAccessibleApplication  = Error: " & Hex( $iComStatus )
                    $iInfo += 1
                  EndIf
                EndIf
              EndIf

              ; IAccessibleComponent interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleComponent ), $pAccessibleComponent )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleComponent    = " & Ptr( $pAccessibleComponent )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleComponent    = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleDocument interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleDocument ), $pAccessibleDocument )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleDocument     = " & Ptr( $pAccessibleDocument )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleDocument     = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleEditableText interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleEditableText ), $pAccessibleEditableText )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleEditableText = " & Ptr( $pAccessibleEditableText )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleEditableText = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleText interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleText ), $pAccessibleText )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleText         = " & Ptr( $pAccessibleText )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleText         = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleText2 interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleText2 ), $pAccessibleText2 )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleText2        = " & Ptr( $pAccessibleText2 )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleText2        = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleHypertext interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleHypertext ), $pAccessibleHypertext )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHypertext    = " & Ptr( $pAccessibleHypertext )
                $iInfo += 1
                $oAccessibleHypertext = ObjCreateInterface( $pAccessibleHypertext, $sIID_IAccessibleHypertext, $dtagIAccessibleHypertext )
                $oAccessibleHypertext.nHyperlinks( $iIA2Hyperlinks )
                If $iIA2Hyperlinks Then
                  $aInfo[$iInfo] = "    " & $sIndent & "$iIA2Hyperlinks = " & $iIA2Hyperlinks
                  $iInfo += 1
                  For $j = 0 To $iIA2Hyperlinks - 1
                    $oAccessibleHypertext.hyperlink( $j, $pHyperlink )
                    If $pHyperlink Then
                      $oHyperlink = ObjCreateInterface( $pHyperlink, $sIID_IAccessibleHyperlink, $dtagIAccessibleHyperlink )
                      $oHyperlink.name( $sIA2HyperlinkName )
                      $aInfo[$iInfo+0] = "    " & $sIndent & "$iHyperlink        = " & $j
                      $aInfo[$iInfo+1] = "    " & $sIndent & "$sIA2HyperlinkName = " & $sIA2HyperlinkName
                      $iInfo += 2
                    EndIf
                  Next
                EndIf
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHypertext    = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleHypertext2 interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleHypertext2 ), $pAccessibleHypertext2 )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHypertext2   = " & Ptr( $pAccessibleHypertext2 )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHypertext2   = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleHyperlink interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleHyperlink ), $pAccessibleHyperlink )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHyperlink    = " & Ptr( $pAccessibleHyperlink )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleHyperlink    = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleImage interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleImage ), $pAccessibleImage )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleImage        = " & Ptr( $pAccessibleImage )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleImage        = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleRelation interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleRelation ), $pAccessibleRelation )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleRelation     = " & Ptr( $pAccessibleRelation )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleRelation     = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleTable interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleTable ), $pAccessibleTable )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTable        = " & Ptr( $pAccessibleTable )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTable        = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleTable2 interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleTable2 ), $pAccessibleTable2 )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTable2       = " & Ptr( $pAccessibleTable2 )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTable2       = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleTableCell interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleTableCell ), $pAccessibleTableCell )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTableCell    = " & Ptr( $pAccessibleTableCell )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleTableCell    = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

              ; IAccessibleValue interface (QueryInterface)
              $iComStatus = $oAccessible2.QueryInterface( DllStructGetPtr( $tIID_IAccessibleValue ), $pAccessibleValue )
              If $iComStatus = $S_OK Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleValue        = " & Ptr( $pAccessibleValue )
                $iInfo += 1
              ElseIf $fShowError Then
                $aInfo[$iInfo] = $sIndent & "QueryInterface $pAccessibleValue        = Error: " & Hex( $iComStatus )
                $iInfo += 1
              EndIf

            EndIf
          EndIf
        EndIf

        PrintElementInfoToArray( $pChildObj, $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )

      EndIf

    EndIf

  Next

EndFunc


Func PrintElementInfoToArray( $pElement, $oElement, $iChild, $sIndent )
  Local $hWnd, $sName, $iRole, $sRole, $iRoleLen
  Local $iState, $sState, $iStateLen
  Local $sValue, $x, $y, $w, $h
  If $iChild <> $CHILDID_SELF Then
    $aInfo[$iInfo] = $sIndent & "$iChildElem = " & $iChild
    $iInfo += 1
  EndIf
  If $iChild = $CHILDID_SELF And $oElement.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
    WindowFromAccessibleObject( $pElement, $hWnd )
    $aInfo[$iInfo] = $sIndent & "$hWnd   = " & $hWnd
    $iInfo += 1
  EndIf
  $oElement.get_accName( $iChild, $sName )
  $aInfo[$iInfo] = $sIndent & "$sName  = " & $sName
  $iInfo += 1
  If $oElement.get_accRole( $iChild, $iRole ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iRole  = 0x" & Hex( $iRole )
    $iInfo += 1
    $iRoleLen = GetRoleText( $iRole, 0, 0 ) + 1
    $sRole = DllStructCreate( "wchar[" & $iRoleLen & "]" )
    GetRoleText( $iRole, DllStructGetPtr( $sRole ), $iRoleLen )
    $aInfo[$iInfo] = $sIndent & "$sRole  = " & DllStructGetData( $sRole, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accState( $iChild, $iState ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iState = 0x" & Hex( $iState )
    $iInfo += 1
    $iStateLen = GetStateText( $iState, 0, 0 ) + 1
    $sState = DllStructCreate( "wchar[" & $iStateLen & "]" )
    GetStateText( $iState, DllStructGetPtr( $sState ), $iStateLen )
    $aInfo[$iInfo] = $sIndent & "$sState = " & DllStructGetData( $sState, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accValue( $iChild, $sValue ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$sValue = " & $sValue
    $iInfo += 1
  EndIf
  IF $oElement.accLocation( $x, $y, $w, $h, $iChild ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$x, $y, $w, $h = " & $x & ", " & $y & ", " & $w & ", " & $h
    $iInfo += 1
  EndIf
  $aInfo[$iInfo] = ""
  $iInfo += 1
EndFunc


Func Close()
  Exit
EndFunc

The example is tested with Firefox, Chrome and Opera. For Firefox accessibility mode is enabled by default. For Chrome and Opera accessibility mode is enabled when you Press Ctrl+w when the mouse is over the window title bar (looking for IAccessible interfaces seems to enable accessibility mode). For Opera you have to Press Ctrl+w once more to get all results.

The textbox shows a small part of the output from the AutoIt home site (http://www.autoitscript.com/site/). The textbox contains info about the top window for the three browsers, the first object with application info, and the AutoIt logo object in the upper left corner.

Mozilla Firefox
===============
$hWnd   = 0x00020508
$sName  = AutoItScript - AutoItScript Website - Mozilla Firefox
$iRole  = 0x00000009
$sRole  = window
$iState = 0x00160004
$sState = focusable
$x, $y, $w, $h = 212, 84, 1200, 937

$pAccessible2     = 0x00C3EDF4
$iIA2Role         = 0x0000000B
$iIA2Relations    = 2
$sIA2RelationType = containingDocument
$sIA2RelationType = containingApplication
$sIA2Attributes   = margin-left:0px;text-align:start;text-indent:0px;id:tabContextMenu;tag:menupopup;margin-right:0px;margin-top:0px;margin-bottom:0px;display:-moz-popup;
QueryInterface $pAccessible2_2          = 0x00C2805C
QueryInterface $pAccessibleAction       = 0x00C8BF9C
QueryService   $pAccessibleApplication  = 0x00C284DC
    $sIA2AppName        = Firefox
    $sIA2AppVersion     = 27.0.1
    $sIA2ToolkitName    = Gecko
    $sIA2ToolkitVersion = 27.0.1
QueryInterface $pAccessibleComponent    = 0x00C28C74
$sName  = 0
$iRole  = 0x0000000B
$sRole  = popup menu
$iState = 0x00019400
$sState = offscreen
$x, $y, $w, $h = 216, 114, 0, 0

$pAccessible2     = 0x00BB2EF4
$iIA2Role         = 0x00000028
$iIA2Relations    = 3
$sIA2RelationType = containingDocument
$sIA2RelationType = containingTabPane
$sIA2RelationType = containingApplication
$sIA2Attributes   = margin-left:0px;src:http\://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png;text-align:left;draggable:true;text-indent:0px;id:logo;tag:img;margin-right:10px;margin-top:0px;margin-bottom:0px;display:block;explicit-name:true;
QueryInterface $pAccessible2_2          = 0x00BB6D5C
QueryInterface $pAccessibleAction       = 0x033B700C
    $iIA2Actions    = 1
    $sIA2ActionName = jump
QueryInterface $pAccessibleComponent    = 0x033B7304
QueryInterface $pAccessibleHyperlink    = 0x00BADFA4
QueryInterface $pAccessibleImage        = 0x033B75AC
$sName  = AutoItScript
$iRole  = 0x00000028
$sRole  = graphic
$iState = 0x00C01000
$sState = traversed
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 324, 197, 210, 72


Google Chrome
=============
$hWnd   = 0x000304B2
$sName  = AutoItScript - AutoItScript Website - Google Chrome
$iRole  = 0x00000009
$sRole  = window
$iState = 0x00160004
$sState = focusable
$x, $y, $w, $h = 212, 84, 1200, 936

$pAccessible2     = 0x00BA8CC4
$iIA2Role         = 0x0000000F
$iIA2Relations    = 0
$sIA2Attributes   = tag:#document
QueryInterface $pAccessibleApplication  = 0x00C9F14C
    $sIA2AppName        = Chrome
    $sIA2AppVersion     = 33.0.1750.117
    $sIA2ToolkitName    = Chrome
    $sIA2ToolkitVersion = Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36
QueryInterface $pAccessibleText         = 0x00B8CA54
QueryInterface $pAccessibleHypertext    = 0x00B436BC
    $iIA2Hyperlinks = 26
    $iHyperlink        = 0
    $sIA2HyperlinkName = 
    $iHyperlink        = 1
    $sIA2HyperlinkName = 
    $iHyperlink        = 2
    $sIA2HyperlinkName = 
    $iHyperlink        = 3
    $sIA2HyperlinkName = 
    $iHyperlink        = 4
    $sIA2HyperlinkName = 
    $iHyperlink        = 5
    $sIA2HyperlinkName = 
    $iHyperlink        = 6
    $sIA2HyperlinkName = 
    $iHyperlink        = 7
    $sIA2HyperlinkName = 
    $iHyperlink        = 8
    $sIA2HyperlinkName = 
    $iHyperlink        = 9
    $sIA2HyperlinkName = 
    $iHyperlink        = 10
    $sIA2HyperlinkName = 
    $iHyperlink        = 11
    $sIA2HyperlinkName = 
    $iHyperlink        = 12
    $sIA2HyperlinkName = 
    $iHyperlink        = 13
    $sIA2HyperlinkName = 
    $iHyperlink        = 14
    $sIA2HyperlinkName = 
    $iHyperlink        = 15
    $sIA2HyperlinkName = 
    $iHyperlink        = 16
    $sIA2HyperlinkName = 
    $iHyperlink        = 17
    $sIA2HyperlinkName = 
    $iHyperlink        = 18
    $sIA2HyperlinkName = 
    $iHyperlink        = 19
    $sIA2HyperlinkName = 
    $iHyperlink        = 20
    $sIA2HyperlinkName = 
    $iHyperlink        = 21
    $sIA2HyperlinkName = 
    $iHyperlink        = 22
    $sIA2HyperlinkName = 
    $iHyperlink        = 23
    $sIA2HyperlinkName = 
    $iHyperlink        = 24
    $sIA2HyperlinkName = 
    $iHyperlink        = 25
    $sIA2HyperlinkName = 
QueryInterface $pAccessibleHyperlink    = 0x00B4350C
$sName  = AutoItScript - AutoItScript Website
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 217, 185, 1175, 1112

$pAccessible2     = 0x00C12DCC
$iIA2Role         = 0x0000001E
$iIA2Relations    = 0
$sIA2Attributes   = display:inline;tag:a
QueryInterface $pAccessibleText         = 0x00C96414
QueryInterface $pAccessibleHypertext    = 0x00C5F104
    $iIA2Hyperlinks = 1
    $iHyperlink        = 0
    $sIA2HyperlinkName = 
QueryInterface $pAccessibleHyperlink    = 0x02670904
$sName  = AutoItScript
$iRole  = 0x0000001E
$sRole  = link
$iState = 0x00D10000
$sState = traversed
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 324, 185, 211, 72

    $pAccessible2     = 0x00B6F594
    $iIA2Role         = 0x00000028
    $iIA2Relations    = 0
    $sIA2Attributes   = display:block;tag:img
    QueryInterface $pAccessibleText         = 0x00CD81BC
    QueryInterface $pAccessibleHypertext    = 0x00B4362C
    QueryInterface $pAccessibleHyperlink    = 0x00CAC93C
    QueryInterface $pAccessibleImage        = 0x0267A09C
    $sName  = AutoItScript
    $iRole  = 0x00000028
    $sRole  = graphic
    $iState = 0x00400040
    $sState = linked
    $sValue = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
    $x, $y, $w, $h = 325, 185, 211, 72


Opera browser
=============
$hWnd   = 0x0006050E
$sName  = AutoItScript - AutoItScript Website - Opera
$iRole  = 0x00000009
$sRole  = window
$iState = 0x00160000
$sState = focusable
$x, $y, $w, $h = 213, 85, 1198, 935

$pAccessible2     = 0x00BB1F44
$iIA2Role         = 0x0000000F
$iIA2Relations    = 0
$sIA2Attributes   = tag:#document
QueryInterface $pAccessibleApplication  = 0x026DACBC
    $sIA2AppName        = OPR
    $sIA2AppVersion     = 19.0.1326.63
    $sIA2ToolkitName    = Chrome
    $sIA2ToolkitVersion = Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36 OPR/19.0.1326.63
QueryInterface $pAccessibleText         = 0x0284C95C
QueryInterface $pAccessibleHypertext    = 0x0284C9A4
    $iIA2Hyperlinks = 26
    $iHyperlink        = 0
    $sIA2HyperlinkName = 
    $iHyperlink        = 1
    $sIA2HyperlinkName = 
    $iHyperlink        = 2
    $sIA2HyperlinkName = 
    $iHyperlink        = 3
    $sIA2HyperlinkName = 
    $iHyperlink        = 4
    $sIA2HyperlinkName = 
    $iHyperlink        = 5
    $sIA2HyperlinkName = 
    $iHyperlink        = 6
    $sIA2HyperlinkName = 
    $iHyperlink        = 7
    $sIA2HyperlinkName = 
    $iHyperlink        = 8
    $sIA2HyperlinkName = 
    $iHyperlink        = 9
    $sIA2HyperlinkName = 
    $iHyperlink        = 10
    $sIA2HyperlinkName = 
    $iHyperlink        = 11
    $sIA2HyperlinkName = 
    $iHyperlink        = 12
    $sIA2HyperlinkName = 
    $iHyperlink        = 13
    $sIA2HyperlinkName = 
    $iHyperlink        = 14
    $sIA2HyperlinkName = 
    $iHyperlink        = 15
    $sIA2HyperlinkName = 
    $iHyperlink        = 16
    $sIA2HyperlinkName = 
    $iHyperlink        = 17
    $sIA2HyperlinkName = 
    $iHyperlink        = 18
    $sIA2HyperlinkName = 
    $iHyperlink        = 19
    $sIA2HyperlinkName = 
    $iHyperlink        = 20
    $sIA2HyperlinkName = 
    $iHyperlink        = 21
    $sIA2HyperlinkName = 
    $iHyperlink        = 22
    $sIA2HyperlinkName = 
    $iHyperlink        = 23
    $sIA2HyperlinkName = 
    $iHyperlink        = 24
    $sIA2HyperlinkName = 
    $iHyperlink        = 25
    $sIA2HyperlinkName = 
QueryInterface $pAccessibleHyperlink    = 0x0292F6EC
$sName  = AutoItScript - AutoItScript Website
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 219, 177, 1169, 1112

$pAccessible2     = 0x03269F8C
$iIA2Role         = 0x0000001E
$iIA2Relations    = 0
$sIA2Attributes   = display:inline;tag:a
QueryInterface $pAccessibleText         = 0x028A1964
QueryInterface $pAccessibleHypertext    = 0x00B6E014
    $iIA2Hyperlinks = 1
    $iHyperlink        = 0
    $sIA2HyperlinkName = 
QueryInterface $pAccessibleHyperlink    = 0x00BA4E44
$sName  = AutoItScript
$iRole  = 0x0000001E
$sRole  = link
$iState = 0x00D10000
$sState = traversed
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 323, 177, 211, 72

    $pAccessible2     = 0x02769AEC
    $iIA2Role         = 0x00000028
    $iIA2Relations    = 0
    $sIA2Attributes   = display:block;tag:img
    QueryInterface $pAccessibleText         = 0x031F980C
    QueryInterface $pAccessibleHypertext    = 0x0287EB84
    QueryInterface $pAccessibleHyperlink    = 0x00CE4A8C
    QueryInterface $pAccessibleImage        = 0x027C86DC
    $sName  = AutoItScript
    $iRole  = 0x00000028
    $sRole  = graphic
    $iState = 0x00400040
    $sState = linked
    $sValue = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
    $x, $y, $w, $h = 324, 177, 211, 72

Note that Firefox treats the logo as a single graphic object with a jump action, while Chrome and Opera treats the logo as a main link object with a child graphic object. Note also that Opera is based on the Chrome toolkit.

You need MSAccessibility.au3 and IAccessible2.au3.

You also need the new (smallest) version of CUIAutomation2.au3 in first post.

LarsJ · March 9, 2014

ISimpleDOM. ISimpleDOM.au3 below contains constants, interface descriptions and functions to implement ISimpleDOM interfaces.

ISimpleDOM is created by Mozilla and is implemented in Firefox and Thunderbird. Google Chrome also supports ISimpleDOM. I'm not aware of other programs that supports ISimpleDOM.

The main interface is ISimpleDOMNode. You get an ISimpleDOMNode interface pointer by using QueryService from an IAccessible object.

There are two other intefaces: ISimpleDOMText and ISimpleDOMDocument. You get pointers to these interfaces with QueryInterface of ISimpleDOMNode.

The documentation is not very accurate. The most accurate documentation can be found in the header files. This is a part of the header files for the three interfaces:

ISimpleDOMNode

///////////////////////////////////////////////////////////////////////////////////////////////////////
//
// ISimpleDOMNode
// ---------------------------------------------------------------------------------------------------=
// An interface that extends MSAA's IAccessible to provide readonly DOM node information via cross-process COM.
//
// get_nodeInfo(
//  /* [out] */ BSTR  *nodeName,   // For elements, this is the tag name
//  /* [out] */ short  *nameSpaceID,
//  /* [out] */ BSTR  *nodeValue, 
//  /* [out] */ unsigned int    *numChildren); 
//  /* [out] */ unsigned int    *uniqueID;  // In Win32 accessible events we generate, the target's childID matches to this
//  /* [out] */ unsigned short  *nodeType,
// ---------------------------------------------------------------------------------------------------=
// Get the basic information about a node.
// The namespace ID can be mapped to an URI using nsISimpleDOMDocument::get_nameSpaceURIForID()
//
// get_attributes(
//  /* [in]  */ unsigned short maxAttribs,
//  /* [out] */ unsigned short  *numAttribs,
//  /* [out] */ BSTR  *attribNames,
//  /* [out] */ short *nameSpaceID,
//  /* [out] */ BSTR  *attribValues);
// ---------------------------------------------------------------------------------------------------=
// Returns 3 arrays - the attribute names and values, and a namespace ID for each
// If the namespace ID is 0, it's the same namespace as the node's namespace
//
// get_attributesForNames(
//  /* [in] */ unsigned short numAttribs,
//  /* [in] */ BSTR   *attribNames,
//  /* [in] */ short  *nameSpaceID,
//  /* [out] */ BSTR  *attribValues);
// ---------------------------------------------------------------------------------------------------=
// Takes 2 arrays - the attribute names and namespace IDs, and returns an array of corresponding values
// If the namespace ID is 0, it's the same namespace as the node's namespace
//
// computedStyle(  
//  /* [in]  */ unsigned short maxStyleProperties,
//  /* [out] */ unsigned short *numStyleProperties, 
//  /* [in]  */ boolean useAlternateView,  // If TRUE, returns properites for media as set in nsIDOMDocument::set_alternateViewMediaTypes
//  /* [out] */ BSTR *styleProperties, 
//  /* [out] */ BSTR *styleValues);
// ---------------------------------------------------------------------------------------------------=
// Returns 2 arrays -- the style properties and their values
//  useAlternateView=FALSE: gets properties for the default media type (usually screen)
//  useAlternateView=TRUE: properties for media types set w/ nsIDOMSimpleDocument::set_alternateViewMediaTypes()
//
// computedStyleForProperties(  
//  /* [in] */  unsigned short numStyleProperties, 
//  /* [in] */  boolean useAlternateView,  // If TRUE, returns properites for media as set in nsIDOMDocument::set_alternateViewMediaTypes
//  /* [in] */  BSTR *styleProperties, 
//  /* [out] */ BSTR *styleValues);
// ---------------------------------------------------------------------------------------------------=
// Scroll the current view so that this dom node is visible.
//  placeTopLeft=TRUE: scroll until the top left corner of the dom node is at the top left corner of the view.
//  placeTopLeft=FALSE: scroll minimally to make the dom node visible. Don't scroll at all if already visible.
//
// scrollTo( 
//  /* [in] */ boolean placeTopLeft); 
// ---------------------------------------------------------------------------------------------------=
// Returns style property values for those properties in the styleProperties [in] array
// Returns 2 arrays -- the style properties and their values
//  useAlternateView=FALSE: gets properties for the default media type (usually screen)
//  useAlternateView=TRUE: properties for media types set w/ nsIDOMSimpleDocument::set_alternateViewMediaTypes()
//
// get_parentNode     (/* [out] */ ISimpleDOMNode **newNodePtr);
// get_firstChild     (/* [out] */ ISimpleDOMNode **newNodePtr);
// get_lastChild      (/* [out] */ ISimpleDOMNode **newNodePtr);
// get_previousSibling(/* [out] */ ISimpleDOMNode **newNodePtr);
// get_nextSibling    (/* [out] */ ISimpleDOMNode **newNodePtr);
// get_childAt        (/* [in] */ unsigned childIndex, /* [out] */ ISimpleDOMNode **newNodePtr);
// ---------------------------------------------------------------------------------------------------=
// DOM navigation - get a different node.
//
// get_innerHTML(/* [out] */ BSTR *htmlText);
// ---------------------------------------------------------------------------------------------------=
// Returns HTML of this DOM node's subtree. Does not include the start and end tag for this node/element.
//
//
// get_localInterface(/* [out] */ void **localInterface);
// ---------------------------------------------------------------------------------------------------=
// Only available in Gecko's process
//
//
// get_language(/* [out] */ BSTR *htmlText);
// ---------------------------------------------------------------------------------------------------=
// Returns the computed language for this node, or empty string if unknown.
//
//
///////////////////////////////////////////////////////////////////////////////////////////////////////

#else   /* C style interface */

    typedef struct ISimpleDOMNodeVtbl
    {
        BEGIN_INTERFACE
        
        HRESULT ( STDMETHODCALLTYPE *QueryInterface )( 
            ISimpleDOMNode * This,
            /* [in] */ REFIID riid,
            /* [iid_is][out] */ 
            __RPC__deref_out  void **ppvObject);
        
        ULONG ( STDMETHODCALLTYPE *AddRef )( 
            ISimpleDOMNode * This);
        
        ULONG ( STDMETHODCALLTYPE *Release )( 
            ISimpleDOMNode * This);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_nodeInfo )( 
            ISimpleDOMNode * This,
            /* [out] */ BSTR *nodeName,
            /* [out] */ short *nameSpaceID,
            /* [out] */ BSTR *nodeValue,
            /* [out] */ unsigned int *numChildren,
            /* [out] */ unsigned int *uniqueID,
            /* [retval][out] */ unsigned short *nodeType);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_attributes )( 
            ISimpleDOMNode * This,
            /* [in] */ unsigned short maxAttribs,
            /* [length_is][size_is][out] */ BSTR *attribNames,
            /* [length_is][size_is][out] */ short *nameSpaceID,
            /* [length_is][size_is][out] */ BSTR *attribValues,
            /* [retval][out] */ unsigned short *numAttribs);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_attributesForNames )( 
            ISimpleDOMNode * This,
            /* [in] */ unsigned short numAttribs,
            /* [length_is][size_is][in] */ BSTR *attribNames,
            /* [length_is][size_is][in] */ short *nameSpaceID,
            /* [length_is][size_is][retval][out] */ BSTR *attribValues);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_computedStyle )( 
            ISimpleDOMNode * This,
            /* [in] */ unsigned short maxStyleProperties,
            /* [in] */ boolean useAlternateView,
            /* [length_is][size_is][out] */ BSTR *styleProperties,
            /* [length_is][size_is][out] */ BSTR *styleValues,
            /* [retval][out] */ unsigned short *numStyleProperties);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_computedStyleForProperties )( 
            ISimpleDOMNode * This,
            /* [in] */ unsigned short numStyleProperties,
            /* [in] */ boolean useAlternateView,
            /* [length_is][size_is][in] */ BSTR *styleProperties,
            /* [length_is][size_is][retval][out] */ BSTR *styleValues);
        
        HRESULT ( STDMETHODCALLTYPE *scrollTo )( 
            ISimpleDOMNode * This,
            /* [in] */ boolean placeTopLeft);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_parentNode )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_firstChild )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_lastChild )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_previousSibling )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_nextSibling )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_childAt )( 
            ISimpleDOMNode * This,
            /* [in] */ unsigned int childIndex,
            /* [retval][out] */ ISimpleDOMNode **node);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_innerHTML )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ BSTR *innerHTML);
        
        /* [local][propget] */ HRESULT ( STDMETHODCALLTYPE *get_localInterface )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ void **localInterface);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_language )( 
            ISimpleDOMNode * This,
            /* [retval][out] */ BSTR *language);
        
        END_INTERFACE
    } ISimpleDOMNodeVtbl;

Note that some of these methods takes arrays as parameters. This is shown in the example.

ISimpleDOMText

///////////////////////////////////////////////////////////////////////////////////////////////////////
//
// ISimpleDOMText
// ---------------------------------------------------------------------------------------------------=
// An interface that extends MSAA's IAccessible to provide important additional capabilities on text nodes
//
// [propget] domText(/* out,retval */ BSTR *domText
// ---------------------------------------------------------------------------------------------------=
// Similar to IAccessible::get_accName, but does not strip out whitespace characters.
// Important for retrieving the correct start/end substring indices to use with other
// methods in ISimpleDOMText.
//
//
// get_[un]clippedSubstringBounds(
//   /* [in] */ unsigned int startIndex,
//   /* [in] */ unsigned int endIndex,
//   /* [out] */ int *x,
//   /* [out] */ int *y,
//   /* [out] */ int *width,
//   /* [out] */ int *height);
// ---------------------------------------------------------------------------------------------------=
// Both methods get_clippedSubstringBounds and get_unclippedSubstringBounds return the screen pixel
// coordinates of the given text substring. The in parameters for start and end indices refer
// to the string returned by ISimpleDOMText::get_domText().
//
//
// scrollToSubstring(
//   /* [in] */ unsigned int startIndex,
//   /* [in] */ unsigned int endIndex);
// ---------------------------------------------------------------------------------------------------=
// In scrollable views, scrolls to ensure that the specified substring is visible onscreen.
// The in parameters for start and end indices refer to the string returned
// by ISimpleDOMText::get_domText().
//
//
// [propget] fontFamily(/* out,retval */ BSTR *fontFamily);
// ---------------------------------------------------------------------------------------------------=
// Return a single computed font family name, which is better than the comma delineated list
// that is returned by the ISimpleDOMNode computed style methods for font-family.
// In other words, return something like 'Arial' instead of 'Arial, Helvetica, Sans-serif'.
///////////////////////////////////////////////////////////////////////////////////////////////////////

#else   /* C style interface */

    typedef struct ISimpleDOMTextVtbl
    {
        BEGIN_INTERFACE
        
        HRESULT ( STDMETHODCALLTYPE *QueryInterface )( 
            ISimpleDOMText * This,
            /* [in] */ REFIID riid,
            /* [iid_is][out] */ 
            __RPC__deref_out  void **ppvObject);
        
        ULONG ( STDMETHODCALLTYPE *AddRef )( 
            ISimpleDOMText * This);
        
        ULONG ( STDMETHODCALLTYPE *Release )( 
            ISimpleDOMText * This);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_domText )( 
            ISimpleDOMText * This,
            /* [retval][out] */ BSTR *domText);
        
        HRESULT ( STDMETHODCALLTYPE *get_clippedSubstringBounds )( 
            ISimpleDOMText * This,
            /* [in] */ unsigned int startIndex,
            /* [in] */ unsigned int endIndex,
            /* [out] */ int *x,
            /* [out] */ int *y,
            /* [out] */ int *width,
            /* [out] */ int *height);
        
        HRESULT ( STDMETHODCALLTYPE *get_unclippedSubstringBounds )( 
            ISimpleDOMText * This,
            /* [in] */ unsigned int startIndex,
            /* [in] */ unsigned int endIndex,
            /* [out] */ int *x,
            /* [out] */ int *y,
            /* [out] */ int *width,
            /* [out] */ int *height);
        
        HRESULT ( STDMETHODCALLTYPE *scrollToSubstring )( 
            ISimpleDOMText * This,
            /* [in] */ unsigned int startIndex,
            /* [in] */ unsigned int endIndex);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_fontFamily )( 
            ISimpleDOMText * This,
            /* [retval][out] */ BSTR *fontFamily);
        
        END_INTERFACE
    } ISimpleDOMTextVtbl;

ISimpleDOMDocument

///////////////////////////////////////////////////////////////////////////////////////////////////////
//
// ISimpleDOMDocument
// ---------------------------------------------------------------------------------------------------=
//
// get_URL(out] BSTR *url)
// ---------------------------------------------------------------------------------------------------=
// Get the internet URL associated with this document.
//
// get_title([out BSTR *title
// ---------------------------------------------------------------------------------------------------=
// Get the document's title from the <TITLE> element
//
// get_mimeType([out BSTR *mimeType
// ---------------------------------------------------------------------------------------------------=
// Get the registered mime type, such as text/html
//
// get_docType([out] BSTR *docType
// ---------------------------------------------------------------------------------------------------=
// Get doctype associated with the <!DOCTYPE ..> element
//
// get_nameSpaceURIForID([in] short nameSpaceID, [out] BSTR *nameSpaceURI)
// ---------------------------------------------------------------------------------------------------=
// Some of the methods for ISimpleDOMNode return a nameSpaceID (-1,0,1,2,3,....)
// This method returns the associated namespace URI for each ID.
//
// set_alternateViewMediaTypes([in] BSTR *commaSeparatedMediaType)
// ---------------------------------------------------------------------------------------------------=
// For style property retrieval on nsISimpleDOMNode elements, 
// set the additional alternate media types that properties are available for.
// [in] BSTR *commaSeparatedMediaTypes is a comma separate list, for example "aural, braille".
// The alternate media properties are requested with nsISimpleDOMNode::get_computedStyle.
// Note: setting this value on a document will increase memory overhead, and may create a small delay.
//
// W3C media Types:
// * all:        Suitable for all devices. 
// * aural:      Intended for speech synthesizers. See the section on aural style sheets for details. 
// * braille:    Intended for braille tactile feedback devices. 
// * embossed:   Intended for paged braille printers. 
// * handheld:   Intended for handheld devices - typically small screen, monochrome, limited bandwidth. 
// * print:      Intended for paged, opaque material and for documents viewed on screen in print preview mode. Please consult the section on paged media for information about formatting issues that are specific to paged media. 
// * projection: Intended for projected presentations, for example projectors or print to transparencies. Please consult the section on paged media for information about formatting issues that are specific to paged media. 
// * screen:     Intended primarily for color computer screens. 
// * tty:        intended for media using a fixed-pitch character grid, such as teletypes, terminals, or portable devices with limited display capabilities. Authors should not use pixel units with the tty media type. 
// * tv:         Intended for television-type devices - low resolution, color, limited-scrollability screens, sound
// * See latest W3C CSS specs for complete list of media types
//
//
///////////////////////////////////////////////////////////////////////////////////////////////////////

#else   /* C style interface */

    typedef struct ISimpleDOMDocumentVtbl
    {
        BEGIN_INTERFACE
        
        HRESULT ( STDMETHODCALLTYPE *QueryInterface )( 
            ISimpleDOMDocument * This,
            /* [in] */ REFIID riid,
            /* [iid_is][out] */ 
            __RPC__deref_out  void **ppvObject);
        
        ULONG ( STDMETHODCALLTYPE *AddRef )( 
            ISimpleDOMDocument * This);
        
        ULONG ( STDMETHODCALLTYPE *Release )( 
            ISimpleDOMDocument * This);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_URL )( 
            ISimpleDOMDocument * This,
            /* [retval][out] */ BSTR *url);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_title )( 
            ISimpleDOMDocument * This,
            /* [retval][out] */ BSTR *title);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_mimeType )( 
            ISimpleDOMDocument * This,
            /* [retval][out] */ BSTR *mimeType);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_docType )( 
            ISimpleDOMDocument * This,
            /* [retval][out] */ BSTR *docType);
        
        /* [propget] */ HRESULT ( STDMETHODCALLTYPE *get_nameSpaceURIForID )( 
            ISimpleDOMDocument * This,
            /* [in] */ short nameSpaceID,
            /* [retval][out] */ BSTR *nameSpaceURI);
        
        /* [propput] */ HRESULT ( STDMETHODCALLTYPE *put_alternateViewMediaTypes )( 
            ISimpleDOMDocument * This,
            /* [in] */ BSTR *commaSeparatedMediaTypes);
        
        END_INTERFACE
    } ISimpleDOMDocumentVtbl;

AccessibleMarshal.dll which is installed with Firefox (and probably Thunderbird) makes ISimpleDOM interfaces known to Windows. For Chrome you have to enable accessibility (chrome://accessibility/, check that global accessibility mode is set to on) to use ISimpleDOM.

I have tested ISimpleDOM with both Firefox and Chrome. The official versions of these programs are running in 32 bit mode. The example script should also be run in 32 bit mode.

The example recursively walks through all the objects of a top window. For each object it creates IAccessible and ISimpleDOMNode interfaces. For each ISimpleDOMNode interface it tries to get pointers for the other ISimpleDOM interfaces. If it succeeds various information is extracted.

Press Ctrl+w when the mouse is over a window. Results are shown with _ArrayDisplay. The array can contain several thousands rows. Give it a few seconds to fill the array.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"
#include "ISimpleDOM.au3"

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Global $aInfo[10000], $iInfo = 0

MainFunc()


Func MainFunc()

  HotKeySet( "^w", "GetWindowInfo" )
  HotKeySet( "{ESC}", "Close" )

  While Sleep(100)
  WEnd

EndFunc


Func GetWindowInfo()
  Local $aMousePos = MouseGetPos(), $pObject, $tVarChild, $vt, $err = 0
  If AccessibleObjectFromPoint( $aMousePos[0], $aMousePos[1], $pObject, $tVarChild ) = $S_OK Then
    $vt = BitAND( DllStructGetData( $tVarChild, 1, 1 ), 0xFFFF )
    If $vt = $VT_I4 Then
      Local $oObject, $hWnd, $sName = ""
      If WindowFromAccessibleObject( $pObject, $hWnd ) = $S_OK Then

        AccessibleObjectFromWindow( $hWnd, $OBJID_CLIENT, $tIID_IAccessible, $pObject )
        $oObject = ObjCreateInterface( $pObject, $sIID_IAccessible, $dtagIAccessible )

        ; Get parent objects to top window
        Local $pParent, $pPrev, $oPrev, $iRole
        Do
          $pPrev = $pParent
          $oPrev = $oObject
          $oObject.get_accParent( $pParent )
          $oObject = ObjCreateInterface( $pParent, $sIID_IAccessible, $dtagIAccessible )
          $oObject.get_accName( $CHILDID_SELF, $sName )
        Until String( $sName ) = "Desktop"

        ; Info for top window (previous to desktop)
        PrintElementInfoToArray( $pPrev, $oPrev, $CHILDID_SELF, "" )

        ; Info for child objects of the top window
        If $oPrev.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
          WindowFromAccessibleObject( $pPrev, $hWnd )
          PrintWindowObjects( $hWnd )
        Else
          $err = 1
        EndIf

      Else
        $err = 1
      EndIf
    Else
      $err = 1
    EndIf
  Else
    $err = 1
  EndIf
  If $err Then _
    MsgBox( 0, "ISimpleDOM interfaces", _
               "AccessibleObjectFromPoint failed." & @CRLF & _
               "Try another point e.g. the window title bar." )
EndFunc


Func PrintWindowObjects( $hWindow )

  ; Object from window
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return

  ; Get window objects
  WalkTreeWithAccessibleChildren( $pWindow, 0, 8 )

  ; Print window objects
  ReDim $aInfo[$iInfo]
  _ArrayDisplay( $aInfo, "ISimpleDOM interfaces" )
  ReDim $aInfo[10000]
  $iInfo = 0

EndFunc


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return
  $oAcc.AddRef()

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "
  Next

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  Local $vt, $pChildObj, $oChildObj
  Local $pService, $oService, $pISimpleDOMNode, $oISimpleDOMNode, $pISimpleDOM, $oISimpleDOM

  ; $oISimpleDOMNode.nodeInfo
  Local $sNodeName, $iNameSpaceID, $sNodeValue, $iNumChildren, $iUniqueID, $iNodeType, $sInnerHTML

  ; $oISimpleDOMNode.attributes
  Local $iMaxAttribs = 100, $iNumAttribs
  Local $tAttribNames  = DllStructCreate( "ptr[" & $iMaxAttribs & "]" ),   $sAttribName
  Local $tNameSpaceIDs = DllStructCreate( "short[" & $iMaxAttribs & "]" ), $iNameSpaceID
  Local $tAttribValues = DllStructCreate( "ptr[" & $iMaxAttribs & "]" ),   $sAttribValue

  ; $oISimpleDOMNode.computedStyle
  Local $iMaxStyleProperties = 100, $iNumStyleProperties
  Local $tStyleProperties = DllStructCreate( "ptr[" & $iMaxStyleProperties & "]" ), $sStyleProperty
  Local $tStyleValues     = DllStructCreate( "ptr[" & $iMaxStyleProperties & "]" ), $sStyleValue

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object

      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )

      If IsObj( $oChildObj ) Then

        ; Try to get an ISimpleDOMNode interface pointer for the child object
        If $oChildObj.QueryInterface( DllStructGetPtr( $tIID_IServiceProvider ), $pService ) = $S_OK Then
          $oService = ObjCreateInterface( $pService, $sIID_IServiceProvider, $dtagIServiceProvider )
          If $oService.QueryService( $tIID_ISimpleDOMNode, $tIID_ISimpleDOMNode, $pISimpleDOMNode ) = $S_OK Then
            $aInfo[$iInfo] = $sIndent & "$pISimpleDOMNode = " & Ptr( $pISimpleDOMNode )
            $iInfo += 1
            $oISimpleDOMNode = ObjCreateInterface( $pISimpleDOMNode, $sIID_ISimpleDOMNode, $dtagISimpleDOMNode )
            If IsObj( $oISimpleDOMNode ) Then
              $oISimpleDOMNode.nodeInfo( $sNodeName, $iNameSpaceID, $sNodeValue, $iNumChildren, $iUniqueID, $iNodeType )
              $aInfo[$iInfo+0] = $sIndent & "$sNodeName       = " & $sNodeName
              $aInfo[$iInfo+1] = $sIndent & "$iNameSpaceID    = " & $iNameSpaceID
              $aInfo[$iInfo+2] = $sIndent & "$sNodeValue      = " & $sNodeValue
              $aInfo[$iInfo+3] = $sIndent & "$iNumChildren    = " & $iNumChildren
              $aInfo[$iInfo+4] = $sIndent & "$iUniqueID       = " & $iUniqueID
              $aInfo[$iInfo+5] = $sIndent & "$iNodeType       = " & $iNodeType & " (" & GetNodeType( $iNodeType ) & ")"
              $iInfo += 6

              ; $oISimpleDOMNode.attributes
              $oISimpleDOMNode.attributes( $iMaxAttribs, $tAttribNames, $tNameSpaceIDs, $tAttribValues, $iNumAttribs )
              $aInfo[$iInfo] = $sIndent & "$iNumAttribs     = " & $iNumAttribs
              $iInfo += 1
              For $j = 1 To $iNumAttribs
                $sAttribName  = SysReadString( DllStructGetData( $tAttribNames, 1, $j ) )
                $sAttribValue = SysReadString( DllStructGetData( $tAttribValues, 1, $j ) )
                $iNameSpaceID = DllStructGetData( $tNameSpaceIDs, 1, $j )
                $aInfo[$iInfo+0] = "    " & $sIndent & "$iAttrib      = " & $j
                $aInfo[$iInfo+1] = "    " & $sIndent & "$sAttribName  = " & $sAttribName
                $aInfo[$iInfo+2] = "    " & $sIndent & "$sAttribValue = " & $sAttribValue
                $aInfo[$iInfo+3] = "    " & $sIndent & "$iNameSpaceID = " & $iNameSpaceID
                $iInfo += 4
                ; Free memory allocated by BSTRs
                SysFreeString( DllStructGetData( $tAttribNames, 1, $j ) )
                SysFreeString( DllStructGetData( $tAttribValues, 1, $j ) )
              Next

              ; $oISimpleDOMNode.computedStyle
              $oISimpleDOMNode.computedStyle( $iMaxStyleProperties, 0, $tStyleProperties, $tStyleValues, $iNumStyleProperties )
              $aInfo[$iInfo] = $sIndent & "$iNumStyleProps  = " & $iNumStyleProperties
              $iInfo += 1
              For $j = 1 To $iNumStyleProperties
                $sStyleProperty = SysReadString( DllStructGetData( $tStyleProperties, 1, $j ) )
                $sStyleValue    = SysReadString( DllStructGetData( $tStyleValues, 1, $j ) )
                $aInfo[$iInfo+0] = "    " & $sIndent & "$iStyle       = " & $j
                $aInfo[$iInfo+1] = "    " & $sIndent & "$sStyleProp   = " & $sStyleProperty
                $aInfo[$iInfo+2] = "    " & $sIndent & "$sStyleValue  = " & $sStyleValue
                $iInfo += 3
                ; Free memory allocated by BSTRs
                SysFreeString( DllStructGetData( $tStyleProperties, 1, $j ) )
                SysFreeString( DllStructGetData( $tStyleValues, 1, $j ) )
              Next

              ;$oISimpleDOMNode.innerHTML( $sInnerHTML )
              ;If String( $sInnerHTML ) <> "0" Then
              ;  $aInfo[$iInfo] = $sIndent & "$sInnerHTML      = " & $sInnerHTML
              ;  $iInfo += 1
              ;EndIf

              If $iNodeType = 3 Then ; $NODETYPE_TEXT
                ; Try to get an ISimpleDOMText interface pointer with $oISimpleDOMNode.QueryInterface
                If $oISimpleDOMNode.QueryInterface( DllStructGetPtr( $tIID_ISimpleDOMText ), $pISimpleDOM ) = $S_OK Then
                  $aInfo[$iInfo] = $sIndent & "$pISimpleDOMText = " & Ptr( $pISimpleDOM )
                  $iInfo += 1
                  $oISimpleDOM = ObjCreateInterface( $pISimpleDOM, $sIID_ISimpleDOMText, $dtagISimpleDOMText )
                  If IsObj( $oISimpleDOM ) Then
                    Local $sDomText
                    $oISimpleDOM.domText( $sDomText )
                    $aInfo[$iInfo] = $sIndent & "$sDomText        = " & $sDomText
                    $iInfo += 1
                  EndIf
                EndIf

              ElseIf $iNodeType = 9 Then ; $NODETYPE_DOCUMENT
                ; Try to get an ISimpleDOMDocument interface pointer with $oISimpleDOMNode.QueryInterface
                If $oISimpleDOMNode.QueryInterface( DllStructGetPtr( $tIID_ISimpleDOMDocument ), $pISimpleDOM ) = $S_OK Then
                  $aInfo[$iInfo] = $sIndent & "$pISimpleDOMDocument = " & Ptr( $pISimpleDOM )
                  $iInfo += 1
                  $oISimpleDOM = ObjCreateInterface( $pISimpleDOM, $sIID_ISimpleDOMDocument, $dtagISimpleDOMDocument )
                  If IsObj( $oISimpleDOM ) Then
                    Local $sURL, $sTitle, $sMimeType, $sDocType
                    $oISimpleDOM.URL( $sURL )
                    $oISimpleDOM.title( $sTitle )
                    $oISimpleDOM.mimeType( $sMimeType )
                    $oISimpleDOM.docType( $sDocType )
                    $aInfo[$iInfo+0] = $sIndent & "$sURL                = " & $sURL
                    $aInfo[$iInfo+1] = $sIndent & "$sTitle              = " & $sTitle
                    $aInfo[$iInfo+2] = $sIndent & "$sMimeType           = " & $sMimeType
                    $aInfo[$iInfo+3] = $sIndent & "$sDocType            = " & $sDocType
                    $iInfo += 4
                  EndIf
                EndIf
              EndIf

            EndIf
          EndIf
        EndIf

        PrintElementInfoToArray( $pChildObj, $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )

      EndIf

    EndIf

  Next

EndFunc


Func PrintElementInfoToArray( $pElement, $oElement, $iChild, $sIndent )
  Local $hWnd, $sName, $iRole, $sRole, $iRoleLen
  Local $iState, $sState, $iStateLen
  Local $sValue, $x, $y, $w, $h
  If $iChild <> $CHILDID_SELF Then
    $aInfo[$iInfo] = $sIndent & "$iChildElem = " & $iChild
    $iInfo += 1
  EndIf
  If $iChild = $CHILDID_SELF And $oElement.get_accRole( $CHILDID_SELF, $iRole ) = $S_OK And $iRole = $ROLE_SYSTEM_WINDOW Then
    WindowFromAccessibleObject( $pElement, $hWnd )
    $aInfo[$iInfo] = $sIndent & "$hWnd   = " & $hWnd
    $iInfo += 1
  EndIf
  $oElement.get_accName( $iChild, $sName )
  $aInfo[$iInfo] = $sIndent & "$sName  = " & $sName
  $iInfo += 1
  If $oElement.get_accRole( $iChild, $iRole ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iRole  = 0x" & Hex( $iRole )
    $iInfo += 1
    $iRoleLen = GetRoleText( $iRole, 0, 0 ) + 1
    $sRole = DllStructCreate( "wchar[" & $iRoleLen & "]" )
    GetRoleText( $iRole, DllStructGetPtr( $sRole ), $iRoleLen )
    $aInfo[$iInfo] = $sIndent & "$sRole  = " & DllStructGetData( $sRole, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accState( $iChild, $iState ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$iState = 0x" & Hex( $iState )
    $iInfo += 1
    $iStateLen = GetStateText( $iState, 0, 0 ) + 1
    $sState = DllStructCreate( "wchar[" & $iStateLen & "]" )
    GetStateText( $iState, DllStructGetPtr( $sState ), $iStateLen )
    $aInfo[$iInfo] = $sIndent & "$sState = " & DllStructGetData( $sState, 1 )
    $iInfo += 1
  EndIf
  If $oElement.get_accValue( $iChild, $sValue ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$sValue = " & $sValue
    $iInfo += 1
  EndIf
  IF $oElement.accLocation( $x, $y, $w, $h, $iChild ) = $S_OK Then
    $aInfo[$iInfo] = $sIndent & "$x, $y, $w, $h = " & $x & ", " & $y & ", " & $w & ", " & $h
    $iInfo += 1
  EndIf
  $aInfo[$iInfo] = ""
  $iInfo += 1
EndFunc


Func Close()
  Exit
EndFunc

The textbox shows a small part of the output from the AutoIt home site (http://www.autoitscript.com/site/). The textbox contains info about the top window for the two browsers, the document object, and the AutoIt logo object in the upper left corner.

Mozilla Firefox
===============
$hWnd   = 0x001C04DE
$sName  = AutoItScript - AutoItScript Website - Mozilla Firefox
$iRole  = 0x00000009
$sRole  = window
$iState = 0x00160004
$sState = focusable
$x, $y, $w, $h = 212, 84, 1200, 937

$pISimpleDOMNode = 0x00BCA104
$sNodeName       = #document
$iNameSpaceID    = 0
$sNodeValue      = 
$iNumChildren    = 3
$iUniqueID       = -164824832
$iNodeType       = 9 ($NODETYPE_DOCUMENT)
$iNumAttribs     = 0
$iNumStyleProps  = 0
$pISimpleDOMDocument = 0x00B16C24
$sURL                = http://www.autoitscript.com/site/
$sTitle              = AutoItScript - AutoItScript Website
$sMimeType           = text/html
$sDocType            = -//W3C//DTD XHTML 1.0 Transitional//EN
$sName  = AutoItScript - AutoItScript Website
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 216, 197, 1175, 820

$pISimpleDOMNode = 0x02CF4064
$sNodeName       = IMG
$iNameSpaceID    = 3
$sNodeValue      = 
$iNumChildren    = 0
$iUniqueID       = -169194944
$iNodeType       = 1 ($NODETYPE_ELEMENT)
$iNumAttribs     = 5
    $iAttrib      = 1
    $sAttribName  = width
    $sAttribValue = 210
    $iNameSpaceID = 0
    $iAttrib      = 2
    $sAttribName  = height
    $sAttribValue = 72
    $iNameSpaceID = 0
    $iAttrib      = 3
    $sAttribName  = id
    $sAttribValue = logo
    $iNameSpaceID = 0
    $iAttrib      = 4
    $sAttribName  = alt
    $sAttribValue = AutoItScript
    $iNameSpaceID = 0
    $iAttrib      = 5
    $sAttribName  = src
    $sAttribValue = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
    $iNameSpaceID = 0
$iNumStyleProps  = 0
$sName  = AutoItScript
$iRole  = 0x00000028
$sRole  = graphic
$iState = 0x00C01000
$sState = traversed
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 324, 197, 210, 72


Google Chrome
=============
$hWnd   = 0x000F0494
$sName  = AutoItScript - AutoItScript Website - Google Chrome
$iRole  = 0x00000009
$sRole  = window
$iState = 0x00160004
$sState = focusable
$x, $y, $w, $h = 212, 84, 1200, 936

$pISimpleDOMNode = 0x00B636F4
$sNodeName       = #document
$iNameSpaceID    = 0
$sNodeValue      = http://www.autoitscript.com/site/
$iNumChildren    = 26
$iUniqueID       = -1045
$iNodeType       = 9 ($NODETYPE_DOCUMENT)
$iNumAttribs     = 0
$iNumStyleProps  = 0
$pISimpleDOMDocument = 0x00C5A364
$sURL                = http://www.autoitscript.com/site/
$sTitle              = AutoItScript - AutoItScript Website
$sMimeType           = text/html
$sDocType            = html
$sName  = AutoItScript - AutoItScript Website
$iRole  = 0x0000000F
$sRole  = document
$iState = 0x00100044
$sState = focusable
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 217, 185, 1175, 1112

$pISimpleDOMNode = 0x00B9C9EC
$sNodeName       = a
$iNameSpaceID    = 0
$sNodeValue      = http://www.autoitscript.com/site/
$iNumChildren    = 1
$iUniqueID       = -1047
$iNodeType       = 1 ($NODETYPE_ELEMENT)
$iNumAttribs     = 1
    $iAttrib      = 1
    $sAttribName  = href
    $sAttribValue = http://www.autoitscript.com/site/
    $iNameSpaceID = 0
$iNumStyleProps  = 1
    $iStyle       = 1
    $sStyleProp   = display
    $sStyleValue  = inline
$sName  = AutoItScript
$iRole  = 0x0000001E
$sRole  = link
$iState = 0x00D10000
$sState = traversed
$sValue = http://www.autoitscript.com/site/
$x, $y, $w, $h = 324, 185, 211, 72

    $pISimpleDOMNode = 0x00BE352C
    $sNodeName       = img
    $iNameSpaceID    = 0
    $sNodeValue      = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
    $iNumChildren    = 0
    $iUniqueID       = -1080
    $iNodeType       = 1 ($NODETYPE_ELEMENT)
    $iNumAttribs     = 5
        $iAttrib      = 1
        $sAttribName  = src
        $sAttribValue = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
        $iNameSpaceID = 0
        $iAttrib      = 2
        $sAttribName  = width
        $sAttribValue = 210
        $iNameSpaceID = 0
        $iAttrib      = 3
        $sAttribName  = height
        $sAttribValue = 72
        $iNameSpaceID = 0
        $iAttrib      = 4
        $sAttribName  = alt
        $sAttribValue = AutoItScript
        $iNameSpaceID = 0
        $iAttrib      = 5
        $sAttribName  = id
        $sAttribValue = logo
        $iNameSpaceID = 0
    $iNumStyleProps  = 1
        $iStyle       = 1
        $sStyleProp   = display
        $sStyleValue  = block
    $sName  = AutoItScript
    $iRole  = 0x00000028
    $sRole  = graphic
    $iState = 0x00400040
    $sState = linked
    $sValue = http://aut1.autoit-cdn.com/site/wp-content/themes/TheCorporation/images/logo@2x.png
    $x, $y, $w, $h = 325, 185, 211, 72

You need MSAccessibility.au3 and ISimpleDOM.au3.

You also need the new (smallest) version of CUIAutomation2.au3 in first post.

junkew · March 10, 2014

nice. Thats a lot of information to get thru the interfaces of iaccessible2 and isimpledom (on top of MSAA and UIAutomation).

Will take a while for me to integrate IUIAutomation, IA2 and ISimpledon into an abstraction search interface as done for IUIautomation

.....

Kolokolcev · March 12, 2014

Hi guys,

I have a very unpleasant application to test but fortunately accessible by means of UI Automation. The problem is that one button behaves differently according to the method to operate with it.

If I apply DoDefaultAction() or Invoke() method the button is clicked but the result is not as I expect. I mean that if I click it using mouse then the button works correctly.

I had a device connected to my PC and if I click this button using mouse connection opening happens, but if I click it using Invoke() "Connection error" always occurs.

So my question - is there any chance to perform other Invoke-like action with no mouse involvement?

Here is my part of code "Incorrect behavior"

; Click (DoDefaultAction) "DSP Calibration" button
  Local $pLegacyButton, $oLegacyButton
  $oButton.GetCurrentPattern($UIA_LegacyIAccessiblePatternId, $pLegacyButton)
  $oLegacyButton = ObjCreateInterface($pLegacyButton, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern)
  If Not IsObj($oLegacyButton) Then Return
  $oLegacyButton.DoDefaultAction()
  ConsoleWrite("""DSP Calibration"" button clicked" & @CRLF & @CRLF)

  ; Click (Toggle) "DSP Calibration" button ; Looks like not a Toggle button
  Local $pToggle, $oToggle
  $oButton.GetCurrentPattern($UIA_TogglePatternId, $pToggle)
  $oToggle = ObjCreateInterface($pToggle, $sIID_IUIAutomationTogglePattern, $dtagIUIAutomationTogglePattern)
  If Not IsObj($oToggle) Then Return
  $oToggle.Toggle()
  ConsoleWrite("""DSP Calibration"" button clicked" & @CRLF & @CRLF)

  ; Click (Invoke) "DSP Calibration" button
  Local $pInvoke, $oInvoke
  $oButton.GetCurrentPattern($UIA_InvokePatternId, $pInvoke)
  $oInvoke = ObjCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
  If Not IsObj($oInvoke) Then Return
  $oInvoke.Invoke()
  ConsoleWrite("""DSP Calibration"" button clicked" & @CRLF & @CRLF)

Here is "Correct behavior" but with mouse involvement

; Create RECT structure
  Local Const $tagRect = "STRUCT;long left;long top;long right;long bottom;ENDSTRUCT;"
  Local $tButtonRect = DllStructCreate($tagRect)
  If @error Then Return

  Local $hResult = $oButton.CurrentBoundingRectangle($tButtonRect)
  If $hResult <> $S_OK Then Return
  ConsoleWrite("hResult = " & $hResult)
  Local $iButtonCenterX = DllStructGetData($tButtonRect, "left") + (DllStructGetData($tButtonRect, "right") - DllStructGetData($tButtonRect, "left")) / 2
  Local $iButtonCenterY = DllStructGetData($tButtonRect, "top") + (DllStructGetData($tButtonRect, "bottom") - DllStructGetData($tButtonRect, "top")) / 2
  MouseClick("left", $iButtonCenterX, $iButtonCenterY)
  ConsoleWrite("""DSP Calibration"" button clicked" & @CRLF & @CRLF)

P.S. I'm not so good in all this COM stuff but is it OK that CurrentBoundingRectangle() method name is not the same as in IUIAutomationElement:: interface (get_CurrentBoundingRectangle). I mean was it a conscious intention to get rid of get_ prefix?

Edited March 12, 2014 by Kolokolcev

LarsJ · March 13, 2014

Kolokolcev, You seem to have tried all possible ways to click that button. The reason why it doesn't work is probably, that the Invoke method isn't coded properly by the provider of the application. I don't think there is much to do about it.

I don't think there are other ways to perform an Invoke-like action. And it probably wouldn't help, because they would all end up in this bad coded Invoke method.

You seem to have a pretty good grasp of the COM stuff. junkew has deliberately removed all get_ prefixes from read only properties (and I agree). get_ and put_ prefixes are used for read/write properties. The method names are only used in AutoIt, so the names doesn't matter. When the method is called in the DLL, it's called by number. The number in the interface description string (including IUnknown) which matches the number in the v-table.

junkew, I think that IAccessible2 and ISimpleDOM are only on top of MSAA. It would be nice with some kind of integration of all these interfaces.

Regards Lars.

Kolokolcev · March 14, 2014

Thank you, LarsJ.

My colleague advised me to update the application to the latest version. Maybe it will solve the problem. Currently it has a bunch of different bugs that I found using only Tab button.

Concerning method names - I just try to get all information needed from msdn so I copy/paste method names into search string trying to find the exact match. That's why I was a little bit confused

JCMCNL86 · March 21, 2014

Hi all,

First off, thank you for all of the work put into creating this.

I have run into a problem and was hoping someone could point out where I may have gone wrong.

Back in December 2013, I started using this UI Automation in AutoIt to create a project that was eventually shelved. This project automated a Silverlight application (in IE) and was working great. However, I have since reinstalled Windows and believe that I have missed something as I can no longer see/access the Silverlight elements.

When using Visual UIA Verify, I can drilldown through the Desktop and IE panes but it stops at "silverlightapp" and I am unable access any of the information that I was able to a few months ago.

1.) Does anyone have any suggestions for what I may have forgotten to install? I have tried everything I could think of but will try anything at this point.

2.) Would it be possible that the developers of the Silverlight application somehow locked down the application to prevent the use of UI Automation? Not sure if this is possible but wanted to confirm.

Thanks in advance for any help, please let me know if any further information is needed.

LarsJ · March 23, 2014

JCMCNL86, I can identify all controls on http://demos.telerik.com/silverlight/#GridView/FirstLook with the Spy script in first post and Visual UIA Verify.

Can you test if you can identify the controls. If you can, there is probably nothing wrong with your installation.

Developers must add code to implement UI Automation. If that code is removed (may be just an option to enable/disable UI Automation) it will prevent the use of UI Automation. But usually UI Automation is considered as an additional and valuable feature, so it is not likely it's removed, unless there are special reasons.

Regards Lars.

IUIAutomation MS framework automate chrome, FF, IE, ....

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

junkew

junkew

junkew

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content