MSAA Active accessibility

junkew · May 17, 2009

Reading thru some of the topics I am wondering if its possible to get this MSAA to work within AutoIt

http://www.codeproject.com/KB/winsdk/XMSAALib.aspx

http://www.codeproject.com/KB/winsdk/msaa_intro.aspx

http://www.accessinteropalliance.org/docs/...ccessibleEx.doc

http://www.autoitscript.com/forum/index.ph...mp;hl=ole32.dll

http://www.autoitscript.com/forum/index.ph...90831&st=15

All the pieces seems to be there all I could not determine so far is the Glue in between to write to get it working within AutoIT

Any further suggestions? Topic seems to have been discussed on the forum several times or is it really a NOT POSSIBLE answer.

I studied a little further in

http://www.codeproject.com/KB/COM/COM_from_scratch_1.aspx

http://www.codeproject.com/KB/COM/COM_from_scratch_2.aspx

From that perspective it should be possible to get to the vTable and indirectly get the functions/methods of the object MSAA

added: 09 jan 2010: references with additional information

http://www.accessinteropalliance.org/docs/What_is_UIAutomation.doc

http://www.codeproject.com/KB/winsdk/msaa_intro.aspx

http://www.autoitscript.com/forum/index.php?showtopic=84532&st=0&p=606255&hl=vtable&fromsearch=1&#entry606255

added: 11 jan 2010: Probably better to go for UI automation API instead of MSAA

http://msdn.microsoft.com/en-us/library/ee684009(VS.85).aspx

http://uiautomationverify.codeplex.com/

Edited January 10, 2010 by junkew

junkew · May 17, 2009

First part of code

;http://www.autoitscript.com/forum/index.php?showtopic=87183
;http://www.autoitscript.com/forum/index.php?showtopic=34477&st=0&p=251658&#entry251658
;http://msdn.microsoft.com/en-us/library/dd318447(VS.85).aspx

#include <memory.au3>
#include "MemArray.au3"
#include "objbase.au3"

; Define the IAccessible GUID
$IID_IAccessible = _GUID("{618736E0-3C3D-11CF-810C-00AA00389B71}")
$IID_IDispatch = _GUID("{00020400-0000-0000-c000-000000000046}")

Global Const $OBJID_WINDOW = 0x00000000

; Open Calculator and obtain handle for the '2' button
Run("calc.exe")
WinWait("[Class:SciCalc]")
Dim $hWnd = ControlGetHandle("Rekenmachine", "", "[TEXT:2]")

;Coinitialize 
_ObjCoInitialize()

; Call to AccessibleObjectFromWindow in the hopes of receiving a valid IDispatch object

Dim $result = DllCall("oleacc.dll", "long", "AccessibleObjectFromWindow", "hwnd", $hWnd, "long", $OBJID_WINDOW, "ptr", DllStructGetPtr($IID_IAccessible), "ptr*", 0)
ConsoleWrite(@error & @CRLF)

; Post the test result to the console
; breaking on idispatch*
;-2147024809| 0x000B0874| 0| 0x01148118|                idispatch
;-2147024809| 0x00040C76| 0| 0x01148270| 0x00000000     ptr
;0| 0x000A084A| 0| 0x01148090| 0x008AB470                 ptr*
;-2147024809| 0x00050C4A| 0| 0x01148118| 0      long

Dim $actualResult = $result[0] & "| " & $result[1] & "| " & $result[2] & "| " & $result[3] & "| " & $result[4]
ConsoleWrite($actualResult & @CRLF)

; Cleanup
WinClose("[Class:SciCalc]")

_ObjCoUninitialize()

junkew · January 5, 2010

Some hope that combined with this article MSAA can work

http://www.autoitscript.com/forum/index.php?showtopic=107678&st=0&gopid=760062&

Any advice? Will take me a while to understand article above but I see some possibilities.

MSAA

The full name of MSAA is Microsoft Active Accessibility. It is similar as distributed COM (DCOM) but not the same. It works like this. UI application exposes an interface, so that another application can use the interface to control the target. The initial goal of MSAA is for incapable people. For example, a blind person cannot see window, but she can connect a USB screen reader. The reader obtains application’s information through the MSAA interface, and sends the information to the blind person in a suitable way.

UI Automation leverages this technology. The interface exposed by MSAA is called IAccessible. The communication between UI and test client is:

· Test client invokes Windows API like AccessibleObjectFromWindow and passes in target UI’s HWND

· AccessibleObjectFromWindow API sends WM_GETOBJECT message to the Window

· UI application should implement the IAccessible interface. When receiving the WM_GETOBJECT message, it creates the IAccessible object and returns it by calling LresultFromObject API.

· Test client gets IAccessible and starts to use it.

The key functions in IAccessible interface:

· IAccessible::get_accChild/ IAccessible::get_accParent The two functions provide UI tree navigation support. Test client uses them to identify UI element.

· IAccessible::accLocation/I Accessible::accHitTest Test client uses them to read and check element location.

· IAccessible::accName/ I Accessible:: accSelect Test client uses them to read element’s information like Name, and do some operation like select items.

· IAccessible::accValue The UI developer can customize the value property. For example, a polyline control developer may expose the coordinate array in this property.

In real situation, tester usually combines MSAA and Win32 API. For reading UI element property and situation simulating, it uses Win32. In other side, it uses MSAA to compensate Win32, for example:

MSAA provides get_accChild method. It allows the UI tree obtaining in test can be different from the real Win32 control tree. It benefits self-drawn window. The developer can define logic tree with such method. In Excel example, each Cell can be exposed separately.

It allows the control developer to provide flexible implementation. Previously we mentioned returning coordinate array for polyline control. Another example is .NET WinForm. Microsoft provides default implementation of IAccessible so that the HWND awkward maintenance detail is well handled.

There are several tools targeting MSAA. AccExplorer works like Spy++, allowing UI tree navigation and property checking.

junkew · January 7, 2010

Its hard to get it working but based on the articles I have the feeling it can work in AutoIT

article how it was done in autohotkey : http://www.autohotkey.com/forum/viewtopic.php?t=24234

My trialcode

1. Get the object from HWND : STDAPI AccessibleObjectFromWindow(__in HWND hwnd, __in DWORD dwObjectID, __in REFIID riid, __out void **ppvObject);

2. Get the HWND from the returned object: STDAPI WindowFromAccessibleObject( __in IAccessible *pacc, __out HWND *phwnd);

If thats working the other parts should become easy

(and probably need the function tricks Monoceres shows in http://www.autoitscript.com/forum/index.....php?showtopic=107678&st=0&gopid=760062&)

ad 1 seems to be working as it returns 0 after the funtion call

ad 2 doing the opposite gives an incorrect parameter error in the DLL Call

;Some functions copied just to get the example code working
; Starts COM
Func CoInitialize()
    $hOle32=DllOpen("Ole32.dll")
    DllCall($hOle32,"ulong","CoInitialize","ptr",0)
EndFunc

; Initalize COM
CoInitialize()

; Prog@ndy
Func _GUID($IID)
    $IID = StringRegExpReplace($IID,"([}{])","")
    $IID = StringSplit($IID,"-")
    Local $_GUID =  "DWORD Data1;  ushort Data2;  ushort Data3;  BYTE Data4[8];"
    Local $GUID = DllStructCreate($_GUID)
    If $IID[0] = 5 Then $IID[4] &= $IID[5]
    If $IID[0] > 5 Or $IID[0] < 4 Then Return SetError(1,0,0)
    DllStructSetData($GUID,1,Dec($IID[1]))
    DllStructSetData($GUID,2,Dec($IID[2]))
    DllStructSetData($GUID,3,Dec($IID[3]))
    DllStructSetData($GUID,4,Binary("0x"&$IID[4]))
    Return $GUID
EndFunc

$IID_IAccessible = _GUID("{618736E0-3C3D-11CF-810C-00AA00389B71}") ; Define the IAccessible GUID
$IID_IDispatch   = _GUID("{00020400-0000-0000-c000-000000000046}") ; Define the IDispatch GUID

Global Const $OBJID_WINDOW = 0x00000000

; Open Calculator and obtain handle for the '2' button
; Should be translated to localized version of calculator 
Run("calc.exe")
WinWait("[Class:SciCalc]")
; Dim $hWnd = ControlGetHandle("Rekenmachine", "", "[TEXT:2]")
Dim $hWnd = ControlGetHandle("Calculator", "", "[TEXT:2]")            ; Should be translated to whatever titlebar text localized calculator has.

Dim $result = DllCall("oleacc.dll", "long", "AccessibleObjectFromWindow", "hwnd", $hWnd, "dword", $OBJID_WINDOW, "ptr", DllStructGetPtr($IID_IAccessible), "ptr*", 0)

; Post the test result to the console based on results by changing last parameter only PTR* returns 0 success
; exception idispatch*
; -2147024809| 0x00010A0A| 0| 0x019FAD78| |  idispatch
; -2147024809| 0x00010AB0| 0| 0x019FAD68| 0x00000000|   ptr
; exception when using the returned object 0| 0x00020882| 0| 0x019FAC70| 0x008DF100|        ptr*
; -2147024809| 0x00020992| 0| 0x019FAC70| 0| 

Dim $actualResult = $result[0] & "| " & $result[1] & "| " & $result[2] & "| " & $result[3] & "| " & $result[4] & "| " 
ConsoleWrite("Errorcode : " & @error  & ":"  & $actualResult & @CRLF)

;And now try to do the opposite getting back the HWND from the object
dim $hwndReturned=0
dim $pAccObj=$result[4]
;~ dim $pAccObj=ConvertPtrToIDispatch($result[4])            ;Will this work with Monoceres article?

;- Tried different variable types but I don't get a returnvalue of 0
Dim $result2=DllCall("oleacc.dll", "long", "WindowFromAccessibleObject", "UINT", $pAccObj, "UINT_PTR", $hWndReturned)
Dim $actualResult2= $result2[0] & "| " & $result2[1] & "| " & $result2[2] & "| " 

ConsoleWrite("Errorcode : " & @error & ":" & $actualresult2 & " hWND in: " & $hWND & " hWND out: " & $hwndReturned & @CRLF)
exit

monoceres · January 7, 2010

Check comments!

;Some functions copied just to get the example code working
; Starts COM
Func CoInitialize()
    $hOle32=DllOpen("Ole32.dll")
    DllCall($hOle32,"ulong","CoInitialize","ptr",0)
EndFunc

; Initalize COM
CoInitialize()

; Prog@ndy
Func _GUID($IID)
    $IID = StringRegExpReplace($IID,"([}{])","")
    $IID = StringSplit($IID,"-")
    Local $_GUID = "DWORD Data1; ushort Data2; ushort Data3; BYTE Data4[8];"
    Local $GUID = DllStructCreate($_GUID)
    If $IID[0] = 5 Then $IID[4] &= $IID[5]
    If $IID[0] > 5 Or $IID[0] < 4 Then Return SetError(1,0,0)
    DllStructSetData($GUID,1,Dec($IID[1]))
    DllStructSetData($GUID,2,Dec($IID[2]))
    DllStructSetData($GUID,3,Dec($IID[3]))
    DllStructSetData($GUID,4,Binary("0x"&$IID[4]))
    Return $GUID
EndFunc

$IID_IAccessible = _GUID("{618736E0-3C3D-11CF-810C-00AA00389B71}") ; Define the IAccessible GUID
$IID_IDispatch = _GUID("{00020400-0000-0000-c000-000000000046}") ; Define the IDispatch GUID

Global Const $OBJID_WINDOW = 0x00000000

; Open Calculator and obtain handle for the '2' button
; Should be translated to localized version of calculator
Run("calc.exe")
WinWait("Calculator")
; Dim $hWnd = ControlGetHandle("Rekenmachine", "", "[TEXT:2]")
; only worked with a handle to a window for me, but meh
Dim $hWnd = WinGetHandle("Calculator")  ; Should be translated to whatever titlebar text localized calculator has.

Dim $result = DllCall("oleacc.dll", "long", "AccessibleObjectFromWindow", "hwnd", $hWnd, "dword", $OBJID_WINDOW, "ptr", DllStructGetPtr($IID_IAccessible), "ptr*", 0)

; Post the test result to the console based on results by changing last parameter only PTR* returns 0 success
; exception idispatch*
; -2147024809| 0x00010A0A| 0| 0x019FAD78| | idispatch
; -2147024809| 0x00010AB0| 0| 0x019FAD68| 0x00000000| ptr
; exception when using the returned object 0| 0x00020882| 0| 0x019FAC70| 0x008DF100|    ptr*
; -2147024809| 0x00020992| 0| 0x019FAC70| 0|

Dim $actualResult = $result[0] & "| " & $result[1] & "| " & $result[2] & "| " & $result[3] & "| " & $result[4] & "| "
ConsoleWrite("Errorcode : " & @error & ":" & $actualResult & @CRLF)

;And now try to do the opposite getting back the HWND from the object
dim $hwndReturned=0
dim $pAccObj=$result[4]
;~ dim $pAccObj=ConvertPtrToIDispatch($result[4])   ;Will this work with Monoceres article?

;- Tried different variable types but I don't get a returnvalue of 0
; You're using deprecated type names here, me fix 
Dim $result2=DllCall("oleacc.dll", "long", "WindowFromAccessibleObject", "ptr", $pAccObj, "hwnd*", $hWndReturned)
Dim $actualResult2= $result2[0] & "| " & $result2[1] & "| " & $result2[2] & "| "
; Even the results of byref's are returned in array
$hwndReturned=$result2[2]
ConsoleWrite("Errorcode : " & @error & ":" & $actualresult2 & " hWND in: " & $hWND & " hWND out: " & $hwndReturned & @CRLF)
exit

junkew · January 7, 2010

thx. This will help me further and I will make a further working example on MSAA (assuming as those 2 functions now work all the other ones should be possible).

trancexx · January 7, 2010

You have one example here. More in that thread.

junkew · January 27, 2010

thanx for that example. (Can read it and understand the logic but most sure I could not have programmed that myself)

Will try to wrap it up and combine it with the IDispatch hooking thread from Monoceres to have an easy to use library making it possible to control a lot of controls that are not directly controllable from AutoIT functions itself.

junkew · February 7, 2010

Somebody able to help out on the AccessibleObjectFromPoint. I get an exception error or now when I use the x and y as separate parameters no exception error but not the result I expected

AccessibleObjectFromWindow is working fine based on the examples as given by searching for oleacc.dll in this forum.

;~ STDAPI AccessibleObjectFromPoint( __in POINT ptScreen, __out IAccessible **ppacc, __out VARIANT *pvarChild);

;~ local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "ptr", DllStructGetPtr($tPOINT), "idispatch*", 0, "ptr", 0)

;~ local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "long", DllStructGetPtr($tPOINT), "idispatch", 0, "ptr", 0)

local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "int", $x, "int", $y, "idispatch*", 0, "ptr", 0)

Func _AccessibleObjectFromPoint($x,$y)
    Local $tPoint=DllStructCreate($tagPOINT)
    dllstructsetdata($tPoint,1,$x)
    dllstructsetdata($tPoint,2,$y)
;~ STDAPI AccessibleObjectFromWindow(  __in   HWND hwnd,  __in   DWORD dwObjectID,  __in   REFIID riid,  __out  void **ppvObject);
;~ STDAPI AccessibleObjectFromPoint(  __in   POINT ptScreen,  __out  IAccessible **ppacc,  __out  VARIANT *pvarChild);
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "ptr", DllStructGetPtr($tPOINT), "idispatch*", 0, "ptr", 0)
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "long", DllStructGetPtr($tPOINT), "idispatch", 0, "ptr", 0)
    local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "int", $x, "int", $y, "idispatch*", 0, "ptr", 0)
    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

;-    Return ConvertPtrToIDispatch($aCall[3])
    Return $aCall[3]

EndFunc   ;==>_AccessibleObjectFromWindow

trancexx · February 7, 2010

IAccessible inherits from IDispatch so you can get available properties like this.

Opt("MustDeclareVars", 1)
Opt("GUIOnEventMode", 1)

Global $oMyError = ObjEvent("AutoIt.Error", "MyErrFunc")

Global Const $hOLEACC = DllOpen("oleacc.dll")

GUICreate("Blah")
GUISetOnEvent(-3, "_Quit")
GUISetState()

Global $aMousePos, $aCall

While 1

    $aMousePos = MouseGetPos()
    $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", _
            "int", $aMousePos[0], _
            "int", $aMousePos[1], _
            "idispatch*", 0, _
            "ptr*", 0)

    If @error Then ContinueLoop

    If IsObj($aCall[3]) Then
        ConsoleWrite("Title: " & $aCall[3] .accName & "       Child count: " & $aCall[3] .accChildCount & @CRLF)
    Else
        ConsoleWrite("! NOT OBJECT" & @CRLF)
    EndIf

    Sleep(300)

WEnd


Func MyErrFunc()
    ConsoleWrite("!COM Error!   " & "Number is: " & Hex($oMyError.number, 8) & "  description is: " & $oMyError.windescription & @CRLF)
EndFunc   ;==>MyErrFunc

Func _Quit()
    Exit
EndFunc   ;==>_Quit

But that's shit, when you think of it. And probably would cause AutoIt to crash eventually (over here at least).

junkew · February 8, 2010

The sample you give is similar to the thing I was doing and I also have this crashing.

The point I don't understand why its crashing. Somewhere leaking memory?

The nice thing on MSAA recognition is that I can recognize just a little bit more controls then AutoIT is recognizing.

ProgAndy · February 8, 2010

I get an error when moving over the Taskbarbutton of Notepad++. Somehow AccessibleObjectFromPoint is failing here with a crash Everything else does work, just the button for Notepad++ not. There's nothing we can do here, since the dll is failing.

Edit: No. the error is a missing VARIANT-structure.

@trancexx: using some of our AutoIt-Object-funcs will work.

Edit: OK, a working example:

Opt("MustDeclareVars", 1)
Opt("GUIOnEventMode", 1)

Global $oMyError = ObjEvent("AutoIt.Error", "MyErrFunc")

Global Const $hOLE32 = DllOpen("ole32.dll")
Global Const $hOLEACC = DllOpen("oleacc.dll")
Global Const $hOleAut = DllOpen("oleaut.dll")

Func __Au3Obj_ConvertPtrToIDispatch($pIDispatch)
    ; Author: monoceres
    Local $iSize = 4+(4*@AutoItX64)
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", "idispatch*", 0, "ptr*", $pIDispatch, "dword", $iSize)
    If @error Then Return SetError(1, 0, 0)
    Return $aCall[1]
EndFunc   ;==>__Au3Obj_ConvertPtrToIDispatch
Func __Au3Obj_VariantClear($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantClear", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantClear
Func __Au3Obj_VariantInit($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantInit", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantInit

GUICreate("Blah")
GUISetOnEvent(-3, "_Quit")
GUISetState()

Global $aMousePos, $aCall, $oIAccesible

Global Const $__Au3Obj_tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;ptr data; ptr"
Global $tVar = DllStructCreate($__Au3Obj_tagVARIANT)
Global $pVar = DllStructGetPtr($tVar)
__Au3Obj_VariantInit($pVar)

While 1

    $aMousePos = MouseGetPos()
    $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", _
            "int", $aMousePos[0], _
            "int", $aMousePos[1], _
            "ptr*", 0, _
            "ptr", $pVar)

    If @error Or $aCall[0]<>0 Then 
        ConsoleWrite("!DLLCall-Error or error on receiving IAccesible" & @CRLF)
        Sleep(500)
        ContinueLoop
    EndIf
    $oIAccesible = __Au3Obj_ConvertPtrToIDispatch($aCall[3]) ; create IDispatch when no error occured
    __Au3Obj_VariantClear($aCall[4])
    If IsObj($oIAccesible) Then
        ConsoleWrite("Title: " & $oIAccesible.accName & "       Child count: " & $oIAccesible.accChildCount & @CRLF)
    Else
        ConsoleWrite("! NOT OBJECT" & @CRLF)
    EndIf
    $oIAccesible = 0 ; free Object
    Sleep(500)

WEnd


Func MyErrFunc()
    ConsoleWrite("!COM Error!   " & "Number is: " & Hex($oMyError.number, 8) & "  description is: " & $oMyError.windescription & @CRLF)
EndFunc   ;==>MyErrFunc

Func _Quit()
    Exit
EndFunc   ;==>_Quit

Edited February 8, 2010 by ProgAndy

junkew · February 8, 2010

very nice. I will definitely watch the idispatch thread for the final functions.

I was already playing with the functions given in the thread on converting pointer to idispatch.

junkew · February 10, 2010

Any function to convert $pVar to the variant value

Then I am able to do

$oIAccesible.accName($pvar)

instead of

$oIAccesible.accName

The $oIAccesible.accName gives the name of the main object where I need the childobject which is in the variant where $pVar is pointing to

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <EditConstants.au3>
#include <StaticConstants.au3>

opt("MustDeclareVars", 1)
Opt("GUIOnEventMode", 1)

Global $oMyError = ObjEvent("AutoIt.Error", "MyErrFunc")

Global Const $hOLE32 = DllOpen("ole32.dll")
Global Const $hOLEACC = DllOpen("oleacc.dll")
Global Const $hOleAut = DllOpen("oleaut.dll")

Func __Au3Obj_ConvertPtrToIDispatch($pIDispatch)
    ; Author: monoceres
    Local $iSize = 4+(4*@AutoItX64)
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", "idispatch*", 0, "ptr*", $pIDispatch, "dword", $iSize)
    If @error Then Return SetError(1, 0, 0)
    Return $aCall[1]
EndFunc   ;==>__Au3Obj_ConvertPtrToIDispatch
Func __Au3Obj_VariantClear($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantClear", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantClear
Func __Au3Obj_VariantInit($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantInit", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantInit

Global $myEdit

GUICreate("Blah")
$myEdit = GUICtrlCreateEdit("First line" & @CRLF, 10, 10, 400, 300, $ES_AUTOVSCROLL + $WS_VSCROLL)
GUISetOnEvent(-3, "_Quit")
GUISetState()

Global $aMousePos, $aCall, $oIAccesible
Global $tInfo


Global Const $__Au3Obj_tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;ptr data; ptr"
Global $tVar = DllStructCreate($__Au3Obj_tagVARIANT)
Global $pVar = DllStructGetPtr($tVar)
__Au3Obj_VariantInit($pVar)

While 1

    $aMousePos = MouseGetPos()
    $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", _
            "int", $aMousePos[0], _
            "int", $aMousePos[1], _
            "ptr*", 0, _
            "ptr", $pVar)

    If @error Or $aCall[0]<>0 Then 
        ConsoleWrite("!DLLCall-Error or error on receiving IAccesible" & @CRLF)
        Sleep(500)
        ContinueLoop
    EndIf
    $oIAccesible = __Au3Obj_ConvertPtrToIDispatch($aCall[3]) ; create IDispatch when no error occured

    If IsObj($oIAccesible) Then
    ; ConsoleWrite("accChild: " & $oAccessibleObject.accChild($Param) & @CRLF) ;<-

    GUICtrlSetData($myedit, "")
    $tInfo=$pVar
    $tInfo = $tinfo & "accChildCount: " & $oIAccesible.accChildCount & @CRLF
    $tInfo = $tinfo & "accDefaultAction: " & $oIAccesible.accDefaultAction & @CRLF
    $tInfo = $tinfo & "accDescription: " & $oIAccesible.accDescription & @CRLF
    $tInfo = $tinfo & "accFocus: " & $oIAccesible.accFocus & @CRLF
    $tInfo = $tinfo & "accHelp: " & $oIAccesible.accHelp & @CRLF
;~     $tInfo = $tinfo & "accHelpTopic: " & $oIAccesible.accHelpTopic($Param) & @CRLF
    $tInfo = $tinfo & "accKeyboardShortcut: " & $oIAccesible.accKeyboardShortcut & @CRLF
    
    $tInfo = $tinfo & "accName: " & $oIAccesible.accName($pvar) & @CRLF
    
    $tInfo = $tinfo & "accParent: " & $oIAccesible.accParent & @CRLF
    $tInfo = $tinfo & "accRole: " & $oIAccesible.accRole & @CRLF
    $tInfo = $tinfo & "accSelection: " & $oIAccesible.accSelection & @CRLF
    $tInfo = $tinfo & "accState: " & $oIAccesible.accState & @CRLF
    $tInfo = $tinfo & "accValue: " & $oIAccesible.accValue & @CRLF
    dim $pX,$pY,$cX,$cY,$v
    $oIAccesible.accLocation($px,$py,$cx,$cy,0)
    
    $tInfo = $tinfo & "accLocation: " & $px & $py & $cx & $cy & $v & @CRLF
    GUICtrlSetData($myedit, $tInfo)
    
    Else
        ConsoleWrite("! NOT OBJECT" & @CRLF)
    EndIf

    __Au3Obj_VariantClear($aCall[4])
    $oIAccesible = 0 ; free Object
    Sleep(500)

WEnd


Func MyErrFunc()
    ConsoleWrite("!COM Error!   " & "Number is: " & Hex($oMyError.number, 8) & "  description is: " & $oMyError.windescription & @CRLF)
EndFunc   ;==>MyErrFunc

Exit

ProgAndy · February 11, 2010

I don't understand you problem. To set the value of a property use $obj.property = "...", to get the value use $value = $obj.property. There is no need to use a parameter.

$oChild = $oAccesible.accChild

or when using the function:

$childid = 123

$oChild = $oAccessible.get_accChild($childid)

; All parameters are automatically converted to VARIANT since IDispatch requires all params to be variant :huggles:

Edited February 11, 2010 by ProgAndy

junkew · February 11, 2010

STDAPI AccessibleObjectFromPoint( __in POINT ptScreen, __out IAccessible **ppacc, __out VARIANT *pvarChild);

So then I have something in *pVarChild

$aMousePos = MouseGetPos()
$aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint",  "int", $aMousePos[0], "int", $aMousePos[1], "ptr*", 0, "ptr", $pVar)
consolewrite($oIAccesible.accName($pvar))

Maybe I am confused by syntax as in Visual Basic the following works

' Get pointer position.
    GetCursorPos p
   ' Get IAccessible interface from object under pointer.
    AccessibleObjectFromPoint p.x, p.y, objAccessible, v
   ' Get name property of object under pointer.
    sName = ""
    On Error Resume Next
    sName = objAccessible.accName(v)

When I run my code in visual basic for applications within excel

I make the call like objAccessible.accName(v)

http://msdn.microsoft.com/en-us/library/dd318448(VS.85).aspx

So I would assume I need to give the same parameter value also in AutoIt but just don't get the same result.

When I hover with the sample program in AutoIt over the tabs in Scite I dont get the text of the tabname but its empty whereas from VBA I get the actual text of the scite tabs.

So under the assumption that $pVar contains the value I need for the call to accname($pVar) whats the syntax to use?

I tried the suggestion you gave on

$oChild=$oIAccesible.accchild

but then

$oIAccesible.accName($oChild)

does not give the result I expect

edit: Actually v is the childnumber so if I hover over tab 1 the value in $pVar return should be 1 but as its a pointer to the value itself I don't know how to get the value

edit: coming close now by doing (based on the idispatch thread)

$oChild=COMVariantToValue($pVar)

Now I have the childnumber indeed in ochild

Edited February 11, 2010 by junkew

junkew · February 11, 2010

Almost got it working but somehow I miss the point. Any answer/help?

;When I use $x=2 I get indeed the text from the second tab of SCITE when I hover the mouse over the SCITE tabs but when I use the $oChild its not working

$x=2
$tInfo = $tinfo & "accName: " & $oIAccesible.accName($x) & @CRLF
$tInfo = $tinfo & "accName: " & $oIAccesible.accName($oChild) & @CRLF
$tInfo = $tinfo & "accName: " & $oIAccesible.accName(number($oChild)) & @CRLF
$tInfo = $tinfo & "accName: " & $oIAccesible.accName(int($oChild)) & @CRLF

EDIT: Tricked the code with (just to see how far I can get information, indeed get actual text when hovering over SCITE tabs)

dim $array[16] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
    if COMVariantToValue($pVar)<15 then
        $oChild=$array[COMVariantToValue($pVar)]
    Else
        $oChild=0
    endif

Second issue is with getting the location. Fails on that call somehow. Any clue?

Full sample code

#include <WinAPI.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <EditConstants.au3>
#include <StaticConstants.au3>
#include <Array.au3>

;~ opt("MustDeclareVars", 1)
Opt("GUIOnEventMode", 1)

Global $oMyError = ObjEvent("AutoIt.Error", "MyErrFunc")

Global Const $hOLE32 = DllOpen("ole32.dll")
Global Const $hOLEACC = DllOpen("oleacc.dll")
Global Const $hOleAut = DllOpen("oleaut.dll")

Global Const $VT_EMPTY = 0
Global Const $VT_NULL = 1
Global Const $VT_I2 = 2
Global Const $VT_I4 = 3
Global Const $VT_R4 = 4
Global Const $VT_R8 = 5
Global Const $VT_CY = 6
Global Const $VT_DATE = 7
Global Const $VT_BSTR = 8
Global Const $VT_DISPATCH = 9
Global Const $VT_ERROR = 10
Global Const $VT_BOOL = 11
Global Const $VT_VARIANT = 12
Global Const $VT_UNKNOWN = 13
Global Const $VT_DECIMAL = 14
Global Const $VT_I1 = 16
Global Const $VT_UI1 = 17
Global Const $VT_UI2 = 18
Global Const $VT_UI4 = 19
Global Const $VT_I8 = 20
Global Const $VT_UI8 = 21
Global Const $VT_INT = 22
Global Const $VT_UINT = 23
Global Const $VT_VOID = 24
Global Const $VT_HRESULT = 25
Global Const $VT_PTR = 26
Global Const $VT_SAFEARRAY = 27
Global Const $VT_CARRAY = 28
Global Const $VT_USERDEFINED = 29
Global Const $VT_LPSTR = 30
Global Const $VT_LPWSTR = 31
Global Const $VT_RECORD = 36
Global Const $VT_INT_PTR = 37
Global Const $VT_UINT_PTR = 38
Global Const $VT_FILETIME = 64
Global Const $VT_BLOB = 65
Global Const $VT_STREAM = 66
Global Const $VT_STORAGE = 67
Global Const $VT_STREAMED_OBJECT = 68
Global Const $VT_STORED_OBJECT = 69
Global Const $VT_BLOB_OBJECT = 70
Global Const $VT_CF = 71
Global Const $VT_CLSID = 72
Global Const $VT_VERSIONED_STREAM = 73
Global Const $VT_BSTR_BLOB = 0xfff
Global Const $VT_VECTOR = 0x1000
Global Const $VT_ARRAY = 0x2000
Global Const $VT_BYREF = 0x4000
Global Const $VT_RESERVED = 0x8000
Global Const $VT_ILLEGAL = 0xffff
Global Const $VT_ILLEGALMASKED = 0xfff
Global Const $VT_TYPEMASK = 0xfff

;~ Global Const $tagGUID = "ulong;ushort;ushort;byte[8]"
Global Const $tagCLSID = $tagGUID
Global Const $tagUUID = $tagGUID
Global Const $CLSCTX_INPROC_SERVER = 0x1
Global Const $S_OK = 0
Global Const $DISP_E_UNKNOWNNAME = 2147614726
Global Const $DISPID_UNKNOWN = 4294967295
Global Const $DISP_E_MEMBERNOTFOUND = 2147614723
Global Const $tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;uint64 data"
Global Const $tagDISPPARAMS = "ptr rgvargs;ptr rgdispidNamedArgs;dword cArgs;dword cNamedArgs;"

Global Const $DISPATCH_METHOD = 0x1
Global Const $DISPATCH_PROPERTYGET = 0x2
Global Const $DISPATCH_PROPERTYPUT = 0x4
Global Const $DISPATCH_PROPERTYPUTREF = 0x8

Global Const $tagLOOKUP_TABLE_ENTRY = "ptr name;ptr value;dword flags;"
Global Const $LOOKUP_TABLE_ENTRY_SIZE = DllStructGetSize(DllStructCreate($tagLOOKUP_TABLE_ENTRY))
Global Const $LOOKUP_TABLE_METHOD = 1
Global Const $LOOKUP_TABLE_PROPERTY = 2
Global Const $LOOKUP_TABLE_PUBLIC = 0
Global Const $LOOKUP_TABLE_PRIVATE = 4

Global Const $tagOBJECT_INFO = "dword refcount;ptr release;"

Global Const $hOLEOUT = DllOpen("oleaut32.dll")

Global Const $SizeOfPtr = DllStructGetSize(DllStructCreate("ptr"))
Global $myEdit

GUICreate("MSAA Spy")
$myEdit = GUICtrlCreateEdit("First line" & @CRLF, 10, 10, 400, 300, $ES_AUTOVSCROLL + $WS_VSCROLL)
GUISetOnEvent(-3, "_Quit")
GUISetState()

Global $aMousePos, $aCall, $oIAccesible
Global $tInfo
global $oChild

Global Const $__Au3Obj_tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;ptr data; ptr"
Global $tVar = DllStructCreate($__Au3Obj_tagVARIANT)
Global $pVar = DllStructGetPtr($tVar)
__Au3Obj_VariantInit($pVar)

While 1

    $aMousePos = MouseGetPos()
    $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", _
            "int", $aMousePos[0], _
            "int", $aMousePos[1], _
            "ptr*", 0, _
            "ptr", $pVar)
;~             "ptr", $pVar)

    If @error Or $aCall[0]<>0 Then 
        ConsoleWrite("!DLLCall-Error or error on receiving IAccesible" & @CRLF)
        Sleep(500)
        ContinueLoop
    EndIf
    $oIAccesible = __Au3Obj_ConvertPtrToIDispatch($aCall[3]) ; create IDispatch when no error occured

    If IsObj($oIAccesible) Then
    ; ConsoleWrite("accChild: " & $oAccessibleObject.accChild($Param) & @CRLF) ;<-

    GUICtrlSetData($myedit, "")
;~  $oChild=$oIAccesible.accchild
    
    $x=int(2+$oChild)-2
    dim $array[16] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
    if COMVariantToValue($pVar)<15 then
        $oChild=$array[COMVariantToValue($pVar)]
    Else
        $oChild=0
    endif
    $tInfo=$x & @CRLF
    $tInfo = $tinfo & "accChildCount: " & $oIAccesible.accChildCount($oChild)& @CRLF
    $tInfo = $tinfo & "accDefaultAction: " & $oIAccesible.accDefaultAction($oChild) & @CRLF
    $tInfo = $tinfo & "accDescription: " & $oIAccesible.accDescription($oChild) & @CRLF
    $tInfo = $tinfo & "accFocus: " & $oIAccesible.accFocus($oChild) & @CRLF
    $tInfo = $tinfo & "accHelp: " & $oIAccesible.accHelp($oChild) & @CRLF
;~     $tInfo = $tinfo & "accHelpTopic: " & $oIAccesible.accHelpTopic($Param) & @CRLF
    $tInfo = $tinfo & "accKeyboardShortcut: " & $oIAccesible.accKeyboardShortcut & @CRLF
    $tInfo = $tinfo & "accName: " & $oIAccesible.accName($oChild) & @CRLF
;~  $tInfo = $tinfo & "accName: " & $oIAccesible.accName(number($oChild)) & @CRLF
;~  $tInfo = $tinfo & "accName: " & $oIAccesible.accName(int($oChild)) & @CRLF
      
    $tInfo = $tinfo & "accParent: " & $oIAccesible.accParent($oChild) & @CRLF
    dim $sText
    
    ;- Get the textual description of the role
    local $aRoleCall = DllCall($hOLEACC, "int", "GetRoleText", "DWORD", $oIAccesible.accRole($oChild), "str", $sText, "int", 255)
    
    $tInfo = $tinfo & "accRole: " & $oIAccesible.accRole($oChild) & ":" & $aRoleCall[2] & @CRLF
    $tInfo = $tinfo & "accSelection: " & $oIAccesible.accSelection($oChild) & @CRLF
    $tInfo = $tinfo & "accState: " & $oIAccesible.accState($oChild) & @CRLF
    $tInfo = $tinfo & "accValue: " & $oIAccesible.accValue($oChild) & @CRLF
    
    dim $pX,$pY,$cX,$cY
;~  $oIAccesible.accLocation($px,$py,$cx,$cy,$oChild)
    $oIAccesible.accLocation($px,$py,$cx,$cy,$pVar)
    
    $tInfo = $tinfo & "accLocation: " & $px & $py & $cx & $cy & @CRLF
    GUICtrlSetData($myedit, $tInfo)
    
    Else
        ConsoleWrite("! NOT OBJECT" & @CRLF)
    EndIf

;~ Do not forget to clean the return result
    __Au3Obj_VariantClear($aCall[4])
    $oIAccesible = 0 ; free Object
    Sleep(500)

WEnd

Func MyErrFunc()
    ConsoleWrite("!COM Error!   " & "Number is: " & Hex($oMyError.number, 8) & "  description is: " & $oMyError.windescription & @CRLF)
EndFunc   ;==>MyErrFunc

; FUNCTIONS:

Func _GetObjectWinHandle($oObject)

    Local $aCall = DllCall("oleacc.dll", "int", "WindowFromAccessibleObject", _
            "idispatch", $oObject, _
            "hwnd*", 0)

    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[2]

EndFunc   ;==>_GetObjectWinHandle

Func _AccessibleObjectFromWindow($hWnd)

    Local $sIID_IAccessible = "{618736E0-3C3D-11CF-810C-00AA00389B71}"
    Local $tGUID = _WinAPI_GUIDFromString($sIID_IAccessible)
    Local $pGUID = DllStructGetPtr($tGUID)

    Local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromWindow", _
            "hwnd", $hWnd, _
            "dword", 0, _ ; OBJID_WINDOW
            "ptr", $pGUID, _
            "idispatch*", 0)

    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[4]

EndFunc   ;==>_AccessibleObjectFromWindow
Func _AccessibleObjectFromPoint($x,$y)
    Local $tPoint=DllStructCreate($tagPOINT)
    dllstructsetdata($tPoint,1,$x)
    dllstructsetdata($tPoint,2,$y)
;~ STDAPI AccessibleObjectFromWindow(  __in   HWND hwnd,  __in   DWORD dwObjectID,  __in   REFIID riid,  __out  void **ppvObject);
;~ STDAPI AccessibleObjectFromPoint(  __in   POINT ptScreen,  __out  IAccessible **ppacc,  __out  VARIANT *pvarChild);
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "ptr", DllStructGetPtr($tPOINT), "idispatch*", 0, "ptr", 0)
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "long", DllStructGetPtr($tPOINT), "idispatch", 0, "ptr", 0)
    local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "int", $x, "int", $y, "idispatch*", 0, "ptr", 0)
    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return ConvertPtrToIDispatch($aCall[3])

EndFunc   ;==>_AccessibleObjectFromWindow
; In the end we still want the autoit object. This function converts a raw pointer to an autoit object
Func ConvertPtrToIDispatch($pIDispatch)
    ; This would have been 10000x easier if autoit had supported the idispatch* type in dllstructs...
    ; Fortunetely memcpy can copy the pointer into a idispatch*, lucky us.
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", _
            "idispatch*", 0, _
            "ptr*", $pIDispatch, _
            "dword", 4)

    If @error Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[1]

EndFunc ;==>ConvertPtrToIDispatch

Func __Au3Obj_ConvertPtrToIDispatch($pIDispatch)
    ; Author: monoceres
    Local $iSize = 4+(4*@AutoItX64)
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", "idispatch*", 0, "ptr*", $pIDispatch, "dword", $iSize)
    If @error Then Return SetError(1, 0, 0)
    Return $aCall[1]
EndFunc   ;==>__Au3Obj_ConvertPtrToIDispatch
Func __Au3Obj_VariantClear($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantClear", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantClear
Func __Au3Obj_VariantInit($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantInit", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantInit

Func VTType2AutoitType($iVT_Type)
    ConsoleWrite("! " & $iVT_Type & @CRLF)
    Switch $iVT_Type
        Case $VT_I1
            Return "byte"
        Case $VT_I2
            Return "short"
        Case $VT_I4
            Return "int"
        Case $VT_BSTR
            Return "wstr"
        Case $VT_R8
            Return "double"
        Case $VT_DISPATCH
            Return "idispatch"
    EndSwitch
EndFunc ;==>VTType2AutoitType
Func COMVariantToValue($pVariant)
    Local $hVariant = DllStructCreate($tagVARIANT, $pVariant)
    ; Translate the vt id to a autoit dllcall type
    $sType = VTType2AutoitType(DllStructGetData($hVariant, "vt"))


    If $sType = "wstr" Then
        $pString = DllStructCreate("ptr", DllStructGetPtr($hVariant, "data"))
        ; Getting random crashes when trusting autoit to automatically use right size.
        ; doing it myself instead (also, it should be a BSTR, but it's not. Is autoit not obeying the rules!?
        $iString_Size = wcslen(DllStructGetData($pString, 1))

        ; Sorry trancexx, doesn't seem to work on large strings (crashes like crazy when trying to use on 1 MB string)
        ;$tSub = DllStructCreate("dword", DllStructGetData($str_ptr, 1) - 4) ; <- move pointer back 4 bytes!

        $hData = DllStructCreate("wchar[" & $iString_Size & "]", DllStructGetData($pString, 1))

    ElseIf $sType = "idispatch" Then

        Return ConvertPtrToIDispatch(DllStructGetData(DllStructCreate("ptr", DllStructGetPtr($hVariant, "data")), 1))

    Else
        $hData = DllStructCreate($sType, DllStructGetPtr($hVariant, "data"))
    EndIf

    Return DllStructGetData($hData, 1)
EndFunc ;==>COMVariantToValue

; Find out length of string. I do not trust autoit to do this.
Func wcslen($pWString)
    $aCall = DllCall("ntdll.dll", "dword:cdecl", "wcslen", "ptr", $pWString)
    Return $aCall[0]
EndFunc ;==>wcslen



Func _Quit()

    Exit

EndFunc   ;==>_Quit
;

Edited February 11, 2010 by junkew

junkew · February 13, 2010

Only location part is not working. Any tips?

Within visual basic for applications I got everything working (to rule out computer specific problems I rebuild the sample in VBA)

#include <WinAPI.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <EditConstants.au3>
#include <StaticConstants.au3>
#include <Array.au3>

;~ opt("MustDeclareVars", 1)
Opt("GUIOnEventMode", 1)
global $tInfo=""

Global $oMyError = ObjEvent("AutoIt.Error", "MyErrFunc")

Global Const $hOLE32 = DllOpen("ole32.dll")
Global Const $hOLEACC = DllOpen("oleacc.dll")
Global Const $hOleAut = DllOpen("oleaut.dll")

; Starts COM
Func CoInitialize()
;~     $hOle32 = DllOpen("Ole32.dll")
    Local $aCall = DllCall($hOle32, "long", "CoInitializeEx", "ptr", 0, "dword", 2) ; COINIT_APARTMENTTHREADED
EndFunc ;==>CoInitialize

CoInitialize()

;~ ObjCoUninitialize()


Global Const $VT_EMPTY = 0
Global Const $VT_NULL = 1
Global Const $VT_I2 = 2
Global Const $VT_I4 = 3
Global Const $VT_R4 = 4
Global Const $VT_R8 = 5
Global Const $VT_CY = 6
Global Const $VT_DATE = 7
Global Const $VT_BSTR = 8
Global Const $VT_DISPATCH = 9
Global Const $VT_ERROR = 10
Global Const $VT_BOOL = 11
Global Const $VT_VARIANT = 12
Global Const $VT_UNKNOWN = 13
Global Const $VT_DECIMAL = 14
Global Const $VT_I1 = 16
Global Const $VT_UI1 = 17
Global Const $VT_UI2 = 18
Global Const $VT_UI4 = 19
Global Const $VT_I8 = 20
Global Const $VT_UI8 = 21
Global Const $VT_INT = 22
Global Const $VT_UINT = 23
Global Const $VT_VOID = 24
Global Const $VT_HRESULT = 25
Global Const $VT_PTR = 26
Global Const $VT_SAFEARRAY = 27
Global Const $VT_CARRAY = 28
Global Const $VT_USERDEFINED = 29
Global Const $VT_LPSTR = 30
Global Const $VT_LPWSTR = 31
Global Const $VT_RECORD = 36
Global Const $VT_INT_PTR = 37
Global Const $VT_UINT_PTR = 38
Global Const $VT_FILETIME = 64
Global Const $VT_BLOB = 65
Global Const $VT_STREAM = 66
Global Const $VT_STORAGE = 67
Global Const $VT_STREAMED_OBJECT = 68
Global Const $VT_STORED_OBJECT = 69
Global Const $VT_BLOB_OBJECT = 70
Global Const $VT_CF = 71
Global Const $VT_CLSID = 72
Global Const $VT_VERSIONED_STREAM = 73
Global Const $VT_BSTR_BLOB = 0xfff
Global Const $VT_VECTOR = 0x1000
Global Const $VT_ARRAY = 0x2000
Global Const $VT_BYREF = 0x4000
Global Const $VT_RESERVED = 0x8000
Global Const $VT_ILLEGAL = 0xffff
Global Const $VT_ILLEGALMASKED = 0xfff
Global Const $VT_TYPEMASK = 0xfff

;~ Global Const $tagGUID = "ulong;ushort;ushort;byte[8]"
Global Const $tagCLSID = $tagGUID
Global Const $tagUUID = $tagGUID
Global Const $CLSCTX_INPROC_SERVER = 0x1
Global Const $S_OK = 0
Global Const $DISP_E_UNKNOWNNAME = 2147614726
Global Const $DISPID_UNKNOWN = 4294967295
Global Const $DISP_E_MEMBERNOTFOUND = 2147614723
Global Const $tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;uint64 data"
Global Const $tagDISPPARAMS = "ptr rgvargs;ptr rgdispidNamedArgs;dword cArgs;dword cNamedArgs;"

Global Const $DISPATCH_METHOD = 0x1
Global Const $DISPATCH_PROPERTYGET = 0x2
Global Const $DISPATCH_PROPERTYPUT = 0x4
Global Const $DISPATCH_PROPERTYPUTREF = 0x8

Global Const $tagLOOKUP_TABLE_ENTRY = "ptr name;ptr value;dword flags;"
Global Const $LOOKUP_TABLE_ENTRY_SIZE = DllStructGetSize(DllStructCreate($tagLOOKUP_TABLE_ENTRY))
Global Const $LOOKUP_TABLE_METHOD = 1
Global Const $LOOKUP_TABLE_PROPERTY = 2
Global Const $LOOKUP_TABLE_PUBLIC = 0
Global Const $LOOKUP_TABLE_PRIVATE = 4

Global Const $tagOBJECT_INFO = "dword refcount;ptr release;"

Global Const $hOLEOUT = DllOpen("oleaut32.dll")

Global Const $SizeOfPtr = DllStructGetSize(DllStructCreate("ptr"))
Global $myEdit

GUICreate("MSAA Spy",600,600,10,10)
$myEdit = GUICtrlCreateEdit("First line" & @CRLF, 10, 10, 580, 580, $ES_AUTOVSCROLL + $WS_VSCROLL)
GUISetOnEvent(-3, "_Quit")
GUISetState()

Global $aMousePos, $aCall, $oIAccesible
Global $tInfo
global $oChild

Global Const $__Au3Obj_tagVARIANT = "ushort vt;ushort r1;ushort r2;ushort r3;ptr data; ptr"
Global $tVar = DllStructCreate($__Au3Obj_tagVARIANT)
Global $pVar = DllStructGetPtr($tVar)
__Au3Obj_VariantInit($pVar)

While 1

    $aMousePos = MouseGetPos()
    $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", _
            "int", $aMousePos[0], _
            "int", $aMousePos[1], _
            "ptr*", 0, _
            "ptr", $pVar)
;~             "ptr", $pVar)

    If @error Or $aCall[0]<>0 Then 
        ConsoleWrite("!DLLCall-Error or error on receiving IAccesible" & @CRLF)
        Sleep(500)
        ContinueLoop
    EndIf
    $oIAccesible = __Au3Obj_ConvertPtrToIDispatch($aCall[3]) ; create IDispatch when no error occured

    If IsObj($oIAccesible) Then
    ; ConsoleWrite("accChild: " & $oAccessibleObject.accChild($Param) & @CRLF) ;<-

    GUICtrlSetData($myedit, "")
;~ TO DO: Somehow its not an integer like expected  
;~  $oChild=$oIAccesible.accchild
    
;~  $x=int(2+$oChild)-2
    dim $array[16] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
    if COMVariantToValue($pVar)<15 then
        $oChild=$array[COMVariantToValue($pVar)]
    Else
        $oChild=0
    endif
    
;~ $oChild=$pVar
    $tInfo = "Info for child id:" & $oChild & @CRLF
    
;~ HRESULT get_accChildCount(  [out, retval]  long *pcountChildren);
;~  $tInfo = $tinfo & "accChildCount: "
;~  $tVal  = $oIAccesible.accChildCount($oChild)
;~  $tInfo = $tinfo &  $tVal & ";" &@CRLF
    $tInfo = $tinfo & "accChildCount: "
    $tVal  = $oIAccesible.accChildCount
    $tInfo = $tinfo &  $tVal & ";" &@CRLF

    $tInfo = $tinfo & "accDefaultAction: "
    $tVal  = $oIAccesible.accDefaultAction($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accDescription: "
    $tVal  = $oIAccesible.accDescription($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
;~ HRESULT get_accFocus(  [out, retval]  VARIANT *pvarID);
;~  $tInfo = $tinfo & "accFocus: "
;~  $tVal  = $oIAccesible.accFocus($oChild)
;~  $tInfo = $tinfo & $tVal & ";" & @CRLF
    $tInfo = $tinfo & "accFocus: "
    $tVal  = $oIAccesible.accFocus
    $tInfo = $tinfo & $tVal & ";" & @CRLF

    $tInfo = $tinfo & "accHelp: "
    $tVal  = $oIAccesible.accHelp($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accHelpTopic: " 
    $tVal  = $oIAccesible.accHelpTopic($oChild) 
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accKeyboardShortcut: "
    $tVal  = $oIAccesible.accKeyboardShortcut
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accName: "
    $tVal  = $oIAccesible.accName($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
;~  HRESULT get_accParent(  [out, retval]  IDispatch **ppdispParent);
;~  $tInfo = $tinfo & "accParent: "
;~  $tVal  = $oIAccesible.accParent($oChild)
;~  $tInfo = $tinfo & $tVal & ";" & @CRLF
    $tInfo = $tinfo & "accParent: "
    $tVal  = $oIAccesible.accParent
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    dim $sText
    
    ;- Get the textual description of the role
    $tInfo = $tinfo & "accRole: "
    $tVal  = $oIAccesible.accRole($oChild)
    local $aRoleCall = DllCall($hOLEACC, "int", "GetRoleText", "DWORD", $tVal, "str", $sText, "int", 255)
    $tInfo = $tinfo & $tVal & ":" & $aRoleCall[2] & ";" & @CRLF
    
    $tInfo = $tinfo & "accSelection: "
    $tVal  = $oIAccesible.accSelection($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accState: "
    $tVal  = $oIAccesible.accState($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
    $tInfo = $tinfo & "accValue: "
    $tVal  = $oIAccesible.accValue($oChild)
    $tInfo = $tinfo & $tVal & ";" & @CRLF
    
;~  All kinds of ways to get the location rectangle
    $i4Int          = DllStructCreate($tagRECT)

    dim $pX,$pY,$cX,$cY
    
;~  This should work but doesn't
;~  $oIAccesible.accLocation($px,$py,$cx,$cy,$oChild)
;~ All kinds of workarounds not working
;~ Just to see if something is changed variables initialized
    dim $p[4]=[1,2,3,4]
    $px=1
    $pY=2
    $cX=3
    $cY=4
$tInfo = $tinfo & "************************* TO DO Location not working ****************************" & ";" & @crlf
$tInfo = $tinfo & "call1" & ";" & @crlf
;~  $aRet1=$oIaccesible.acclocation()
    $aRet1= $oIAccesible.accLocation($p[0],$p[1],$p[2],$p[3],$oChild)
    $tInfo = $tinfo & "call2" & ";" & @crlf
    $aRet2=$oIAccesible.accLocation(DllStructGetPtr($i4Int,1) ,DllStructGetPtr($i4Int,2),DllStructGetPtr($i4Int,3),DllStructGetPtr($i4Int,4),$oChild) 
;~  $aRet2=$oIAccesible.accLocation(DllStructGetPtr($i4Int),$oChild) 
$tInfo = $tinfo & "call3" & ";" & @crlf
    $aRet3=$oIAccesible.accLocation($px ,$py,$cx,$cy,0) 
    
    $tInfo = $tinfo & $aRet1 & "accLocation 1 : (" & $p[0] & "," & $p[1] & "," & $p[2]& "," & $p[3] & ")" & ";" & @CRLF
    $tInfo = $tinfo & $aRet2 & "accLocation 2 : (" & $px & "," & $py & "," & $cx & "," & $cy & ")" & ";" & @CRLF
    $tInfo = $tinfo & $aRet3 & "accLocation 3 : " & dllStructGetData($i4Int,1) & dllStructGetData($i4Int,2) & dllStructGetData($i4Int,3) & dllStructGetData($i4Int,4) & ";" & @CRLF

    GUICtrlSetData($myedit, $tInfo)
    
    Else
        ConsoleWrite("! NOT OBJECT" & @CRLF)
    EndIf

;~ Do not forget to clean the return result
    __Au3Obj_VariantClear($aCall[4])
    $oIAccesible = 0 ; free Object
    Sleep(500)

WEnd

Func MyErrFunc()
;~     ConsoleWrite("!COM Error!   " & "Number is: " & Hex($oMyError.number, 8) & "  description is: " & $oMyError.windescription & @CRLF)
     $tInfo=$tInfo & "COM Err #: " & Hex($oMyError.number, 8) & " description: " & $oMyError.windescription 
;~   & @CRLF

EndFunc   ;==>MyErrFunc

; FUNCTIONS:

Func _GetObjectWinHandle($oObject)

    Local $aCall = DllCall("oleacc.dll", "int", "WindowFromAccessibleObject", _
            "idispatch", $oObject, _
            "hwnd*", 0)

    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[2]

EndFunc   ;==>_GetObjectWinHandle

Func _AccessibleObjectFromWindow($hWnd)

    Local $sIID_IAccessible = "{618736E0-3C3D-11CF-810C-00AA00389B71}"
    Local $tGUID = _WinAPI_GUIDFromString($sIID_IAccessible)
    Local $pGUID = DllStructGetPtr($tGUID)

    Local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromWindow", _
            "hwnd", $hWnd, _
            "dword", 0, _ ; OBJID_WINDOW
            "ptr", $pGUID, _
            "idispatch*", 0)

    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[4]

EndFunc   ;==>_AccessibleObjectFromWindow
Func _AccessibleObjectFromPoint($x,$y)
    Local $tPoint=DllStructCreate($tagPOINT)
    dllstructsetdata($tPoint,1,$x)
    dllstructsetdata($tPoint,2,$y)
;~ STDAPI AccessibleObjectFromWindow(  __in   HWND hwnd,  __in   DWORD dwObjectID,  __in   REFIID riid,  __out  void **ppvObject);
;~ STDAPI AccessibleObjectFromPoint(  __in   POINT ptScreen,  __out  IAccessible **ppacc,  __out  VARIANT *pvarChild);
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "ptr", DllStructGetPtr($tPOINT), "idispatch*", 0, "ptr", 0)
;~     local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "long", DllStructGetPtr($tPOINT), "idispatch", 0, "ptr", 0)
    local $aCall = DllCall($hOLEACC, "int", "AccessibleObjectFromPoint", "int", $x, "int", $y, "idispatch*", 0, "ptr", 0)
    If @error Or $aCall[0] Then
        Return SetError(1, 0, 0)
    EndIf

    Return ConvertPtrToIDispatch($aCall[3])

EndFunc   ;==>_AccessibleObjectFromWindow
; In the end we still want the autoit object. This function converts a raw pointer to an autoit object
Func ConvertPtrToIDispatch($pIDispatch)
    ; This would have been 10000x easier if autoit had supported the idispatch* type in dllstructs...
    ; Fortunetely memcpy can copy the pointer into a idispatch*, lucky us.
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", _
            "idispatch*", 0, _
            "ptr*", $pIDispatch, _
            "dword", 4)

    If @error Then
        Return SetError(1, 0, 0)
    EndIf

    Return $aCall[1]

EndFunc ;==>ConvertPtrToIDispatch

Func __Au3Obj_ConvertPtrToIDispatch($pIDispatch)
    ; Author: monoceres
    Local $iSize = 4+(4*@AutoItX64)
    Local $aCall = DllCall("kernel32.dll", "none", "RtlMoveMemory", "idispatch*", 0, "ptr*", $pIDispatch, "dword", $iSize)
    If @error Then Return SetError(1, 0, 0)
    Return $aCall[1]
EndFunc   ;==>__Au3Obj_ConvertPtrToIDispatch
Func __Au3Obj_VariantClear($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantClear", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantClear
Func __Au3Obj_VariantInit($pvarg)
    ; Author: Prog@ndy
    Local $aCall = DllCall($hOleAut, "long", "VariantInit", "ptr", $pvarg)
    If @error Then Return SetError(1, 0, 1)
    Return $aCall[0]
EndFunc   ;==>__Au3Obj_VariantInit

Func VTType2AutoitType($iVT_Type)
;~     ConsoleWrite("! " & $iVT_Type & @CRLF)
    Switch $iVT_Type
        Case $VT_I1
            Return "byte"
        Case $VT_I2
            Return "short"
        Case $VT_I4
            Return "int"
        Case $VT_BSTR
            Return "wstr"
        Case $VT_R8
            Return "double"
        Case $VT_DISPATCH
            Return "idispatch"
    EndSwitch
EndFunc ;==>VTType2AutoitType
Func COMVariantToValue($pVariant)
    Local $hVariant = DllStructCreate($tagVARIANT, $pVariant)
    ; Translate the vt id to a autoit dllcall type
    $sType = VTType2AutoitType(DllStructGetData($hVariant, "vt"))


    If $sType = "wstr" Then
        $pString = DllStructCreate("ptr", DllStructGetPtr($hVariant, "data"))
        ; Getting random crashes when trusting autoit to automatically use right size.
        ; doing it myself instead (also, it should be a BSTR, but it's not. Is autoit not obeying the rules!?
        $iString_Size = wcslen(DllStructGetData($pString, 1))

        ; Sorry trancexx, doesn't seem to work on large strings (crashes like crazy when trying to use on 1 MB string)
        ;$tSub = DllStructCreate("dword", DllStructGetData($str_ptr, 1) - 4) ; <- move pointer back 4 bytes!

        $hData = DllStructCreate("wchar[" & $iString_Size & "]", DllStructGetData($pString, 1))

    ElseIf $sType = "idispatch" Then

        Return ConvertPtrToIDispatch(DllStructGetData(DllStructCreate("ptr", DllStructGetPtr($hVariant, "data")), 1))

    Else
        $hData = DllStructCreate($sType, DllStructGetPtr($hVariant, "data"))
    EndIf

    Return DllStructGetData($hData, 1)
EndFunc ;==>COMVariantToValue

; Find out length of string. I do not trust autoit to do this.
Func wcslen($pWString)
    $aCall = DllCall("ntdll.dll", "dword:cdecl", "wcslen", "ptr", $pWString)
    Return $aCall[0]
EndFunc ;==>wcslen



Func _Quit()
consolewrite($tinfo)
    Exit

EndFunc   ;==>_Quit
;

yehia · February 23, 2010

omg finally someone thought of looking into this

thanks alot junkew for the pm :mellow:

ProgAndy · February 23, 2010

@junkew: There is no way to use the accLocation directly. You have to create a wrapper with _AutoItObject_WrapperCreate, at least for thejis method. To do this you have to know the vTable of this Object. There is no other possibility to overcome this AutoIt-limitation. (Apart from doing the IDispatch::Invoke manually with some internal functions of AutoItObject)

MSAA Active accessibility

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members