Google Chrome automation with UIA

[maniootek]

1. Is it possible to keep option "Global accessibility mode: on" for all the time? For example, even if I run chrome browser by click on link from some windows app?

2. Is it possible to enable accessibility mode for specified tab in Chrome on command from autoit?

3. Is it possible that accessibility mode slow down my chrome browser?

4. Is it good idea to use UIA for auto-login in some webpages (set value on edit box) or you have any better idea? At this moment I use ImageSearch which is not perfect.

Edited April 25, 2016 by maniootek

[junkew]

1. Yes
2. Yes
3. Yes
4. Yes

All goes.either with startup parameter or thru adressbar chrome://accessibility and yes accessibility slows down as it builds besides dom tree a uia tree but doubt if you will notice on nowadays cpu.

 

https://www.chromium.org/developers/design-documents/accessibility#TOC-How-Chrome-detects-the-presence-of-Assistive-Technology

Chrome sends a system alert you can catch and respond to. Its in the iuiathread or in the examples.

 

[maniootek]

1. What if that "windows app" has link with no option to set startup parameter? Chrome starts without accessibility on. Any chance to keep accessibility always on with other way?

2. How to enable accessibility for existing selected tab in chrome from AutoIt without using adressbar `chrome://accessibility`?

2a. Is it possible to read via autoit if selected tab in chrome has accessibility on?  Is it done by look for `class:=Chrome_WidgetWin_1`? 

3. My PC is quite powerfull but turning accessibility on slow down browser a lot (freezes when scrolling down page)

[junkew]

You did not read my answer and did nit read the link. 

Chrome broadcasts a message which you can catch in autoit program and if you do a sendmessage chrome will listen to that and turn on accessibility.

[maniootek]

11 hours ago, junkew said:

 

You did not read my answer and did nit read the link. 

 

Yes I did but simple I don't understand it.

I didn't know it's possible to listen chrome message and sendmessage. Best way I am learning is to study some example. Do you have any?

[junkew]

 

Understanding uia takes a lot of reading. In above link you will see the handshake. Search sendmessage in uia examples thread if link above is not to the right plaze.

 

[maniootek]

@junkew I have Chrome web browser window with opened www.google.com webpage. Can you make for me code example which input some text into search box on website?

 

[BugFix]

51 minutes ago, maniootek said:

Can you make for me code example which input some text into search box on website?

To perform a Google search, it is not easier to access the default browser with keyword?

_GoogleSearch('autoit forum')

Func _GoogleSearch($sSearch)
    ShellExecute('www.google.com/search?q=' & $sSearch)
EndFunc

 

[maniootek]

This is more about "set text" function on specified input box on webpage to achieve auto login script (set text on username and password field)

[JLogan3o13]

6 hours ago, maniootek said:

Can you make for me code example which input some text into search box on website?

@maniootek I told you in your last thread on this same subject the same thing junkew is telling you now. You need to do some reading and experimentation and learn how to do what you're after; that is what this forum is here to help you do (and you've been around long enough to know this). All the resources you need are there, but no one is going to spoon feed the code to you because you don't feel like putting any effort into it.

Read the links you've now been pointed to multiple times, try it out on your own. Then, if you're stuck, come back and show the code you've tried.

[junkew]

Check the simplespy it will give you example code for any element you spy on.

[junkew]

Some tips and the working code (its just not easy with HTML as it gets sometimes weird UI Hierarchy).
Be aware my system is dutch so probably if you need english change Zoek to Search or make it like title:=((Zoek)|(Search))

1. Start with the simplespy code given
2. Check with inspect your hierarchy of parents and important is to see if all parents are highlightable. If not remove them from the actions like you see on the first parent of the combobox
3. Check if HTML tags / elements have names/titles if not most likely you have to count to use index:=## as properties identifying are not unique
4. If needed use _UIA_DumpThemAll you will get the hierarchy in the log (personally I prefer to use inspect.exe)
   !!! be aware getobject as given below works only at the moment direct after the action with the logical name!!!
5. setvalue action of iuiAutomation is frequently not working on elements so i use the setvalue using keys
6. I will leave the click on the search button to you as an exercise

expandcollapse popup

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Google - Google Chrome;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1")   ;Google - Google Chrome
_UIA_setVar("oP2","Title:=Google;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")    ;Google
_UIA_setVar("oP3","Title:=;controltype:=UIA_GroupControlTypeId;class:=")    ;

;~ $oUIElement=_UIA_getObjectByFindAll("Zoek.mainwindow", "title:=Zoek;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Zoek;controltype:=UIA_EditControlTypeId;class:=") ;ControlType:=UIA_EditControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~ _UIA_setVar("oUIElement","Title:=Zoek.*;controltype:=ComboBox;class:=") ;ControlType:=UIA_EditControlTypeId;classname:=")


_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

;~ _UIA_DumpThemAll(_UIA_Action("oP3","getobject"), $treescope_children)

;~ _UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=;index:=3")   ;

;~ _UIA_Action("oP4","highlight")
;~ _UIA_Action("oP4","setfocus")

;~ If stuff is good enough in the property definition above with setvar it will work

Consolewrite("Before the actual element")
sleep(500)
_UIA_action("oUIElement","highlight")
sleep(500)
Consolewrite("After the actual element")
_UIA_action("oUIElement","setvalue using keys","iuiwrappers")
;~ _UIA_action("oUIElement","setvalue","iuiwrappers")

;~_UIA_action($oUIElement,"click")
;~_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")

 

Edited September 1, 2016 by junkew

[LarsJ]

The code below shows how to automate a Google search in Chrome directly with the UI Automation interfaces (answer to a PM).

Open Chrome and enable accessibility mode with chrome://accessibility.
select a Google search page eg. https://www.google.co.uk/.

Run this code to start a search:

expandcollapse popup

;#include "..\..\Include\CUIAutomation2.au3"
#include "CUIAutomation2.au3"

$hWindow = WinGetHandle( "Google - Google Chrome" ) ; AutoIt Windows Info tool
If Not $hWindow Then Exit ConsoleWrite( "$hWindow ERR" & @CRLF )
ConsoleWrite( "$hWindow OK" & @CRLF )

$hControl = ControlGetHandle( $hWindow, "", "[CLASS:Chrome_RenderWidgetHostHWND]" ) ; AutoIt Windows Info tool
If Not $hControl Then Exit ConsoleWrite( "$hControl ERR" & @CRLF )
ConsoleWrite( "$hControl OK" & @CRLF )

; Create UI Automation object
$oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
If Not IsObj( $oUIAutomation ) Then Exit ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
ConsoleWrite( "$oUIAutomation OK" & @CRLF )

; Get UI Automation element from control handle
Local $pControl, $oControl
$oUIAutomation.ElementFromHandle( $hControl, $pControl )
$oControl = ObjCreateInterface( $pControl, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oControl ) Then Exit ConsoleWrite( "$oControl ERR" & @CRLF )
ConsoleWrite( "$oControl OK" & @CRLF )

; Condition to find ComboBox
Local $pCondition ; Use Inspect.exe or Simple spy demo to verify that the search box is a ComboBox
$oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_ComboBoxControlTypeId, $pCondition )
If Not $pCondition Then Exit ConsoleWrite( "$pCondition ERR" & @CRLF )
ConsoleWrite( "$pCondition OK" & @CRLF )

; Find ComboBox
Local $pUIElementArray, $oUIElementArray, $iElements
$oControl.FindAll( $TreeScope_Descendants, $pCondition, $pUIElementArray )
$oUIElementArray = ObjCreateInterface( $pUIElementArray, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
$oUIElementArray.Length( $iElements )
If Not ( $iElements = 1 ) Then Exit ConsoleWrite( "$iElements ERR" & @CRLF )
ConsoleWrite( "$iElements OK" & @CRLF )

; Get ComboBox element
Local $pUIElement, $oUIElement
$oUIElementArray.GetElement( 0, $pUIElement )
$oUIElement = ObjCreateInterface( $pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

; Get Value object
Local $pValue, $oValue
$oUIElement.GetCurrentPattern( $UIA_ValuePatternId, $pValue )
$oValue = ObjCreateInterface( $pValue, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
If Not IsObj( $oValue ) Then Exit ConsoleWrite( "$oValue ERR" & @CRLF )
ConsoleWrite( "$oValue OK" & @CRLF )

; Set search text
;Local $sText
;$oValue.CurrentValue( $sText ) ; This works
;ConsoleWrite( "$sText = " & $sText & @CRLF )
$oValue.SetValue( "AutoIt" ) ; This does not work but sets input focus to the search box
Send( "AutoIt" )  ; Text to search
Send( "{Enter}" ) ; Perform search

 

Edited September 20, 2016 by LarsJ

[LarsJ]

Since there is only one ComboBox you can use FindFirst instead of FindAll. FindFirst finds the ComboBox directly while FindAll finds an array of ComboBox elements. This code is slightly shorter and faster:

expandcollapse popup

;#include "..\..\Include\CUIAutomation2.au3"
#include "CUIAutomation2.au3"

$hWindow = WinGetHandle( "Google - Google Chrome" ) ; AutoIt Windows Info tool
If Not $hWindow Then Exit ConsoleWrite( "$hWindow ERR" & @CRLF )
ConsoleWrite( "$hWindow OK" & @CRLF )

$hControl = ControlGetHandle( $hWindow, "", "[CLASS:Chrome_RenderWidgetHostHWND]" ) ; AutoIt Windows Info tool
If Not $hControl Then Exit ConsoleWrite( "$hControl ERR" & @CRLF )
ConsoleWrite( "$hControl OK" & @CRLF )

; Create UI Automation object
$oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
If Not IsObj( $oUIAutomation ) Then Exit ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
ConsoleWrite( "$oUIAutomation OK" & @CRLF )

; Get UI Automation element from control handle
Local $pControl, $oControl
$oUIAutomation.ElementFromHandle( $hControl, $pControl )
$oControl = ObjCreateInterface( $pControl, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oControl ) Then Exit ConsoleWrite( "$oControl ERR" & @CRLF )
ConsoleWrite( "$oControl OK" & @CRLF )

; Condition to find ComboBox
Local $pCondition ; Use Inspect.exe or Simple spy demo to verify that the search box is a ComboBox
$oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_ComboBoxControlTypeId, $pCondition )
If Not $pCondition Then Exit ConsoleWrite( "$pCondition ERR" & @CRLF )
ConsoleWrite( "$pCondition OK" & @CRLF )

; Find ComboBox
Local $pUIElement, $oUIElement
$oControl.FindFirst( $TreeScope_Descendants, $pCondition, $pUIElement )
$oUIElement = ObjCreateInterface( $pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oUIElement ) Then Exit ConsoleWrite( "$oUIElement ERR" & @CRLF )
ConsoleWrite( "$oUIElement OK" & @CRLF )

; Get Value object
Local $pValue, $oValue
$oUIElement.GetCurrentPattern( $UIA_ValuePatternId, $pValue )
$oValue = ObjCreateInterface( $pValue, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
If Not IsObj( $oValue ) Then Exit ConsoleWrite( "$oValue ERR" & @CRLF )
ConsoleWrite( "$oValue OK" & @CRLF )

; Set search text
;Local $sText
;$oValue.CurrentValue( $sText ) ; This works
;ConsoleWrite( "$sText = " & $sText & @CRLF )
$oValue.SetValue( "AutoIt" ) ; This does not work but sets input focus to the search box
Send( "AutoIt" )  ; Text to search
Send( "{Enter}" ) ; Perform search

 

[junkew]

And for people interested to get more from chrome https://developer.chrome.com/devtools/docs/debugger-protocol reveals more but harder then IUIAutomation / iSimpleDom / iAccessible interfaces

Edited September 20, 2016 by junkew

[maniootek]

On 26.04.2016 at 8:59 PM, junkew said:

Chrome broadcasts a message which you can catch in autoit program and if you do a sendmessage chrome will listen to that and turn on accessibility.

I have study your example

#include <APISysConstants.au3>
#include <GUIMenu.au3>
#include <WinAPIProc.au3>
#include <WinAPISys.au3>
#include <WindowsConstants.au3>

Local $hEventProc = DllCallbackRegister('_EventProc', 'none', 'ptr;dword;hwnd;long;long;dword;dword')
Global $g_tRECT, $g_iIndex, $g_hMenu = 0

OnAutoItExitRegister('OnAutoItExit')

Local $hEventHook = _WinAPI_SetWinEventHook($EVENT_SYSTEM_ALERT, $EVENT_SYSTEM_ALERT, DllCallbackGetPtr($hEventProc))

While 1
    Sleep(1000)
WEnd

Func OnAutoItExit()
    _WinAPI_UnhookWinEvent($hEventHook)
    DllCallbackFree($hEventProc)
EndFunc   ;==>OnAutoItExit

Func _EventProc($hEventHook, $iEvent, $hWnd, $iObjectID, $iChildID, $iThreadID, $iEventTime)
    #forceref $hEventHook, $iObjectID, $iChildID, $iThreadID, $iEventTime
    Switch $iEvent
        Case $EVENT_SYSTEM_ALERT ;When new tab in Chrome was opened and accessibility was OFF then CEF applications broadcast "Hello, anyone there" with $EVENT_SYSTEM_ALERT 
            if $iObjectID=1 then ; what is objectid=1?
                $result = _SendMessage($hWnd, $WM_GETOBJECT, 0, 1) ;any screenreader/automation can answer "yes, hello I am here" by sending back $WM_GETOBJECT.
                ConsoleWrite("Chrome gave a call so we replied " & $iObjectID)
            EndIf
    EndSwitch
EndFunc   ;==>_EventProc

But still can't figure out how to:

1. Turn accessibility ON for currenlty opened tab in chrome

2. Do some automation action like settext

3. When it's done, turn accessibility OFF for this tab

Are you able to direct me to any similar example?

[junkew]

1. the broadcast happens only when a new tab opens

2. the other parts are reachable thru chrome://accessibility and if its off there you have to click on on

as in 2 its hard to click on right position you have to fallback to type in addressbar javascript:<yourjavascript> to turn it on