IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

done for the basic links and first few examples

[mLipok]

Thanks.

[jushusted]

I tried ex5_chrome.au3 and it returned an undefined object error.

 

I was able to get it to open a new browser and then it would highlight the window in red before it errored out.

One more consideration, I had to disable the Google Hangouts extension before this process would work would work at all.

 

I've attached the log file. It looks like it errored right after it read through my bookmarks toolbar.

I very much would like to use this library for Chrome if possible. I have a working automation for IE using the IE.au3 library. I tried using the ChromeUDF library but was only able to get _ChromeShutdown to work.

20150709-123220189.XML

[Flipper]

NOTHING in this post is intended as an attack or criticism of the good works done!

ok, so here's the thing. I'm a System Software tester, with limited scripting (not coding -  I know how to use AutoIT and thats about it) abilities. I can test Windows applications just fine using AutoIT, using the AutoIT Help file and the SciTE editor.

 I've just spent rather a long time looking at this thread and the examples, and managed a couple of very simple "monkey-see monkey-do" tasks. Now I want to start to exercise a web page. I've Managed to open Chrome (using UIA - but using vanilla AutoIT is easier!) and log on to my web service, but thats as far as I managed.

Here's a sample of my grief:

Looking further at the examples, EG. ex31_Notepad and while I can understand what's going on, I can't modify to any effect.

I run SimpleSyp on Notepad.exe and this displays a pile of code.

I look at the example _UIA_setVar("notepad.mainwindow","title:=((.*Kladblok)|(.*Notepad));classname:=Notepad") and I have no reference as to what is going on.

I guess we're assigning to "notepad.mainwindow" a title parameter with 2 titles delimited by a pipe character, and a classname.

Later in the script we see

_UIA_action("notepad.mainwindow","setfocus")

which I guess sets focus to the notepad.

Now for the question. Where can I find the friendly Help  information? IE

What commands are available? - I spend alot of time in the help file reading the descriptions of the functions.

What is the syntax of each command? - after selecting the function, I then need to correctly add it to my script.

I'm not looking for specific assistance, as I don't know what to ask, I just need to know where to look.

Thanks for the efforts

[junkew]

Its still work in progress so no help file made so far. The simplespy should give the basic examples. 1. You can use the core uia functions or 2. Use the uia wrapper functions. The 2nd are more powerfull but still some issues to resolve to make it work stable and give clues logging information in logfile why its not finding an object. The .mainwindow you refer to is just a quick reference to give the framework a hint to where to start searching for other object to prevent scanning whole tree of objects. The setvar logic is only there to give meaningfull names to your objects. You can do it also with regular variables but then there is less flexibility in dynamic manipulation which you probably need in a ui test framework. The first post has a list of all commands and i hope they are logical enough to understand with 1 word. Just ask for support in general section and i will try to help.

[Flipper]

Many thanks - I'll continue digging - what harm can I do ;-)

(last time I said that I managed to destroy the Universe. Luckily I had a backup!)

[Iczer]

i have function for context menu in FF, but it fail on line $hMenu = _SendMessage($hWnd, $MN_GETHMENU, 0, 0)

what can be _UIA replacement for it? Basically i need make right-click in FF and select some line in opened context menu.

Func Click_RightClickMenuFF10($sRequired_Text = "")
    Local $previousMousePos, $hWnd, $hMenu, $iCount, $iIndex, $sText, $aPos, $iOldOpt
    $previousMousePos = MouseGetPos()
    If $sRequired_Text == "" Then Return SetError(1)
    Opt("WinWaitDelay", 16) ;250 milliseconds
    If WinWait("[CLASS:MozillaDropShadowWindowClass]","",7) Then
        $hWnd = WinGetHandle("[CLASS:MozillaDropShadowWindowClass]")
        $hMenu = _SendMessage($hWnd, $MN_GETHMENU, 0, 0)
        If _GUICtrlMenu_IsMenu($hMenu) Then
            $iCount = _GUICtrlMenu_GetItemCount($hMenu)
            For $iIndex = 0 To $iCount - 1
                $sText = StringReplace(_GUICtrlMenu_GetItemText($hMenu, $iIndex),"&","")
                ConsoleWrite($sText&@CRLF)
                If $sText == $sRequired_Text Then
                    $aPos = _GUICtrlMenu_GetItemRect($hWnd, $hMenu, $iIndex)
                    $iOldOpt = Opt("MouseCoordMode", 0)
                    MouseClick("primary", $aPos[0] + ($aPos[2] - $aPos[0]) / 10, $aPos[1] + ($aPos[3] - $aPos[1]) / 2, 1, 1)
                    Opt("MouseCoordMode", $iOldOpt)
                EndIf
            Next
        EndIf
    EndIf
    Sleep(10)
    Opt("WinWaitDelay", 250) ;250 milliseconds
    MouseMove($previousMousePos[0], $previousMousePos[1], 0)
EndFunc

 

[junkew]

Check what simplespy gives back when you hover over the contextmenu and press ctrl w.

It sounds like a simple click you can do.

[Iczer]

it seems simplespy cannot focus on context menu - red line do not appear around menu after ctrl-w. instead it appear around image behind context menu...

[junkew]

You could try with inspect.exe or the event examples to see which events are occurring.

[jushusted]

Okay, so I've worked with this for over a week now and have had success in writing a script that populates text into long text fields in Firefox and IE from an excel document. However, when I try to create a similar script in Chrome, I run into a roadblock. The accessibility tree simply does not dive down far enough into the page, it only goes as far as the div for the form, without the subsequent subtrees.

I've done some googling but haven't found a way to interact with the DOM from the accessibility tree. I'm starting to think that I'm going to have to resort to pressing down the "TAB" key multiple times to get to the correct field in the form, but that leaves all sorts of room for error.

The name for the element I'm trying to capture is "mainForm:UAL_EDIT_content:actionItemsBrowse:0:response"

Is there any way to capture this element if it's not displayed in the accessibility tree?

Edited July 30, 2015 by jushusted

[junkew]

thats the nice part from chrome. try in your addressbar: chrome://accessibility

you can see there the whole accessibility tree thats exposed and accessible for uiautomation/msaa and other screenreader apps.

If its not there it becomes a little harder but still you can go to the advanced way of doing stuff with bookmarklets/favlets thru the addressbar and that way (with any browser) you can access the whole DOM tree with short javascript syntax. This is not incorparated in the framework (its on my longlist) but you can do real fancy stuff with bookmarklets/favlets in automating html applications if the normal accessibility libraries are not helping.

bookmarklets / favlets concept that work(s)

1. type in your addressbar

• javascript:alert(document.body.innerHTML);void(0);

2. then with regular autoit commands you can get the content of the popup

thru addressbar you can insert javacript functions to the current DOM page. Only limitation is the length of text you can enter at once.
But with that you can create a websocket in javascript connecting to the autoit listening tcp server and then you can add more dynamic javascript over a socket to your webpage and do whatever your imagination wants with the DOM tree.

Biggest disadvantage is when the url changes all your dynamic added JavaScript will go away and you have to resend it thru addressbar.

experimental examples (not finished and fully working but explains the concept)

You have to type javascript: in your addressbar yourself Copy/Paste will not work

javascript:var head= document.getElementsByTagName('head')[0];
javascript:var script= document.createElement('script');
javascript:script.type= 'text/javascript';void(0);
javascript:script.src= "function fHelloWorld(){alert('I am an advanced way of hijacking your browserinformation');};";void(0);

javascript:alert(script.src);void(0);

javascript:eval(script.src);void(0);

javascript:head.appendChild(script);void(0);

Edited July 31, 2015 by junkew
added bookmarklet example links and simple example

[jushusted]

thats the nice part from chrome. try in your addressbar: chrome://accessibility

you can see there the whole accessibility tree thats exposed and accessible for uiautomation/msaa and other screenreader apps.

If its not there it becomes a little harder but still you can go to the advanced way of doing stuff with bookmarklets/favlets thru the addressbar and that way (with any browser) you can access the whole DOM tree with short javascript syntax. This is not incorparated in the framework (its on my longlist) but you can do real fancy stuff with bookmarklets/favlets in automating html applications if the normal accessibility libraries are not helping.

bookmarklets / favlets concept that work(s)

1. type in your addressbar

• javascript:alert(document.body.innerHTML);void(0);

2. then with regular autoit commands you can get the content of the popup

thru addressbar you can insert javacript functions to the current DOM page. Only limitation is the length of text you can enter at once.
But with that you can create a websocket in javascript connecting to the autoit listening tcp server and then you can add more dynamic javascript over a socket to your webpage and do whatever your imagination wants with the DOM tree.

Biggest disadvantage is when the url changes all your dynamic added JavaScript will go away and you have to resend it thru addressbar.

experimental examples (not finished and fully working but explains the concept)

You have to type javascript: in your addressbar yourself Copy/Paste will not work

javascript:var head= document.getElementsByTagName('head')[0];
javascript:var script= document.createElement('script');
javascript:script.type= 'text/javascript';void(0);
javascript:script.src= "function fHelloWorld(){alert('I am an advanced way of hijacking your browserinformation');};";void(0);

javascript:alert(script.src);void(0);

javascript:eval(script.src);void(0);

javascript:head.appendChild(script);void(0);

Wow! TIL!

Entering the javascript into the address bar worked perfectly.

Here's a working example for Chrome that used my web form, but could be adapted for use by others. It doesn't have great error handling - something I'll work on later.

 

Thanks for the help, junkew!

UAL Load - Chrome Example.au3

[bootybay]

I tried both chrome examples after changing filepath and language constants.

Both times I get an error. This one for ex5_chrome.au3:

"RedactedPath\UIAWrappers.au3" (1726) : ==> Variable must be of type "Object".:
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$t = $obj^ ERROR

And this one for ex6_chrome2.au3

Example constants please change text to english or other language to identify controls
Action 1
Action 2
"RedactedPath\UIAWrappers.au3" (1464) : ==> Variable must be of type "Object".:
$oElementStart.FindAll($treeScope, $UIA_oTRUECondition, $pElements)
$oElementStart^ ERROR

Can someone see where I went wrong?

 

I only changed following lines to this after starting chrome with --force-renderer-accessibility and using the spy on it to get the names.

$strChromeExeFolder="C:\Program Files (x86)\Google\Chrome\Application\"

;~ Make this language specific
Const $cToolbarByName = "name:=Google Chrome Toolbar"
Const $cAddressBarByName = "name:=Address and search bar"
Const $cChromeNewTabByName = "name:=New Tab"

 

[junkew]

what is example1 doing? windows language? windows version? basic simplespy information?

please post all that stuff in generic help and support in new thread

 

[hmobron]

I tried both chrome examples after changing filepath and language constants.

Both times I get an error. This one for ex5_chrome.au3:

"RedactedPath\UIAWrappers.au3" (1726) : ==> Variable must be of type "Object".:
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$t = $obj^ ERROR

And this one for ex6_chrome2.au3

Example constants please change text to english or other language to identify controls
Action 1
Action 2
"RedactedPath\UIAWrappers.au3" (1464) : ==> Variable must be of type "Object".:
$oElementStart.FindAll($treeScope, $UIA_oTRUECondition, $pElements)
$oElementStart^ ERROR

Can someone see where I went wrong?

 

I only changed following lines to this after starting chrome with --force-renderer-accessibility and using the spy on it to get the names.

$strChromeExeFolder="C:\Program Files (x86)\Google\Chrome\Application\"

;~ Make this language specific
Const $cToolbarByName = "name:=Google Chrome Toolbar"
Const $cAddressBarByName = "name:=Address and search bar"
Const $cChromeNewTabByName = "name:=New Tab"

 

I got the same problem on Windows 10, Google Chrome Versie 44.0.2403.155 m

FF got the same problem, IE is working

Edited August 21, 2015 by hmobron

[kcvinu]

 @junkew , I have played a little with simple spy. I got a pile of important information. like 

UIA_id:= <254064992>

UIA_handle:= <1312594>

UIA_ProcessId:= <3148>

UIA_ControlType:= <50033>

UIA_AutomationId:= <254064992>

UIA_NativeWindowHandle:= <1312594> 

etc.

I wonder how to deal with these info. Do i need to read all 22 pages from this forum or is there any tutorial ?. I mean a pdf or a web page or something like that. 

Edit; All i need to do is --- if i saw this post and want to save all pages in this topic for offline reading, i need to automate the task of saving each webpage one by one.  save the current page and then click the next page number button, and then wait for the page to load, and then started saving. and so on.. This was my intention. 

Edited August 21, 2015 by kcvinu

[junkew]

Start on top with the main post and references. Then understand each example as it grows in difficulty. Depending on your knowledge i assume no need to read all 22 pages. But if you are a starter it coukd be you have to read more on google microsoft iuiautomation.

[kcvinu]

Thanks @junkew 

 

[jackchen]

I've tried to make a script "Chrome_KeepLastTab.au3" to prevent Chrome from exit if the last tab is closed.

This script monitors the mouse click.If the mouse clicks on the close button on the last tab, it opens a new tab and then close the previous tab.

This script works but not stable.Can somebody point me out what I am wrong?

expandcollapse popup

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Compile_Both=y
#AutoIt3Wrapper_UseX64=Y
#AutoIt3Wrapper_Run_Au3Stripper=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "UIAWrappers.au3"

Global $doubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")
$doubleClickTime = $doubleClickTime[0]
Global $aMouseEvent[2]


;Register callback
$hMouseEvent = DllCallbackRegister("_Mouse_Event", "int", "int;ptr;ptr")
$hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0))
OnAutoItExitRegister("UnhookMouse_OnExit")

While 1
    Sleep(100)
WEnd
ToolTip("")


; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/
Func _Mouse_Event($nCode, $wParam, $lParam)
    Local $info, $mouseData, $time, $timeDiff
    If $nCode < 0 Then
        Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing
    EndIf

    $tPoint = _WinAPI_GetMousePos()
    $hWnd = _WinAPI_WindowFromPoint($tPoint)
    If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome") Then
        Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam)
    EndIf

    Local $aMousePos[2]
    $aMousePos[0] = DllStructGetData($tPoint, "X")
    $aMousePos[1] = DllStructGetData($tPoint, "Y")
    ToolTip($aMousePos[0] & @CRLF & $aMousePos[1])

    Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo"
    $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam)
    $mouseData = DllStructGetData($info, 3)
    $time = DllStructGetData($info, 5)
    $timeDiff = $time - $aMouseEvent[1]
    Switch $wParam
        Case $WM_LBUTTONUP
            If $aMouseEvent[0] = "LClick" And ($timeDiff) < $doubleClickTime Then
                $aMouseEvent[0] = "LDClick"
                $aMouseEvent[1] = $time
            EndIf

            ;Set LClick if other events haven't fired
            If $aMouseEvent[1] <> $time Then
                $aMouseEvent[0] = "LClick"
                $aMouseEvent[1] = $time
                ConsoleWrite("X = " & DllStructGetData($tPoint, "X") & ", " & "Y = " & DllStructGetData($tPoint, "Y") & @CRLF)
                KeepLastTab($hWnd, $aMousePos)
            EndIf

;~         Case $WM_RBUTTONUP
;~             If $aMouseEvent[0] = "RClick" And ($timeDiff) < $doubleClickTime Then
;~                 $aMouseEvent[0] = "RDClick"
;~                 $aMouseEvent[1] = $time
;~             EndIf

;~             ;Set RClick if other events haven't fired
;~             If $aMouseEvent[1] <> $time Then
;~                 $aMouseEvent[0] = "RClick"
;~                 $aMouseEvent[1] = $time
;~             EndIf
    EndSwitch
    Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing
EndFunc   ;==>_Mouse_Event

Func UnhookMouse_OnExit()
    _WinAPI_UnhookWindowsHookEx($hMouseHook)
    $hMouseHook = 0
    DllCallbackFree($hMouseEvent)
    $hMouseEvent = 0
EndFunc   ;==>UnhookMouse_OnExit

Func KeepLastTab($hWnd, $aMousePos = "")
    If Not IsObj($UIA_oDesktop) Then Return

    ;$oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "handle:=" & $hWnd, $treescope_children)  ; <== Why this line not work?

    $oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Chrome_WidgetWin_1", $treescope_children)
    If Not IsObj($oChrome) Then Return
    ConsoleWrite("Got Chrome object." & @CRLF)

    $oChromeTabs = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
    If Not IsObj($oChromeTabs) Then Return
    ConsoleWrite("Got Chrome Tabs object." & @CRLF)

    Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray
    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
    If Not IsObj($oCondition) Then Return

    $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements)
    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    If Not IsObj($oAutomationElementArray) Then Return

    $oAutomationElementArray.Length($iLength)
    ConsoleWrite("Tabs count: " & $iLength-1 & @CRLF)
    If $iLength > 2 Then Return ; ignore if more than one tab

    ; try to get tab close button object
    Local $UIA_pUIElement
    $oAutomationElementArray.GetElement(1, $UIA_pUIElement)
    $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oTab) Then Return

    $oTabClose = _UIA_getFirstObjectOfElement($oTab, "controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
    $t = StringSplit(_UIA_getPropertyValue($oTabClose, $UIA_BoundingRectanglePropertyId), ";", 2)
    If UBound($t) < 4 Then Return

    ConsoleWrite("Close button on the last tab: " & $t[0] & ";" & ($t[2] + $t[0]) & ";" &  $t[1]& ";" &  ($t[3] +  $t[1]) & @CRLF)
    If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+ $t[1] Then
        ; Mouse is not on the tab close button
        Return
    EndIf
    ConsoleWrite("Mouse is on the Tab close button, so we have to open a new tab to prevent Chrome from exit." & @CRLF)

    ; send Ctrl + t to open a new tab within chrome
    Send("^t")

    Sleep(100)
EndFunc