IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

Posted new version 0.7 udf and examples in first post

[junkew]

Regarding the variable definitions with _UIA_setvar like simplespy is given those can be put in UIA.CFG like this

Example for notepad in 2 languages english/dutch

[notepad]
mainwindow=title:=((Naamloos.*Kladblok)|(Untitled.*pad));classname:=Notepad
mainwindow.edit=title:=;classname:=Edit
closewindow.save=title:=Save;classname:=CCPushButton
closewindow.Don'tSave=title:=((Don't Save)|(Niet opslaan));classname:=CCPushButton
closewindow.Cancel=title:=((Cancel)|(Annuleren));classname:=CCPushButton

Above instead of 

;~ Give logical names
_UIA_setVar("notepad.mainwindow","title:=((Naamloos.*Kladblok)|(.*Notepad));classname:=Notepad")
_UIA_setVar("notepad.mainwindow","title:=((Naamloos.*Kladblok)|(.*Npad));classname:=Notepad")
_UIA_setVar("notepad.mainwindow.edit","title:=;classname:=Edit")

;~ When closing when text is changed
_UIA_setVar("Notepad.closewindow","title:=((Kladblok)|(Notepad));classname:=#32770")
;~ The three buttons
_UIA_setVar("Notepad.closewindow.Save","title:=Save;classname:=CCPushButton")
_UIA_setVar("Notepad.closewindow.Don'tSave","title:=((Don't Save)|(Niet opslaan));classname:=CCPushButton")
_UIA_setVar("Notepad.closewindow.Cancel","title:=((Cancel)|(Annuleren));classname:=CCPushButton")

you just put the definitions in the separate configuration uia.cfg file so you can split your page definitions away from your coding.

Example 31 for notepad will then change to something like below

#include <Array.au3>
#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

; Run Notepad
Run("notepad.exe")

; Wait 10 seconds for the Notepad window to appear.
WinWaitActive("[CLASS:Notepad]", "", 10)

#REGION UIA_LOGICAL_NAMES
;~ see UIA.CFG section notepad
#ENDREGION
;~_arraydisplay($UIA_Vars)

#REGION ACTIONS
;~ Do some actions on the logical named objects
_UIA_action("notepad.mainwindow","setfocus")
_UIA_action("notepad.mainwindow","move",300,300)
_UIA_action("notepad.mainwindow","resize",300,300)
_UIA_action("notepad.mainwindow","minimize",300,300)
_UIA_action("notepad.mainwindow","maximize",300,300)
_UIA_action("notepad.mainwindow","resize",400,400)

_UIA_action("notepad.mainwindow.edit","setfocus")
_UIA_action("notepad.mainwindow.edit","setvalue","set value: hello world")
_UIA_action("notepad.mainwindow.edit","type","type command: hello world")

_UIA_action("notepad.mainwindow","close",400,400)
if _UIA_action("Notepad.closewindow","exists") Then
    _UIA_action("Notepad.closewindow.Don'tSave","click")
Else
EndIf
#ENDREGION

 

[Viszna]

Welcome
My problem is the same objects . 

I have an intranet page where I have to click on the next links in the table.
The identification of the link objects is not different.

UIA always finds the first object in which to indicate the next link?

The table that I serve has 200 rows. I can not put screenshots (confidential data).

To show what the problem I have created, the test.html and Simple UIA Spy code

test.html

How do you click View in the Piotr line?

expandcollapse popup

#include-once
#include<UIAWrappers.au3>

Global $WindowChromePID
$sLink = @ScriptDir & "\test.html"

If FileExists("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe") Then
    $WindowChromePID = Run("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe --force-renderer-accessibility " & $sLink)
ElseIf FileExists("C:\Program Files\Google\Chrome\Application\chrome.exe") Then
    $WindowChromePID = Run("C:\Program Files\Google\Chrome\Application\chrome.exe --force-renderer-accessibility " & $sLink)
Else
    MsgBox(8192+16, "Error","Chrome.exe not found")
EndIf
Sleep(5000)


_UIA_setVar("oP1","Title:=test.html - Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1")  ;test.html - Google Chrome
_UIA_setVar("oP2","Title:=;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")  ;
_UIA_setVar("oP3","Title:=;controltype:=UIA_TableControlTypeId;class:=")    ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=View;controltype:=UIA_DataItemControlTypeId;class:=") ;View
_UIA_setVar("oP6","Title:=View;controltype:=UIA_HyperlinkControlTypeId;class:=")    ;View

;~ $oUIElement=_UIA_getObjectByFindAll("View.mainwindow", "title:=View;ControlType:=UIA_TextControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=View;controltype:=UIA_TextControlTypeId;class:=") ;ControlType:=UIA_TextControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

_UIA_Action("oP1","highlight")
;~ _UIA_Action("oP1","setfocus")
_UIA_Action("oP2","highlight")
;~ _UIA_Action("oP2","setfocus")
_UIA_Action("oP3","highlight")
;~ _UIA_Action("oP3","setfocus")
_UIA_Action("oP4","highlight")
;~ _UIA_Action("oP4","setfocus")
_UIA_Action("oP5","highlight")
;~ _UIA_Action("oP5","setfocus")
_UIA_Action("oP6","highlight")
;~ _UIA_Action("oP6","setfocus")

;~ _UIA_action("oUIElement","highlight")
_UIA_action("oUIElement","click")

Screen from MS Inspect

[junkew]

Use index or indexrelative. Search in source of examples for an example.

[BigG3]

I'm having a bit of trouble getting the library to NOT create log xml files in the LOG directory.  I want the script to run daily, so I don't need log files every day.

I have added this line to the code:

; Turn off debug log file generation - Doesn't seem to work either.
_UIA_setVar("Global.Debug",false)

 

As well as setting these two in UIA.CFG:

[Global]
Debug=false
Debug.file=false

 

But I keep getting XML files in the LOG directory named:  20180729-130940736.XML

With this content:

<?xml version="1.0" encoding="UTF-8"?><log space="preserve"><logline level="5" timestamp="20190729-130940737"> <information> Information _UIA_VersionInfo version: T0.7-0; Release date: 20190501; OS Version: WIN_7</information></logline>
</log>
 

Help?

 

[junkew]

Not sure and no computer with me to check. Just scan the au3-file to see where its created.

[P00PDOG]

I'm working on a simple login script where I create a GUI and call a function that opens chrome, finds an element types a username goes to the next element and types the password. when I run the function again I get the below error so I am guessing the array needs to be cleared out before I run the function again? I basically used simplespy to generate the code. It works perfectly when I just re-run the function in Scite, but when I include it into my GUI I get the below error on the second run, first run is fine.

here is the function only

expandcollapse popup

Func website()

;~ Start chrome

If processexists($strChromeExeOnly) Then
    WinClose("website","")
    sleep (1000)
    run($strChromeExe & $strChromeStartup & " " &"--user-data-dir=C:\AutoLogontestCrome\userData" & " " & "--disable-single-click-autofill" & " " & $MedtronicURL,"", @SW_MAXIMIZE )
    ProcessWait($strChromeExeOnly)
    ;~ Just to give some time to start
    sleep(3000)

Else
    run($strChromeExe & $strChromeStartup & " " &"--user-data-dir=C:\AutoLogontestCrome\userData" & " " & "--disable-single-click-autofill" & " " & $MedtronicURL,"", @SW_MAXIMIZE )
    ProcessWait($strChromeExeOnly)
    ;~ Just to give some time to start
    sleep(3000)
EndIf

local $oChrome=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
$oChrome.setfocus()


AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=website - Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1")    ;website Network - Google Chrome
_UIA_setVar("oP2","Title:=Medtronic CareLink Network;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")    ;website
_UIA_setVar("oP3","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP6","Title:=;controltype:=UIA_TableControlTypeId;class:=")    ;
_UIA_setVar("oP7","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP8","Title:=CareLink® Network • Sign In Please enter your User ID and password to enter the Medtronic CareLink Clinician Website. User ID: Password: Sign In Patient Information Site >> Forgot Password?;controltype:=UIA_DataItemControlTypeId;class:=") ;website • Sign In Please enter your User ID and password to enter the Website. User ID: Password: Sign In bla bla >> Forgot Password?
_UIA_setVar("oP9","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP10","Title:=;controltype:=UIA_TableControlTypeId;class:=")   ;
_UIA_setVar("oP11","Title:=;controltype:=UIA_CustomControlTypeId;class:=")  ;

;~ $oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_DataItemControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=;controltype:=UIA_DataItemControlTypeId;class:=") ;ControlType:=UIA_DataItemControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

_UIA_Action("oP1","")
;;_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
;;_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
;;_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
;;_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
;;_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
;;_UIA_Action("oP6","setfocus")
;~_UIA_Action("oP7","highlight")
;;_UIA_Action("oP7","setfocus")
_UIA_Action("oP8","")
;;_UIA_Action("oP8","setfocus")
;~_UIA_Action("oP9","highlight")
;;_UIA_Action("oP9","setfocus")
;~_UIA_Action("oP10","highlight")
;;_UIA_Action("oP10","setfocus")
;~_UIA_Action("oP11","highlight")
;;_UIA_Action("oP11","setfocus")
_UIA_action("oP11","click")
Send("blabla")
sleep(500)
_UIA_action("oP10","click")
Send("blabla")
sleep(500)
;Send("{ENTER}")
;GUICtrlSetState($Button2, $GUI_ENABLE)
EndFunc

 

Output

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\AutoLogontestCrome\login_GUI_v2.au3" /UserParams    
+>19:35:18 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.4.1.2.0   Keyboard:00000409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\user1\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user1\AppData\Local\AutoIt v3\SciTE
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\AutoLogontestCrome\login_GUI_v2.au3
+>19:35:19 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\AutoLogontestCrome\login_GUI_v2.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"C:\Program Files (x86)\AutoIt3\Include\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR
->19:36:11 AutoIt3.exe ended.rc:1
+>19:36:11 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 53.91

[junkew]

You are right I basically cache things in the array by prefixing it with rti. For runtime type information whenever the object is found it reuses that info. As I am on holiday without laptop hard to say in which example I remove items from array that start with rti.

[Heinzl]

Did anybody recently try to run the Firefox automation example? After doing some modifications, I have been able to access tree elements like tabs and click to them, so basic things work. But I can't access the document itself, I always get ""Action 6 failed". Any Ideas?

Thanks for any help.

Heinz

[junkew]

Try if simplespy highlights your document area and or elements. Most likely during upgrades they change element order and as such your document identification or element tree changed. Which ff version are you running.

[Heinzl]

On 8/8/2019 at 7:30 PM, junkew said:

Try if simplespy highlights your document area and or elements. Most likely during upgrades they change element order and as such your document identification or element tree changed. Which ff version are you running.

Today, I had the chance to do some more testing with simplespy.au3:

• I used machine #1 with 64bit Windows 7 build 7601 and almost current Firefox 68.0 64bit
• I ran the same tests on machine #2 with 32bit Windows 7 build 7601 and current Firefox 68.0.1 32bit
• Same Tests were made on another machine #3 with 64bit Windows 10 1803 and current Firefox 68.0.1 32bit

More details about my environment:

• All machines have German versions of Windows and Firefox.
• On the 64bit machines #1 and #3, I executed simplespy.au3 in both modes x86 and x64, using "Run Script (x64)" and "Run Script (x86)"
• During the tests, Firefox is already started when I run simplespy.au3
• I have to confirm an UAC Dialog, before simplespy.au3 starts
• I have autoit version 3.3.14.5 installed on all machines

How I did the tests:

• I can place my mouse over almost any Element of Firefox (tried toolbar, address bar, favourite bookmarks), press Ctrl+W and get information about that element in the simplespy window on any machine.
• If i place my mouse anywhere in the content area of the browser (where the web content is displayed) and press Ctrl+W there, simpplespy.au3 crashes on machines #1 and #2 (with the additional information as shown below). On machine #3, everything works as expected.

Conclusion:

• Access to content area of firefox doesn't seem to be possible with windows 7, but works fine with windows 10

Do you have any idea what's going wrong with Windows 7?

 

Appendix - Additional Error Information

Machine #1, 32bit mode and Machine #2, 32bit mode

Problemsignatur:
  Problemereignisname:    APPCRASH
  Anwendungsname:    AutoIt3.exe
  Anwendungsversion:    3.3.14.5
  Anwendungszeitstempel:    5aaa71bf (different each time, of course)
  Fehlermodulname:    uiautomationcore.dll
  Fehlermodulversion:    7.0.0.0
  Fehlermodulzeitstempel:    56f58458 (different each time, of course)
  Ausnahmecode:    c0000005
  Ausnahmeoffset:    00053671
  Betriebsystemversion:    6.1.7601.2.1.0.256.48
  Gebietsschema-ID:    1031
  Zusatzinformation 1:    0a9e
  Zusatzinformation 2:    0a9e372d3b4ad19135b953a78882e789
  Zusatzinformation 3:    0a9e
  Zusatzinformation 4:    0a9e372d3b4ad19135b953a78882e789

Machine #1, 64bit mode

Problemsignatur:
  Problemereignisname:    APPCRASH
  Anwendungsname:    AutoIt3_x64.exe
  Anwendungsversion:    3.3.14.5
  Anwendungszeitstempel:    5aaa71d3
  Fehlermodulname:    uiautomationcore.dll
  Fehlermodulversion:    7.0.0.0
  Fehlermodulzeitstempel:    56f58b9a
  Ausnahmecode:    c0000005
  Ausnahmeoffset:    000000000005bffe
  Betriebsystemversion:    6.1.7601.2.1.0.256.48
  Gebietsschema-ID:    1031
  Zusatzinformation 1:    a04c
  Zusatzinformation 2:    a04c4c7ad5cbb615489808b507a01638
  Zusatzinformation 3:    c91a
  Zusatzinformation 4:    c91af45283a621aa3e02406f3227c8f3

 

 

[InnI]

1 hour ago, Heinzl said:

Access to content area of firefox doesn't seem to be possible with windows 7

Browse to about:config, confirm the risk, find all parameters browser.tabs.remote.autostart, browser.tabs.remote.autostart.1, browser.tabs.remote.autostart.2 etc, turn them all to false and restart Firefox. Now you will see only one process Firefox.exe but IUIAutomation should work.

Oops... this worked on version 67 but does not work on 68. Sorry.

Edited August 12, 2019 by InnI

[junkew]

Hard to say but I feel its a firefox issue.

 

You can doublecheck with inspect.exe whats happening and if you can reproduce report a bug with mozilla firefox and/or feature request.

I am on W10 nowadays and not working iwth W7 machines so hard to help in anlalysing.

validate these links on FF. I have been reading this and see litlle info on UIA but they support MSAA (and UIA puts a layer if needed overMSAA)

about:config in addressbar and then type accessibility maybe you can tweak some settings

• http://asurkov.blogspot.com/2012/06/uia-implementation-in-firefox.html
• https://wiki.mozilla.org/Accessibility/Windows.Next#3._Kill_buffers_by_using_UI_Automation (interesting page to read and especially last line)
• https://support.mozilla.org/en-US/kb/accessibility-features-firefox-make-firefox-and-we#w_compatibility-with-assistive-technologies
• https://developer.mozilla.org/en-US/docs/Mozilla/Accessibility/Gecko_info_for_Windows_accessibility_vendors
• https://wiki.mozilla.org/Accessibility/Electrolysis2
• https://github.com/mozilla/gecko-dev/blob/master/accessible/windows/msaa/CompatibilityUIA.cpp

 

 

[Heinzl]

Thank you junkew and InnI for your assistance!

I won't spend any more time on Windows 7 and will continue work with Windows 10.

Now that I am able to find the content object of type UIA_DocumentControlTypeId, I have another question:

Is there a way to "convert" this to something that is more similar to a DomTree (like ISimpleDom)? Or, in other words: How do I acces porperties like innerHTML or innerText?

Thank you for any assistance

[junkew]

This is a little harder. If I have that need in firefox or chrome I use JavaScript in the addressbar to either directly get the info or add an textboxelement to the page with the infon

[junkew]

This is an example on how to get whole DOM thru javascript and addressbar in chrome/ff/edge/IE.

• Its easier in general to test this information thru IE (when its correct there chances are low its different in other browser)
• I use UIA mainly for "black box" testing and then you are less interested in the html code itself but as you can see you can get to the information you want.
• In 99% of the cases I just alert what I need to a popup/message alertbox and read that with "standard" functions.
• Below example makes an additional textarea where I can put anything I want with multiple javascript miniscripts I execute thru addressbar

This textarea I then read with _UIA_action( <ID of textarea>,"getvalue). So after below script is finished you should see at the bottom of your html page a huge textarea( normally I make them 1x1 pixel)

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oChromeWindow",".*Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1") ;AutoIt General Help and Support - AutoIt Forums - Google Chrome
_UIA_setVar("oAdressbar","Title:=Adres") ;ControlType:=UIA_ButtonControlTypeId;classname:=")

_UIA_Action("oChromeWindow","setfocus")

;~ _UIA_action("oAdressbar","highlight")
;~ _UIA_action("oAdressbar","setfocus")

local $jsSource="javascript:"
$jsSource&="var UIAel=document.createElement(""textarea"");"
$jsSource&="UIAel.cols=80;"
$jsSource&="UIAel.rows=40;"
$jsSource&="UIAel.value=document.body.innerHTML;"
$jsSource&="document.body.appendChild(UIAel);"
$jsSource&="void(0);"

consolewrite(stringreplace($jsSource,";",";"&@CRLF))
;~ Alternative
;~ $jsSource="javascript:alert(document.body.innerHTML;"
;~ $jsSource&="void(0);"

_UIA_action("oAdressbar","setvalue",$jsSource)
sleep(1500)
_UIA_action("oAdressbar","sendkeys","{ENTER}")

more advanced

never needed it and only made a demo once with websockets

https://www.tutorialspoint.com/html5/html5_websocket.htm

with javascript approach you can start a websocket in the page and communicate with it from AutoIT

 

Edited August 13, 2019 by junkew
added websocket info

[Heinzl]

Thank you for your sample code. Unfortunately, javascript is not an option in my case. So I try to slowly approach the ISimpleDomNode. According to https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Accessibility/AT-APIs/ImplementationFeatures/MSAA#ISimpleDOMNode this should be possible in Firefox. In your sample file ex30_ISimpleDom.au3, I found that you are doing quite the same thing for chrome. So I tried to create a very simple test, which looks like this:
 

; Test for Firefox Access

#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

; find Firefox Window
ConsoleWrite("Find Firefox ... ")
Local $oFF = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=MozillaWindowClass", $treescope_children)
If IsObj($oFF) Then
    ConsoleWrite("found" & @CRLF)
Else
    _UIA_DumpThemAll($UIA_oDesktop, $treescope_children)
EndIf

; find Google TAB
ConsoleWrite("Find Document ... ")
Local $oDoc = _UIA_getObjectByFindAll($oFF, "Title:=.*Google Search;controltype:=UIA_DocumentControlTypeId", $treescope_subtree)
If IsObj($oDoc) Then
    ConsoleWrite("found" & @CRLF)
Else
    _UIA_DumpThemAll($oFF, $treescope_subtree)
EndIf

; convert to Accessible
Local $oAccessible = ObjCreateInterface($oDoc, $sIID_IAccessible, $dtagIAccessible)
If IsObj($oAccessible) Then
    ConsoleWrite("Converted to IAccessible" & @CRLF)
else
    ConsoleWrite("Conversion failed" & @CRLF)
EndIf

I would expect to get an Instance of IAccessible, but I get the following output instead:

Find Firefox ... found
Find Document ... found
Conversion failed

Maybe this is because ObjCreateInterface expects a pointer not an object?

Can you give any advice? Thanks in advance

Heinz

[junkew]

AccessibleObjectFromPoint

See msaccessibility.au3 as part of examples.zip. yes most likely you need the pointer.

[seadoggie01]

I get an error right off the bat with a new copy of UIAWrappers.au3... it says "Variable used without being declared" and it points to "$UIA_OptimizeForVisualContentPropertyId"... Am I missing a required include or something?

Edit: (I'm following the quickstart guide with UIA_V0_70 and AutoIt V3.3.14.5)

Edit 2: Even stranger is that I can run the 'simple uia spy' without issue. There must a missing include file.

Edit 3: No idea how it happened, but apparently I had an old CUIAutomation2 file in my includes which didn't have that variable. Worked fine in the download folder because it was in the zip.

Edited September 18, 2019 by seadoggie01

[takayo72]

support win10 edge chromium  browser ?