IUIAutomation MS framework automate chrome, FF, IE, ....

[NassauSky]

@junkew thanks. I couldn't find out how to adjust the demo example basic code for the new google chrome so I am working with the webdriver for now.

 

Thanks again.

[DiegoQueiroz]

On 9/20/2020 at 7:31 PM, NassauSky said:

Hi @junkew,

I've used this UDF a few years ago and it was nice.  I'm trying to run the Example #5 "Chrome example" as is and I'm seeing a unique error in the UDF:

Here is the resulting output and error:

address: 
We have 2 tabs
Title is: <New Tab>    Class   := <>    controltype:= <50019>    
Title is: <New Tab>    Class   := <>    controltype:= <50000>    
"T:\Cool\Autoit-T\Scripts\STATScraper\ScrapingOptionsWebDriverOrIUI\UIA_V0_70\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR

I see the UDF highlight the address bar in red then the program crashes with the above error.  

I'm running the following Chrome:

Google Chrome 

Version 85.0.4183.102 (Official Build) (64-bit)

 

Any tips, tricks, ideas or suggestions would be great.

 

Thanks for all ! 🙂

I am facing this problem too.

I am not able to tell why it happens. The code works for hours, and this randomly happens.

[DiegoQueiroz]

Important to tell that I am facing this problem in my own code, not the example.

Nothing related to Google Chrome or anything similar to NassauSky.

[DDodge]

Do you have an example of how to stop Chrome from Loading in order to be able to recognize 'Select a Certificate' popup. I am using AutoiTX and works for IE and Edge but Chrome is a problem for this particular scenario.  I have found no solution and have looked all over for anyone who has been able to solve this.  Thank you!

[gi_jimbo]

junkew, your work is excellent.

I'm trying to use it for a script that attempts to log into a program that's limited to a certain number of user seats.  If login fails, the program closes.  The script attempts the login every minute until logged in.  I'm using a simple loop for this.  The code below contains the function I'm using for the program launch and login attempt.

When it first launches the program, it recognizes the login window and successfully attempts the login.  If the login fails, the script successfully relaunches the program but then it doesn't appear to get the new login window because it doesn't fill out the username and password fields.  Any ideas where I'm going wrong?  I tried to clear out the oP1 and oUIElement variables but it's not making any difference.

Any help is greatly appreciated.

Func LaunchProg()
    if WinExists("Shop Manager") then Exit
    Run('\\NAS1\Shop\Login\Shop.exe')
    WinWait("Shop Manager")

    ;~ *** UIA Standard code maintainable ***
    _UIA_setVar("oP1","Title:=Shop Manager;controltype:=UIA_WindowControlTypeId;class:=Window") ;Managed Shop Manager sign-in window
    _UIA_setVar("oUIElement1","Title:=;controltype:=UIA_EditControlTypeId;class:=TextBox") ;Username field
    _UIA_setVar("oUIElement2","Title:=;controltype:=UIA_EditControlTypeId;class:=PasswordBoxEdit") ;Password box
    _UIA_setVar("oUIElement3","Title:=Login;controltype:=UIA_ButtonControlTypeId;class:=Button") ;Login button

    _ArrayDisplay($UIA_Vars)

    Sleep(700)
    _UIA_Action("oP1","setfocus") ;Window focus
    _UIA_Action("oUIElement1","setvalue", $temparray[1]) ;Set Username
    ;MsgBox(0, "Debug", "Username should be set")
    _UIA_Action("oUIElement2","setvalue", $temparray[2]) ;Set Password
    ;MsgBox(0, "Debug", "Password should be set")
    ;_UIA_Action("oUIElement3","click") ;Click Login
    ;_UIA_Action("oUIElement3","invoke") ;Click Login (invoke avoids moving the mouse)

    _UIA_setVar("oP1","")
    _UIA_setVar("oUIElement1","")
    _UIA_setVar("oUIElement2","")
    _UIA_setVar("oUIElement3","")
    _ArrayDisplay($UIA_Vars)
    ;MsgBox(0, "DEBUG", "END FUNC")
EndFunc

 

Edited November 17, 2020 by gi_jimbo
typo

[junkew]

Internally it caches the variables with an rti prefix to find 2nd time quicker but thats not working correctly so removing those will most likely help.

In general I allways try to highlight first in a loop to get windows and elements stable recognized and then start doing actual actions.

Small sleeps of 50 100 help and there is one default set if you look in the wrappers.

[topten]

junkew,

Thanx a lot for your great library.

Could you give me a hint

In firefox I am finding elements this way:

 

#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

Local $oP10=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Announcements and Site News - AutoIt Forums — Mozilla Firefox;controltype:=UIA_WindowControlTypeId;class:=MozillaWindowClass", $treescope_children)
_UIA_Action($oP10,"setfocus")
Local $oP9=_UIA_getObjectByFindAll($oP10, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)
Local $oP8=_UIA_getObjectByFindAll($oP9, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP7=_UIA_getObjectByFindAll($oP8, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=Announcements and Site News - AutoIt Forums;controltype:=UIA_DocumentControlTypeId;class:=", $treescope_children)
ConsoleWrite ("ok" &$oP6)
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Announcements and Site News;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
ConsoleWrite ("ok" &$oP0)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("AnnouncementsandSiteNews.mainwindow", "title:=Announcements and Site News;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Announcements and Site News;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

Is it possible to get the contents of the browser window into var?

[junkew]

yes, that is possible although its getting more "complicated" by using the bookmarklet trick thru the addressbar.

Approach 1 just by setting the addresbar value followed by enter with a small javascript

javascript:alert(document.body.innerHTML);

you get a messagebox with all the information you can read out with the regular autoit or UIA functions and then click the ok button.

Approach 2: Still thru addressbar add a textbox and read that out with the uia functions

to see what happens just go to www.google.com and copy/paste below in addressbar (make sure you type javascript: in front ) and press ok. 

Look then at bottom of the page and you should see a new textbox with all your information

javascript:var tb = document.createElement("input");tb.value = document.body.innerHTML;document.body.appendChild(tb);  

Approach 3: Continue on above and use eval from javascript to inject as much as you want to your HTML

approach 2 can make the textbox which you can fill with ui with javascript coding which you then finally can do eval(tb.value). This approach I have used in the past to automate "non automatable browsers"  or "closed" company browsers where you cannot install addons.

Approach 4 Probably easiest is having a addon you communicate with (never had time to do that)

         

[junkew]

Just another simple spy in powershell that helps in automating and trying out different patterns

Usage

1. Open powershell ISE (5.1 is fine)
2. Run the script it will extend the ISE environment with a new menu (but more importantly a hotkey)
3. Hover your mouse to your object of interest
4. Press ctrl+shift+w
5. You should see some output in your ISE window
6. Play with your $ae automation element from the cmdline
7. Extend the get-elementInfo function to your own ideas (run only function as it will reload directly)

#region load the most important GUI assemblies
Add-Type –AssemblyName UIAutomationClient
Add-Type –AssemblyName UIAutomationTypes
Add-Type –AssemblyName UIAutomationProvider
Add-Type –AssemblyName UIAutomationClientsideProviders

Add-Type -AssemblyName System.Windows.Forms
[void] [System.Reflection.Assembly]::LoadWithPartialName("'Microsoft.VisualBasic")
#endregion

#region addons
$spyName='Spy'
$MyEntry = $psISE.CurrentPowerShellTab.AddOnsMenu.Submenus | ?{$_.DisplayName -eq $spyName}
if ($myEntry -ne $null) {$psISE.CurrentPowerShellTab.AddOnsMenu.Submenus.Remove($MyEntry)}

$psISE.CurrentPowerShellTab.AddOnsMenu.Submenus.Add(
$spyName,
{
   get-elementInfo 
},
"CTRL+SHIFT+W")

#endregion

function get-elementInfo() {
    $Mouse = [System.Windows.Forms.Cursor]::Position
    $global:ae = [System.Windows.Automation.AutomationElement]::FromPoint([system.windows.point]::new($mouse.x,$mouse.y))
    $ae.current
}

So you could hover over your browser addressbar to get it "selected" 

And then play around with the different patterns from the blue cmdline ise window

$ae.getcurrentpattern([Windows.Automation.ValuePattern]::Pattern).setvalue("Hello world")

and to click if its a button

$ae.GetCurrentPattern([Windows.Automation.InvokePattern]::Pattern).Invoke()

[Earthshine]

way too cool. thanks. this is awesome

Edited August 31, 2021 by Earthshine

[p4sCh]

I'm trying to check if a table cell on a webpage (chrome) contains a certain text.  How can I specify which grid row (UIA_GridItemRow) I want to check?
Atm I'm doing this: 

Local $oUIRowElement = _UIA_getObjectByFindAll($oP2, "title:=Sample Text;ControlType:=UIA_DataItemControlTypeId", $treescope_subtree)
If $oUIRowElement <> "" Then
    ...

The text appears at most once per row and only in column one. This snippet finds the first appearance of the searched string. But what to do if I want to check in a specific row? Simplespy shows that the UIA_GridItemRow property is different for each row, however I don't know how to use this.
Properties of highlighted element (1st row):

expandcollapse popup

*** Detailed properties of the highlighted element ***
UIA_title:= <Sample Text>
UIA_text:= <Sample Text>
UIA_regexptitle:= <Sample Text>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_RuntimeId:= <42;855218;4;-452608>       ;~ differs
UIA_BoundingRectangle:= <905;581;485;81>    ;~ differs
UIA_ProcessId:= <15860>
UIA_ControlType:= <50029>
UIA_LocalizedControlType:= <item>
UIA_Name:= <Sample Text>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <False>
UIA_IsEnabled:= <True>
UIA_Culture:= <9>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Chrome>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <True>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <True>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <True>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <1>                       ;~ differs
UIA_GridItemColumn:= <1>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Sample Text>
UIA_LegacyIAccessibleRole:= <29>
UIA_LegacyIAccessibleState:= <0>
UIA_IsDataValidForForm:= <True>
UIA_ProviderDescription:= <[pid:15860,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (IAccessible2) (unmanaged:UIAutomationCore.DLL)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>
UIA_OptimizeForVisualContent:= <False>
UIA_IsObjectModelPatternAvailable:= <False>
UIA_AnnotationAnnotationTypeId:= <60000>
UIA_IsAnnotationPatternAvailable:= <False>
UIA_IsTextPattern2Available:= <False>
UIA_StylesStyleId:= <0>
UIA_StylesFillColor:= <0>
UIA_StylesFillPatternColor:= <0>
UIA_IsStylesPatternAvailable:= <False>
UIA_IsSpreadsheetPatternAvailable:= <False>
UIA_IsSpreadsheetItemPatternAvailable:= <False>
UIA_Transform2CanZoom:= <False>
UIA_IsTransformPattern2Available:= <False>
UIA_LiveSetting:= <0>
UIA_IsTextChildPatternAvailable:= <True>
UIA_IsDragPatternAvailable:= <False>
UIA_DragIsGrabbed:= <False>
UIA_IsDropTargetPatternAvailable:= <False>
UIA_Transform2ZoomLevel:= <1>
UIA_Transform2ZoomMinimum:= <1>
UIA_Transform2ZoomMaximum:= <1>
UIA_IsTextEditPatternAvailable:= <False>
UIA_IsPeripheral:= <False>
UIA_IsCustomNavigationPatternAvailable:= <False>
UIA_PositionInSet:= <0>
UIA_SizeOfSet:= <0>
UIA_Level:= <0>
UIA_LandmarkType:= <0>
UIA_FillType:= <0>
UIA_VisualEffects:= <0>
UIA_IsSelectionPattern2Available:= <False>
UIA_Selection2ItemCount:= <0>
UIA_HeadingLevel:= <80050>
UIA_IsDialog:= <False>

Three properties are different in 2nd row:

...
UIA_RuntimeId:= <42;855218;4;-452637>
UIA_BoundingRectangle:= <905;661;485;81>
...
UIA_GridItemRow:= <2>
...

Thanks.

[junkew]

For that the wrappers will not be much of help. You have to build your own logic either with findall or a treewalker

• Study the findall functionality
• Give it a combined condition (with and- ing them together).
• Then iterate over the array of elements given back by the findall (assumption that if you want to do partial match on text you have to do that yourself as thats not possible with the conditions out of the box)

The UIA wrappers where made to be more "friendly" in reading and filtering on partial text not for speed or for all functionality thats in the UIA framework then you have to call the direct functions, patterns, .....

Maybe this thread gives some indication 

 

 

[p4sCh]

Thanks for your response! I've took a look at the link you provided and at the Grid Control Pattern. Seems quiet complicated to me😅. Also I've encountered a problem with other items later on in the table that appear in a tooltip when you hover over certain cells. It seems I can not distinguish these items by their properties with the UIA framework.

I'm atm short on time so I've decided not to take a in depth look into the MS IUIAutomation framework. Instead I'll just use the UIA framework to download all pages with tables and then process the html files to get the desired data. (The pages are not static and need javascript in order to be evaluated, therefore a simple InetGet does not work and downloading them from the browser with UIA works fine)

[junkew]

What you potentially can do is to use the addressbar to type in small javascript bits and pieces. 

Search for bookmarklets on the internet but basically its a find addressbar, activate/focus it and do a sendkeys

sendkeys("javascript:<your short code>")

example

sendkeys("javascript:alert(document.body.innerText);")

and there are almost no limits on what you can retrieve this way from an html page

• You can add a dummy textbox on the webpage first and fill that with javascript and retrieve it with UIA commands to read the text if you do not like to get it from an alert box.
• You can put in the textbox some code and use the eval from javascript to interpret it as code (If your javascript is lenghty to get thru addressbar this is your workaround)
• Its a handy approach on systems where you only have VBA or Powershell to give small POC on testautomation on "locked" down computers

attention point

• It doesn't work when you choose new tab (Seems you have to be on an url before it works)

 

[eatfishdk]

Edit to delete, probably not the right topic.
I apologize 

Edited January 10, 2022 by eatfishdk
Not right place

[SaeidN]

Does this send text to inactive window?

I've tried this:_UIA_action("notepad.Edit,"type","this is a test{enter}")

But it activated the windows first.

 

[junkew]

There are many ways but No, type is sending it with activating window first.

You can use send

[SaeidN]

4 hours ago, junkew said:

There are many ways but No, type is sending it with activating window first.

You can use send

I think there is no command calling "send" with this IUIAuto. There is only one" sendkeys", which works same as type which is only for active windows

I've tried SendControl from au3, and returns 1, but doesn't work on this chromium based app.

[junkew]

You can set textvalues directly but then you have to study how this uia model works. Its a huge model and there are many ways to get values into a textbox completely depending on its class. Why do you not want to activate a window?

[SaeidN]

6 hours ago, junkew said:

You can set textvalues directly but then you have to study how this uia model works. Its a huge model and there are many ways to get values into a textbox completely depending on its class. Why do you not want to activate a window?

Because it's a bot that I want to run in background and I use other softwares.

I might be able to get the class of textbox using inspect elements. I'll check it later.