IUIAutomation MS framework automate chrome, FF, IE, ....

[mycloutier]

Ok will fix it that xml should not be created in later version

Has this been fixed? I had a modified UIAWrappers where the xml's were not generated. I lost that file and for some reason I can't get this to work.

[junkew]

Thats a nice assumption that i remember something said 5 months ago. Download stuff from first post. Then retry and if not working ask for help preferably in general support section with clear reproducing script.

[mikeyr]

I am trying to update a very old script that used IE.au3 on Win2003 and IE7, we had to finally update system and that script no longer works

I am now using this IUIautomation and Firefox but having issues, I am not finding links in the pages when I look for them.  The HTML comes from ASP and unfortunately I can't change it, it seems that most links do not have a "name" attribute, just a "id". The first part I thought was working but it turns out that with only 2 fields it was just working by default, I am now trying to "click a link" but I can't find when I do a UIA_getObject... I get the messages that I cant find UserNameTextBox but as I mentioned that continues and works.  I can't get the Import part to work.

consolewrite("Log In" & @CRLF)
    $oFFToolbar=_UIA_getFirstObjectOfElement($oFF,$cToolbarByName, $treescope_subtree)
    $oFFAddressBar=_UIA_getObjectByFindAll($oFFToolbar, $cAddressBarByName  , $treescope_subtree)
    _UIA_action($oFFAddressBar,"setvalue using keys","MDID/{ENTER}")
    ;~  give some time to open website
    sleep(2000)
    $oDocument=_UIA_getObjectByFindAll($oFF,$cstrDocument, $treescope_subtree)
    $oUserNameBox=_UIA_getObjectByFindAll($oDocument,"name:=UsernameTextBox", $treescope_subtree) ;FAILS but because there are only 2 fields on the form, it works.
    if not isobj($oUserNameBox) Then
        ConsoleWrite("Failed to find UserNAme textbox even though we know its there" & @CRLF)
        _UIA_DumpThemAll($oFF,$treescope_subtree)
    EndIf
    _UIA_action($oUserNameBox,"setvalue using keys","XYZ{TAB}")
    $oPasswordTextBox=_UIA_getObjectByFindAll($oFF,"name:=PasswordTextBox", $treescope_subtree)
    _UIA_action($oPasswordTextBox,"setvalue using keys","XYZ{ENTER}")
    consolewrite("Log In Finished" & @CRLF)

    sleep(3000)

    consolewrite("Go to Import page" & @CRLF)
    ;since I can't seem to find links on the page AND I am logged in successfully, enter the URL manually (would rather go clicking around on links though)
    _UIA_action($oFFAddressBar,"setvalue using keys","http://mdid/import.aspx?id=29{ENTER}")
    sleep(3000)
    ;button created by <input name="FileUpload" id="FileUpload" type="file" /> in html that I can't change.
    $oBrowseButton=_UIA_getObjectByFindAll($oFF,"FileUpload", $treescope_subtree) ; Also tried "text:=Read File;indexrelative:=2" since button has ID but no "name"
    if not isobj($oBrowseButton) Then
        ConsoleWrite("Failed to find input button FileUpload even though we know its there" & @CRLF)
        _UIA_DumpThemAll($oFF,$treescope_subtree)
    EndIf
    ;_UIA_action($oBrowseButton,"leftclick")
    ; Move the mouse to do what I can't seem to do up above.
MouseMove(237,490)
MouseDown("left")
MouseUp("left")
    consolewrite("Import page Finished" & @CRLF)

 

[ViciousXUSMC]

So I am learning this stuff (a little bit) and I am brutally murdering the IE example and making it work piece by piece by learning what I can reference objects by and how to find them under the parent object.

I have this working so far but have a few questions.

1.) Is there a "wait to load" so that we do not have to use Sleep() and hope things have loaded before the next step executes?  

2.) If I try to run the script without closing the IE window first it gives an error instead of simply loading another window, why is that?

 

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ Make this language specific
const $cToolbarByName = "class:=Address Band Root" ;Navigation bar
const $cAddressBarByName = "name:=Address and search using Bing"
const $cIENewTabByName="name:=New Tab.*"
const $cBrowser="classname:=IEFrame"
;~ const $cDocumentWindow="controltype:=" & $UIA_DocumentControlTypeId
const $cDocumentWindow="classname:=Internet Explorer_Server"
const $cBoardItem="name:=controlsend doesn't work"
const $cEdtSearch="name:=Search.*"
const $cSearchButton="name:=Search.*; indexrelative:=3"

$strIEExeFolder=@programfilesdir & "\Internet Explorer\"
$strIEStartup=""
$strIEExe=$strIEExeFolder & "iexplore.exe "

;~ Start chrome
if fileexists($strIEExe) Then
    if not processexists("iexplore.exe") Then
        run($strIEExe & $strIEStartup,"", @SW_MAXIMIZE )
        ProcessWait("iexplore.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    consolewrite("No clue where to find IE on your system, please start manually:" & @CRLF )
    consolewrite($strIEExe & $strIEStartup & @CRLF)
EndIf

;~ Find the IE window
$oIE=_UIA_getFirstObjectOfElement($UIA_oDesktop, $cBrowser, $treescope_children)
if not isobj($oIE) Then
    _UIA_DumpThemAll($UIA_oDesktop,$treescope_children)
EndIf

;~ Make sure chrome is front window
$oIE.setfocus()

 ;get the IE toolbar
;~  $oIEToolbar=_UIA_getFirstObjectOfElement($oIE,"controltype:=" & $UIA_ToolBarControlTypeId, $treescope_subtree)
    $oIEToolbar=_UIA_getFirstObjectOfElement($oIE,$cToolbarByName, $treescope_subtree)
    if not isobj($oIEToolbar) Then
        _UIA_DumpThemAll($oIE,$treescope_subtree)
    EndIf

consolewrite("Action 5" & @CRLF)
    $oIEAddressBar=_UIA_getObjectByFindAll($oIEToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29

    if not isobj($oIEAddressbar) Then
        _UIA_DumpThemAll($oIEToolbar,$treescope_subtree)
    EndIf

    $t=stringsplit(_UIA_getPropertyValue($oIEAddressBar, $UIA_BoundingRectanglePropertyId),";")
    ;_UIA_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
    _UIA_action($oIEAddressBar,"leftclick")
    _UIA_action($oIEAddressBar,"setvalue using keys","www.autoitscript.com/{ENTER}")


    ;~  give some time to open website
    sleep(2000)
    $oDocument=_UIA_getFirstObjectOfElement($oIE,$cDocumentWindow, $treescope_subtree)
    if not isobj($oDocument) Then
        _UIA_DumpThemAll($oIE,$treescope_subtree)
    Else
        $t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
        _UIA_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
    EndIf

    sleep(500)


    $oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)
;~ All document items
    if not isobj($oForumLink) Then
        _UIA_DumpThemAll($oDocument,$treescope_subtree)
    EndIf
    _UIA_action($oForumLink,"leftclick")
    sleep(3000)

    $oDonateLink= _UIA_getObjectByFindAll($oDocument,"name:=Donate and help the website!", $treescope_subtree)
        if not isobj($oDonateLink) Then
        _UIA_DumpThemAll($oDocument,$treescope_subtree)
    EndIf
    _UIA_action($oDonateLink,"leftclick")
    sleep(3000)

As I get the hang of it, its very nifty!  I think the gist is just everything is an object and you need to filter down from a parent object to the child you need to interact with based on some identifying property and then interact with that child object as needed.

So what I need to do is find the best way to find all those objects, what interactions I can have, and keep practicing.

Somebody who really knows this should do a video tutorial, I sure will when I am good enough but I am far from right now.  

All the _UIA_DumpThemAll() where does it actually dump that info by the way?  I have not seen it in my Scite console or anywhere else. 

[junkew]

Logging depends on if you turn it on or off. There should be an xml when you turn it on. Video tutorial is a nice suggestion but learning it is just start doing it. Its hard to have one tool that can see all objects. Unfortunately there is no standard way of determining ready state it differs per browser what is smart to do. Sleep helps but checking pixels , hourglass or certain text to be on screen is more stable to synchronize.

[mikeyr]

I have given up trying to find links on the screen in my example above and just maximizing Firefox and then moving the mouse to where it needs to be.  I know this is resolution dependent but it works.

 Now my question is how to search the screen for specific text, using IE.au3 I was doing

While 1
        _IELoadWait($oIE) ; wait for any refreshes to complete
        $tmpHTML = _IEBodyReadText($oIE)
        If StringInStr($tmpHTML, "The last data import for this collection successfully finished.") Then ExitLoop
        Sleep(1000)
    WEnd

 Not finding (yet) the equivalent in this new way with Firefox.

[ViciousXUSMC]

More practice moved on to try Outlook 2010 out.

Got this working.  My issues finding the New Message Window I thought name:=string.* acted as a wildcard but for me it refused to find the window until I gave it an exact title including the space at the end of the string.

Tried invoke to click on things but guess that only works on some things so ended up using "leftclick" action.

Also if my window is minimized .SetFocus does not bring it to the front or restore/maximise.  Is there a way to do that?

Here is what I was able to scramble together, would like feedback on what to do better and such.

P.S.  You can use name (title) class, and I saw in some of the examples classname (I had not had much luck with that yet) but can you use the ID's? If so what is the proper way to state the use of an ID?

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

$oOutlook=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=rctrl_renwnd32", $treescope_children)
if not isobj($oOutlook) Then
    _UIA_DumpThemAll($UIA_oDesktop,$treescope_children)
EndIf

;~ Make sure Outlook is front window
$oOutlook.setfocus()

$oOutlookHome = _UIA_getFirstObjectOfElement($oOutlook, "name:=Home", $treescope_subtree)
_UIA_action($oOutlookHome,"leftclick")

$oNewMail =_UIA_getFirstObjectOfElement($oOutlook, "name:=New E-mail", $treescope_subtree)
_UIA_action($oNewMail,"leftclick")
Sleep(1000)

$oOutlookMessage=_UIA_getFirstObjectOfElement($UIA_oDesktop,"name:=Untitled - Message (HTML) ", $treescope_children)
$oOutlookMessage.setfocus()
$oWriteMessage =_UIA_getFirstObjectOfElement($oOutlookMessage, "name:=Insert", $treescope_subtree)
_UIA_action($oWriteMessage, "leftclick")
$oBody = _UIA_getFirstObjectOfElement($oOutlookMessage, "class:=_WwG", $treescope_subtree)
;_UIA_action($oBody, "leftclick")
_UIA_action($oBody,"setvalue using keys","Test of auto typing a new email.") ;

 

[ViciousXUSMC]

Practiced over the weekend, made good progress I think!

Came up with a couple of new questions.

To help navigate items where you may need to wait, if we have a Next button greyed out can we check its status similar to how it can be done with ControlCommand()?

While Not ControlCommand("Outlook 2007 Startup", "", "Button2", "IsEnabled", "")
    Sleep(500)
WEnd

Also while I like the red boxes for debugging and building the script, is there a global flag to change on the final script to remove the red boxes highlighting everything?

Regards,

[junkew]

Yes there are flags to turn highlighting off. Search in the udf for highlight. The concept of iuiautomation is different so hard to give a generic approach on simulating controlcommand logic and difference between ff chrome and ie browser.

[junkew]

Enabled / disabled stuff

["IsEnabled", $UIA_IsEnabledPropertyId]

_UIA_getPropertyValue($UIA_oMyObject, $UIA_IsEnabledPropertyId)

click/leftclick are synonyms

maximize window

first check property ["WindowCanMaximize", $UIA_WindowCanMaximizePropertyId]

_UIA_Action($MyWindow,"maximize") which is implemented like

        Case "maximize"
            $tPattern = _UIA_getPattern($obj2ActOn, $UIA_WindowPatternId)
            $tPattern.SetWindowVisualState($WindowVisualState_Maximized)

settings like highlight / debug

should come from a config file but seems not stable so use so workaround use true/false

;~     Some settings to use as a default
    _UIA_setVar("Global.Debug", True)
    _UIA_setVar("Global.Debug.File", True)
    _UIA_setVar("Global.Highlight", True)

using id's

["id", $UIA_AutomationIdPropertyId]

so it should be like:     id:=<yourid>

simplespy should give the actual id to use

 

[Domdom]

Hi all,

I am quite new with AutoIt but I must have missed something. I just installed AutoIt and I want to run the ISimpleDom example. The first time is says : CLSISFromString "Unknown function name" in module ISimpleDom.au3.

So I try to find some code or it and found:

Func CLSIDFromString($sGUID)
    Local $tGUID = DllStructCreate("ulong Data1;ushort Data2;ushort Data3;byte Data4[8]")
    DllCall("ole32.dll", "long", "CLSIDFromString", "wstr", $sGUID, "struct*", $tGUID)
    Return $tGUID
EndFunc   ;==>CLSIDFromString

After it complained about redefined Constants in MSAccessibility.au3. Again I found in a comment that you should comment some lines ;-(

It finally ran but I sson as I clicked on the Chrome window it said : WindowFromAccessibleObject "unknown function name"

Either I have missed something in my setup or AutoIt had changed or ...?

Thanks in advance

Dominque

[palpatin]

I DON"T NEED YOUR STUPID XML FILE! IT LITTERED MY DESKTOP.

Secondly, where is documentation of _UIA_getPropertyValue,_UIA_getObjectByFindAll,_UIA_getFirstObjectOfElement etc.

[palpatin]

why do I get 1000 views videos when I search ublock and 100000 views videos when I search adblock?

[JohnOne]

Are you some sort of internet crackpot?

Serious question.

Happy for it to be deleted.

[junkew]

@Domdom: AutoIT does not install the iuiautomation and iaccessible and isimpledom stuff you should get it from this thread. Did you try to run the examples. I will have some time coming weeks so I will try to rerun all stuff with latest AutoIT and Windows 10 combinatons. (including EDGE browser)

@Palpatin: Not sure if I should answer. The logfile can be turned off with  _UIA_setVar("Global.Debug.File", False)
  documentation should not be needed if you just study the examples and read this thread it should not be that difficult but feel free to volunteer on making a help file

[cabrunco]

Could I make a suggestion? I think it would be nice for some users like me to have some reading abilities inside _UIA_action. I myself enabled that by adding these few lines to the big Switch statement at line 1440 of the UIAWrappers.au3 file, just after DumpThemAll:

        case "PickName"; useful to get the label of a button, for example
            $retVal=_UIA_getPropertyValue($obj2ActOn, $UIA_NamePropertyId)
        case "PickValue";useful to get the value inside an edit box, for example
            $retVal=_UIA_getPropertyValue($obj2ActOn, $UIA_ValueValuePropertyId)
        case "IsEnabled"; useful to get the status of a button, if enabled, that is, clickable (returns True), or not (returns False)
            $retVal=_UIA_getPropertyValue($obj2ActOn, $UIA_IsEnabledPropertyId)
        case "IsSelected"; useful to get the status of a radio button, True for selected, False for not selected
            $retVal=_UIA_getPropertyValue($obj2ActOn, $UIA_SelectionItemIsSelectedPropertyId)

I imagine there are many other of such small additions, I only listed 4 that are specifically useful for me.

By the way, excellent job, junkew, you are an inspiration!

[junkew]

Thx for suggestions. I am still in doubt about staying on the larger growing case statemen  or make many small wrapperfunction. The idea is to have a base case statement and if an action is unknown do an eval("ext" &  straction) so implementation and extension can be independent.

Func ext......

Endfunc

[foozoor]

I have some questions about this amazing UDF:

• When will it be available inside the core of AutoIt? Is it still planned?
• Could we automate javafx, qt, gtk+, wxwidgets based gui applications in the future?
• I saw your github and I am wondering why do you use AutoIt and not C# for this library?

[junkew]

lot of work to properly document for a udf to become core.

qt 5 is working the others are dependent if those widgets implemen iuiautomation clientproviders

c# implementation examples you can find at microsoft as its their native accessibility library

[Chrishna]

Hello all , 

i'm new to Autoit,and trying to retrieve the url information from Chrome.I tried the example 5, but getting the error " Error on Opening file #include "CUIAutomation2.au3". I have included the "CUIAutomation2.au3" and "UIAWrappers.au3" in include folder of AutoIT.

Can anyone please assist me on getting the chrome url

 

Versions

Auto IT : 3.3.14

Chrome : 47

Thanks in advance