IUIAutomation MS framework automate chrome, FF, IE, ....

[x26s]

Hi Junkew,
I am trying IUIAutomation to get a tata photon dongle interface (ISP) to reconnect every time it disconnects. Here is the thread where I posted the original problem.

'?do=embed' frameborder='0' data-embedContent>>

Here is what I found with your script on this page:

expandcollapse popup

Mouse position is retrieved 638-480
At least we have an element [][QWidget]
Having the following values for all properties:
Title is: <>    Class   := <QWidget>    controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    
*** Parent Information ***
Title is: <>    Class   := <QWidget>    controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=593;458;80;36
UIA_ClassNamePropertyId :=QWidget
UIA_ClickablePointPropertyId :=
UIA_ControllerForPropertyId :=
UIA_ControlTypePropertyId :=50033
UIA_CulturePropertyId :=0
UIA_DescribedByPropertyId :=
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FlowsToPropertyId :=
UIA_FrameworkIdPropertyId :=Win32
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemContainingGridPropertyId :=
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=True
UIA_HelpTextPropertyId :=
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=False
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=True
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=False
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_ItemStatusPropertyId :=
UIA_ItemTypePropertyId :=
UIA_LabeledByPropertyId :=
UIA_LegacyIAccessibleChildIdPropertyId :=0
UIA_LegacyIAccessibleDefaultActionPropertyId :=SetFocus
UIA_LegacyIAccessibleDescriptionPropertyId :=
UIA_LegacyIAccessibleHelpPropertyId :=
UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=
UIA_LegacyIAccessibleNamePropertyId :=
UIA_LegacyIAccessibleRolePropertyId :=10
UIA_LegacyIAccessibleSelectionPropertyId :=
UIA_LegacyIAccessibleStatePropertyId :=1048580
UIA_LegacyIAccessibleValuePropertyId :=
UIA_LocalizedControlTypePropertyId :=pane
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_MultipleViewSupportedViewsPropertyId :=
UIA_NamePropertyId :=
UIA_NativeWindowHandlePropertyId :=3016084
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=38408
UIA_ProviderDescriptionPropertyId :=[pid:37052,hwnd:0x2E0594 Main:Nested [pid:38408,hwnd:0x2E0594 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=42;3016084
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionselectionPropertyId :=
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_SelectionItemSelectionContainerPropertyId :=
UIA_TableColumnHeadersPropertyId :=
UIA_TableItemColumnHeaderItemsPropertyId :=
UIA_TableRowHeadersPropertyId :=
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_TableItemRowHeaderItemsPropertyId :=
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_ValueValuePropertyId :=
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

Can you please guide as to what I must do next?
Thanks
 

[junkew]

please use the latest simplespy.au3 from first post.

1. One that gives you also some  *** Standard code ***, post that output

2. Post the screenshot of the rectangle thats highlighted after ctrl+ w.
    If its the surrounding of your button you can automate it. Is it the whole window then you can only act on coordinates

 

code looks like something

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd", $treescope_children)
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=TrayNotifyWnd", $treescope_children)
_UIA_Action($oP0,"setfocus")
_UIA_setVar("NotificationChevron.mainwindow","title:=NotificationChevron;classname:=Button")
_UIA_action("NotificationChevron.mainwindow","setfocus")

[PinetSebastien]

@Junkew 

So I got your Simplespy.au3 to work.

This inside InternetExplorer 8, using a javascript page.

And I get this from Simpleau :

==========================================================================

Mouse position is retrieved 388-374

At least we have an element [iD Interaction :][]

Having the following values for all properties: 

Title is: <ID Interaction Class   := <> controltype:= <UIA_EditControlTypeId> ,<50004> , (0000C354)

 

===========================================================================

 

But now, how can I use this information to get Autoit to Copy what is inside that field and paste it in a string variable?

[junkew]

You have some sourcecode shown after you press ctrl+w in the textbox

after 

_UIA_Action(<blabla>,"focus")

or

_UIA_Action(<blabla>,"click")

you put

_UIA_Action(<blabla>,"sendkeys", "{CTRLDOWN}c{CTRLUP}")

[x26s]

 

please use the latest simplespy.au3 from first post.

1. One that....

 

Hi junkew,

I couldnt get the spy working. got these errors. Can you plz check it out? (output in simplespy.au3 below)

>"G:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /prod /in "G:\Program Files\AutoIt3\UIA\simplespy.au3"
+>08:04:48 Starting AutoIt3Wrapper v.2.1.0.33    Environment(Language:0409  Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X86)
>Running AU3Check (1.54.22.0)  from:G:\Program Files\AutoIt3
G:\Program Files\AutoIt3\UIA\simplespy.au3(133,51) : ERROR: _UIA_NiceString(): undefined function.
    $controlIDString=_UIA_NiceString($controlIDString)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
G:\Program Files\AutoIt3\UIA\simplespy.au3(242,45) : ERROR: FileReadToArray(): undefined function.
    $UIA_CodeArray = FileReadToArray($hFileOpen)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
G:\Program Files\AutoIt3\UIA\simplespy.au3 - 2 error(s), 0 warning(s)
!>08:04:49 AU3Check ended. Press F4 to jump to next error.rc:2
>Exit code: 0    Time: 2.020

expandcollapse popup

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include <Misc.au3>
#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

#AutoIt3Wrapper_UseX64=N  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

const $AutoSpy=0 ;2000 ; SPY about every 2000 milliseconds automatically, 0 is turn of use only ctrl+w

global $oldUIElement ; To keep track of latest referenced element
global $frmSimpleSpy, $edtCtrlInfo , $lblCapture, $lblEscape, $lblRecord, $edtCtrlRecord, $msg, $x, $y, $oUIElement, $oTW, $objParent, $oldElement, $text1, $t
global $i ; Just a simple counter to measure time expired in main loop
global $UIA_CodeArray

;~ Some references for reading
;~ [url=http://support.microsoft.com/kb/138518/nl]http://support.microsoft.com/kb/138518/nl[/url]  tagpoint structures attention point
;~ [url=http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/]http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/[/url]
;~ [url=http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx]http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx[/url]

HotKeySet("{ESC}", "Close") ; Set ESC as a hotkey to exit the script.
HotKeySet("^w", "GetElementInfo") ; Set Hotkey Ctrl+M to get some basic information in the GUI
HotKeySet("^r", "GenCode") ; Set Hotkey Ctrl+R to generate some code line in a basic way

#Region ### START Koda GUI section ### Form=
$frmSimpleSpy = GUICreate("Simple UIA Spy", 801, 601, 181, 4)
$edtCtrlInfo = GUICtrlCreateEdit("", 18, 18, 512, 580)
GUICtrlSetData(-1, "")
$lblCapture = GUICtrlCreateLabel("Ctrl+W to capture information", 544, 10, 528, 17)
$lblEscape = GUICtrlCreateLabel("Escape to exit", 544, 53, 528, 17)
$edtCtrlRecord = GUICtrlCreateEdit("", 544, 72, 233, 520)
GUICtrlSetData(-1, "//TO DO edtCtrlRecord")
$lblRecord = GUICtrlCreateLabel("Ctrl + R to record code", 544, 32, 527, 17)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

_UIA_Init()

loadCodeTemplates() ; To use templates per class/controltype

; Run the GUI until the dialog is closed
While true
    $msg = GUIGetMsg()
    sleep(100)
    ;~ if _ispressed(01) Then
    ;~ getelementinfo()
    ;~ endif

    ;Just to show anyway the information about every n ms so ctrl+w is not interfering / removing window as unwanted side effects
    $i=$i+100
    if ($autoSpy<>0) and ($i>= $autoSpy) then
        $i=0
        getelementinfo()
    EndIf

    If $msg = $GUI_EVENT_CLOSE Then ExitLoop
WEnd

Func GetElementInfo()
    Local $hWnd, $i, $parentCount
    Local $tStruct = DllStructCreate($tagPOINT) ; Create a structure that defines the point to be checked.
    ;~ Local $tStruct = DllStructCreate("INT64,INT64")
    ;~  ToolTip("")
    ;~ Global $UIA_oUIAutomation            ;The main library core CUI automation reference
    ;~ Global $UIA_oDesktop, $UIA_pDesktop       ;Desktop will be frequently the starting point

    ;~ Global $UIA_oUIElement, $UIA_pUIElement  ;Used frequently to get an element
    ;~ Global $UIA_oTW, $UIA_pTW         ;Generic treewalker which is allways available

    $x=MouseGetPos(0)
    $y=MouseGetPos(1)
    DllStructSetData($tStruct, "x", $x)
    DllStructSetData($tStruct, "y", $y)
;~  consolewrite(DllStructGetData($tStruct,"x") & DllStructGetData($tStruct,"y"))

;~ consolewrite("Mouse position is retrieved " & @crlf)

    $UIA_oUIAutomation.ElementFromPoint($tStruct,$UIA_pUIElement )

    ;~ consolewrite("Element from point is passed, trying to convert to object ")
    $oUIElement = objcreateinterface($UIA_pUIElement,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)

    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    $oTW=ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        msgbox(1,"UI automation treewalker failed", "UI Automation failed failed",10)
    EndIf

    local $oParentHandle[10] ; Max 10 (grand)parents

;~  at least 1
    $i=0
    $oTW.getparentelement($oUIElement,$oparentHandle[$i])
    $oParentHandle[$i]=objcreateinterface($oparentHandle[$i],$sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If IsObj($oParentHandle[$i]) = 0 Then
        msgbox(1,"No parent", "UI Automation failed",10)
    Else
        while ($i <=9) and (IsObj($oParentHandle[$i])=true)
            $i=$i+1
            $oTW.getparentelement($oparentHandle[$i-1],$oparentHandle[$i])
            $oParentHandle[$i]=objcreateinterface($oparentHandle[$i],$sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        wend
        $parentCount=$i-1
        consolewrite($parentCount & " parents found" & @crlf)
    EndIf

    if isobj($oldUIElement) Then
        if $oldUIElement=$oUIElement then
            return
        endif
    endif
    _WinAPI_RedrawWindow(_WinAPI_GetDesktopWindow(), 0, 0, $RDW_INVALIDATE + $RDW_ALLCHILDREN) ; Clears Red outline graphics.

    GUICtrlSetData($edtCtrlInfo, "Mouse position is retrieved " & $x & "-" & $y & @CRLF)
    $oldElement=$oUIElement

If IsObj($oUIElement) Then
    local $title=_UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId)
    local $class=_UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid)
    local $controltypeName=_UIA_getControlName(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId))
    local $controltypeId=_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)
    local $controlIDString=$title
    local $nativeWindow=_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)
    local $pos=stringinstr($controlIDString,"-")

    if $pos > 0 Then
        $controlIDString=stringleft($controlIDString,$pos)
    EndIf
    $controlIDString=_UIA_NiceString($controlIDString)

;~  ConsoleWrite("At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF)
    GUICtrlSetData($edtCtrlInfo, "At least we have an element "  & "[" & $title & "][" & $class & "]" & @CRLF,1)
    $text1="Title is: <" &  $title &  ">" & @TAB _
            & "Class   := <" & $class &  ">" & @TAB _
            & "controltype:= "  & "<" &  $controltypeName &  ">" & @TAB  _
            & ",<" &  $controltypeId &  ">" & @TAB & ", (" &  hex($controltypeId )&  ")" & @TAB & @CRLF


local $codeText1=""

if $nativeWindow <> 0 Then
        $codetext1=$codetext1 & "_UIA_setVar(""" & $controlIDString & ".mainwindow"",""title:=" & $title &";classname:=" & $class & """)" & @CRLF
        $codetext1=$codetext1 & "_UIA_action(""" & $controlIDString & ".mainwindow"",""setfocus"")" & @CRLF
Else
        $codetext1=$codetext1 & ";~ First find the object in the parent before you can do something" & @CRLF
        $codetext1=$codetext1 & ";~$oUIElement=_UIA_getObjectByFindAll(""" & $controlIDString & ".mainwindow"", ""title:=" & $title &";ControlType:=" & $controltypeName & """, $treescope_subtree)" & @CRLF
        $codetext1=$codetext1 & "Local $oUIElement=_UIA_getObjectByFindAll($oP0, ""title:=" & $title &";ControlType:=" & $controltypeName & """, $treescope_subtree)" & @CRLF
        $codetext1=$codetext1 & "_UIA_action($oUIElement,""click"")" & @CRLF
EndIf

    $text1=$text1 & "*** Parent Information top down ***" & @CRLF
    local $pText1=""
    local $pCodeText2=""

;~ parentcount-1 As thats the $UIA_oDesktop
    for $i=$parentcount to 0 step -1
            $objParent=$oParentHandle[$i]
            local $ptitle=_UIA_getPropertyValue($objParent,$UIA_NamePropertyId)
            local $pclass=_UIA_getPropertyValue($objParent,$uia_classnamepropertyid)
            local $pcontroltypeName=_UIA_getControlName(_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId))
            local $pControltypeId=_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId)
            local $pDefaultExpression="""Title:=" & $pTitle & ";" & "controltype:=" & $pControlTypeName & ";" & "class:=" & $pClass & """"
            local $pNativeWindow=_UIA_getPropertyValue($objParent, $UIA_NativeWindowHandlePropertyId)
            $ptext1=$pText1 & $I & ": Title is: <" &  $ptitle &  ">" & @TAB _
                    & "Class   := <" & $pclass &  ">" & @TAB _
                    & "controltype:= " & "<" &  $pcontroltypeName &  ">" & @TAB  _
                    & ",<" &  $PcontroltypeId &  ">" & @TAB & ", (" &  hex($PcontroltypeId) &  ")" & @TAB & @CRLF
            $ptext1=$ptext1 &  $pdefaultExpression & @TAB & @CRLF
            if $i=$parentcount-1 Then
                $pCodeText2=$pCodeText2 & "Local $oP" &$i & "=_UIA_getObjectByFindAll($UIA_oDesktop, " & $pdefaultExpression & ", $treescope_children)" & @TAB & @CRLF
            Else
                if $i<=$parentcount-2 then
                    $pCodeText2=$pCodeText2 & "Local $oP" &$i & "=_UIA_getObjectByFindAll($oP" & $i+1 & ", " & $pdefaultExpression & ", $treescope_children)" & @TAB & @CRLF

                endif

            endif
            if ($pnativeWindow <> 0) and ($i<>$ParentCount) Then
                $pCodeText2=$pCodeText2 & "_UIA_Action($oP" & $i & ",""setfocus"")"  & @CRLF
            endif
    Next

    $text1=$text1 & $ptext1

    $text1=$text1 & "*** Standard code ***" & @CRLF
    $text1=$text1 & "#include ""UIAWrappers.au3""" & @CRLF
    $text1=$text1 & "AutoItSetOption(""MustDeclareVars"", 1)" & @CRLF & @CRLF

    $text1=$text1 & $pCodeText2
    $text1=$text1 & $codetext1

    $text1=$text1 & "*** Detailed properties of the highlighted element ***"
    $text1= $text1 & @CRLF & _UIA_getAllPropertyValues($oUIElement)

    GUICtrlSetData($edtCtrlInfo, "Having the following values for all properties: " & @crlf & $text1 & @CRLF, 1)

    _GUICtrlEdit_LineScroll($edtCtrlInfo, 0, 0 - _GUICtrlEdit_GetLineCount($edtCtrlInfo))

    $t=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId),";")
    _UIA_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf

EndFunc   ;==>GetElementInfo

Func Close()
Exit
EndFunc   ;==>Close



func genCode()
    local $i, $tLine
    $i=0
    while $i<>ubound($UIA_CodeArray)-1
        $i=$i+1

;~      ["name",$UIA_NamePropertyId], _
;~ Global Const $UIA_RuntimeIdPropertyId=30000
        $tLine=$UIA_CodeArray[$i]
    WEnd


    ; Display the first line of the file.
;~     MsgBox($MB_SYSTEMMODAL, "", "First line of the file:" & @CRLF & $aArray[1])

EndFunc

func loadCodeTemplates()
    Local Const $sFilePath = @scriptdir & "\codeTemplates.txt"

    Local $hFileOpen = FileOpen($sFilePath, $FO_READ)
    If $hFileOpen = -1 Then
        consolewrite( "//TODO codetemplates.txt not available An error occurred when reading the file.")
        Return False
    EndIf

;~  Read the whole file straight into an array
    $UIA_CodeArray = FileReadToArray($hFileOpen)

    FileClose($hFileOpen)
EndFunc

[PinetSebastien]

You have some sourcecode shown after you press ctrl+w in the textbox

after 

_UIA_Action(<blabla>,"focus")

or

_UIA_Action(<blabla>,"click")

you put

_UIA_Action(<blabla>,"sendkeys", "{CTRLDOWN}c{CTRLUP}")

 

I don't understand your UIA_action function.

What am I supposed to use to indentify the Textbox?

I understand that I have to use simplespy to highlight the inputbox, then I get a bunch of information about the field.

But what do I type to tell Autoit to focus on that field?

Is it the elementID?

Like _UIA_Action("ElementID","focus") ?

I use Internet explorer, so I would use Autoit to focus the Internet explorer window, that`s fine for me.

But now how Do i get the information inside the highlighted field?

[junkew]

@x26s: See first post UIA_WIP15082014_2100.zip apparently I misposted misaligned versions

@pinetSebastien: please open a thread in general help and support so others can help you on basics for AutoIT.
If you are a starter first read the help of AutoIT and tools.

Play first with calculator and notepad examples before trying to do more

Simple example

1. start simplespy.au3

2. start notepad

3. hover mouse over edit section of notepad

4. press ctrl+w

you get this

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP0=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Naamloos - Kladblok;controltype:=UIA_WindowControlTypeId;class:=Notepad", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar(".mainwindow","title:=;classname:=Edit")
_UIA_action(".mainwindow","setfocus")

and you have to transform manually to better readable and maintainable script

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("Notepad.mainwindow", "Title:=Naamloos - Kladblok;controltype:=UIA_WindowControlTypeId;class:=Notepad")
_UIA_setVar("Notepad.edit", "title:=;classname:=Edit")

_UIA_Action("Notepad.mainwindow","setfocus")
_UIA_action("Notepad.edit","setvalue","Yes I made my first hello world script and am ready to make more complex scripts with calculator")
Exit

[Berfs]

Hello,

1. I want to try your chrome example. I´ve downloaded and copied the content of the UIA_V0_4.zip into C:Program Files (x86)AutoIt3Includes. Do i have to use more of your files, or is this enought (i´m using Windows 7 Professional)?

2. I need to edit this: 

;~ Make this language specific
const $cToolbarByName = "name:=Google Chrome Toolbar"
const $cAddressBarByName = "name:=Adres- en zoekbalk"
const $cChromeNewTabByName="name:=Neuer Tab"

So what do i have to enter for the AddressBarByName and ToolbarByName (i dont have a toolbar installed and using the german chrome).

Edited August 20, 2014 by Berfs

[junkew]

probably best to download the WIP zip as it will work just no regression done on all examples.

1. run simplespy.au3 and hover over the objects you want to do something on and press ctrl+w and see the output section on the left with some direction on how your source should look like including the recognition properties.

2. install your toolbar

• sorry to say but why do you try to run a toolbar example without having a toolbar.

 

As I assume you want to navigate in chrome you start simplespy.au3

hover your mouse to the location where you enter your address and then in spy press ctrl+w and see you get some sample code which you should modify.

[Berfs]

First of all thanks for your help.

I downloaded the    UIA_WIP15082014_2100.zip and copied the files in C:Program Files (x86)AutoIt3Includes.The thing is in google chrome is no toolbar available, because the functions of it are already included. So i don´t know what to enter for the $cToolbarByName. Or do i missunderstand something?

;~ Make this language specific
const $cToolbarByName = "name:=Google Chrome Toolbar"
const $cAddressBarByName = "Adress- und Suchleiste"
const $cChromeNewTabByName="name:=Neuer Tab"

[junkew]

So whats the spy giving on the element you want to control?

If you do not have a toolbar example will ofcourse not work.

[jvanegmond]

Hi junkew,

This part taken from _UIA_Action contains a bug:

 

$t=stringsplit(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId),";")
$x=int($t[1]+($t[3]/2))
$y=int($t[2]+$t[4]/2)

_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId) returns 771; 342; 48; 25

StringSplit("771; 342; 48; 25", "; ") returns an array:

 

Row|Col 0
[0]|7
[1]|771
[2]|
[3]|342
[4]|
[5]|48
[6]|
[7]|25

so array elements 1 2 3 4 are incorrectly used in the formula above and they should be 1 3 5 and 7. Or you should give StringSplit last parameter 1 to use the full string for a split. I'm kind of surprised no one else mentioned this because it seriously breaks the functionality of _UIA_Action with clicks. Edited August 22, 2014 by Manadar

[junkew]

@Manadar: Tried myself with this code

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
Local $oP0

_UIA_setVar("RTI.SEARCHCONTEXT", $UIA_oDesktop)


$oP0=_UIA_action("Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd", "getobject")
_UIA_Action($oP0,"move")

_UIA_action("title:=Start.*;classname:=Button;controltype:=UIA_ButtonControlTypeId","move")
_UIA_action("title:=Start.*;classname:=Button;controltype:=UIA_ButtonControlTypeId","click")

;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","move")
;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","invoke")


Exit

No problems  and in the uiawrappers

Local $t
            $t = StringSplit(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId), ";")
            consolewrite(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId) & $t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)

gives

0;1078;1920;400;1078;1920;40

0;1040;54;400;1040;54;40

0;1033;54;540;1033;54;54

 

so no spaces as far as i can see

 

platform windows 7 64 bits

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\Elwin\Documents\UIA\testSimpleSpy.au3" /UserParams    
+>00:06:21 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0   Keyboard:00020409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0413)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\Elwin\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\Elwin\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.12.0)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\Elwin\Documents\UIA\testSimpleSpy.au3
+>00:06:22 AU3Check ended.rc:0
>Running:(3.3.12.0):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\Elwin\Documents\UIA\testSimpleSpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
0;1078;1920;400;1078;1920;40
0;1040;54;400;1040;54;40
0;1033;54;540;1033;54;54
+>00:06:24 AutoIt3.exe ended.rc:0
+>00:06:24 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.776

and in func _UIA_getPropertyValue i construct the string with a ; delimiter and no space (this way no hassle with safearray to array and viceversa)

However I added an additional trim/stripws

Func _UIA_getPropertyValue($obj, $id)
    Local $tval
    Local $tStr
    Local $i

    If Not IsObj($obj) Then
        SetError(1, 0, 0)
        Return "** NO PROPERTYVALUE DUE TO NONEXISTING OBJECT **"
    EndIf

    $obj.GetCurrentPropertyValue($id, $tval)
    $tStr = "" & $tval
    If IsArray($tval) Then
        $tStr = ""
        For $i = 0 To UBound($tval) - 1
            $tStr = $tStr & stringstripws($tval[$i],$STR_STRIPLEADING + $STR_STRIPTRAILING )
            If $i <> UBound($tval) - 1 Then
                $tStr = $tStr & ";"
            EndIf
        Next
        Return $tStr
    EndIf
    Return $tStr
EndFunc   ;==>_UIA_getPropertyValue

Edited August 22, 2014 by junkew

[jvanegmond]

Alright found the issue. The version of _UIA_getPropertyValue that I had has this but it's already fixed in the latest version.

Func _UIA_getPropertyValue($obj, $id)
    Local $tval
    Local $tStr
    Local $i

    If Not IsObj($obj) Then
        Return SetError(1, 0, "** NO PROPERTYVALUE DUE TO NONEXISTING OBJECT **")
    EndIf

    $obj.GetCurrentPropertyValue($id, $tval)
    $tStr = "" & $tval
    If IsArray($tval) Then
        $tStr = ""
        For $i = 0 To UBound($tval) - 1
            $tStr = $tStr & $tval[$i]
            If $i <> UBound($tval) - 1 Then
                $tStr = $tStr & "; "
            EndIf
        Next
        Return $tStr
    EndIf
    Return $tStr
EndFunc   ;==>_UIA_getPropertyValue

Edited August 23, 2014 by Manadar

[junkew]

Posted a new Work In Progress to first post

2 new examples (please post translation definitions if you have them for me in english/germany/italy or other language)

Example 31 Notepad and some basic window commands like move, resize, minimize, maximize

expandcollapse popup

#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

;~ Start the application
run("notepad.exe")

#REGION UIA_LOGICAL_NAMES
;~ Give logical names
_UIA_setVar("notepad.mainwindow","title:=((.*Kladblok)|(.*Notepad));classname:=Notepad")
_UIA_setVar("notepad.mainwindow.edit","title:=;classname:=Edit")

;~ When closing when text is changed
_UIA_setVar("Notepad.closewindow","title:=Notepad;classname:=#32770")
;~ The three buttons
_UIA_setVar("Notepad.closewindow.Save","title:=Save;classname:=CCPushButton")
_UIA_setVar("Notepad.closewindow.Don'tSave","title:=((Don't Save)|(Niet opslaan));classname:=CCPushButton")
_UIA_setVar("Notepad.closewindow.Cancel","title:=((Cancel)|(Annuleren));classname:=CCPushButton")
#ENDREGION

#REGION ACTIONS
;~ Do some actions on the logical named objects
_UIA_action("notepad.mainwindow","setfocus")
_UIA_action("notepad.mainwindow","move",300,300)
_UIA_action("notepad.mainwindow","resize",300,300)
_UIA_action("notepad.mainwindow","minimize",300,300)
_UIA_action("notepad.mainwindow","maximize",300,300)
_UIA_action("notepad.mainwindow","resize",400,400)

_UIA_action("notepad.mainwindow.edit","setfocus")
_UIA_action("notepad.mainwindow.edit","setvalue","set value: hello world")
_UIA_action("notepad.mainwindow.edit","type","type command: hello world")

_UIA_action("notepad.mainwindow","close",400,400)
if _UIA_action("Notepad.closewindow","exists") Then
    _UIA_action("Notepad.closewindow.Don'tSave","click")
Else
EndIf
#ENDREGION

Example 32 showing the three browsers doing similar things

Subtle difference is that firefox supports less on iaccessible and iuiautomation (probably IA2 is better supported)

expandcollapse popup

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

run("iexplore.exe")
run("chrome.exe --force-renderer-accessibility")
run("firefox")

;~ IE definitions
_UIA_setVar("browser1.mainwindow","classname:=IEFrame;controltype:=WindowControl;instance:=1")
_UIA_setVar("browser1.addressbar","Title:=Adresbalk;controltype:=UIA_PaneControlTypeId;class:=Address Band Root;indexrelative:=1")

;~ 2nd and 3rd browser definition (as an example)
_UIA_setVar("browser2.mainwindow","classname:=IEFrame;controltype:=WindowControl;instance:=2")
_UIA_setVar("browser3.mainwindow","classname:=IEFrame;controltype:=WindowControl;instance:=3")

;~ Chrome definitions
_UIA_setVar("browser4.mainwindow","classname:=Chrome_WidgetWin_1;controltype:=WindowControl;instance:=1")
_UIA_setVar("browser4.addressbar","title:=Adres- en zoekbalk;ControlType:=EditControl;instance:=1")

;~ Firefox definitions
_UIA_setVar("browser5.mainwindow","Title:=Mozilla Firefox.*;controltype:=WindowControl;class:=MozillaWindowClass;instance:=1")
_UIA_setVar("browser5.addressbar","title:=Search or enter address;controltype:=EditControl")

;~ _UIA_setVar("browser4.addressbar","ControlType:=EditControl;instance:=1")

;~ So first on internet explorer
_UIA_action("browser1.mainwindow","setfocus")
_UIA_action("browser1.addressbar","setfocus")
_UIA_action("browser1.addressbar","click")
_UIA_action("browser1.addressbar","setvalue", "www.autoitscript.com")
_UIA_action("browser1.addressbar","sendkeys", "{ENTER}")
;~ Just to show something happened before we go home again
sleep(5000)
_UIA_action("browser1.addressbar","sendkeys", "{BROWSER_HOME}")

;~ and now on Chrome
_UIA_action("browser4.mainwindow","setfocus")
_UIA_action("browser4.addressbar","setfocus")
_UIA_action("browser4.addressbar","click")
_UIA_action("browser4.addressbar","setvalue", "www.autoitscript.com")
_UIA_action("browser4.addressbar","sendkeys", "{ENTER}")
;~ Just to show something happened before we go home again
sleep(5000)
_UIA_action("browser4.addressbar","sendkeys", "{BROWSER_HOME}")

;~ and now on Firefox
_UIA_action("browser5.mainwindow","setfocus")
_UIA_action("browser5.addressbar","setfocus")
_UIA_action("browser5.addressbar","click")
_UIA_action("browser5.addressbar","sendkeys", "www.autoitscript.com")
_UIA_action("browser5.addressbar","sendkeys", "{ENTER}")
;~ Just to show something happened before we go home again
sleep(5000)
_UIA_action("browser5.addressbar","sendkeys", "{BROWSER_HOME}")


Exit

Edited August 25, 2014 by junkew

[tfabris]

Junkew, I am encountering a problem with this test framework. It fails to run and gets an error in the script code. 

I tried to follow the Quickstart steps listed here:  

Here is exactly what I did: 

- Downloaded and unzipped "UIA_V0_4" from the top post in this thread: http://www.autoitscript.com/forum/index.php?app=core&module=attach&section=attach&attach_id=44102

- Downloaded "simplespy.AU3" from the top post in this thread: http://www.autoitscript.com/forum/index.php?app=core&module=attach&section=attach&attach_id=44661

- Placed the downloaded simplespy into the same folder as the unzipped files from "UIA_V0_4".

- Double-clicked on simplespy which opens it in my SCITE script editor.

- Pressed the F5 key in the script editor to run the script.

Result:

>"C:Program Files (x86)AutoIt3SciTE..autoit3.exe" /ErrorStdOut "C:TempUIA_V0_4simplespy.au3"    

"C:TempUIA_V0_4UIAWrappers.au3" (150) : ==> Variable used without being declared.:

$sections=IniReadSectionNames($strFName)

^ ERROR

>Exit code: 1    Time: 0.309

 

 

Can you help me figure out how to get this running?

[tfabris]

Never mind. The solution is that I must not depend on the "UIA_V0_4" as being the latest. 

Instead I must download the that, PLUS the latest "UIA_WIP*" work in progress file and unzip that as well. After that it runs.

[Bangonkali]

Hi Guys! I've been following this thread since it started though i never really bothered to register. I'm an avid user of Autoit. I'm really interested with this project. Do you have any intention of using a source control and wiki such as github or bitbucket? I don't know what your policy/intention is in sharing this system (license) but I would really love and help out in terms of improving code, documentation and making sure this thing will really be usable especially both in small scale test automation and large scale implementation. As a test automation guy, it's really hard to find good & free UI Automation tools out there, this one is very good with huge potential. 

In terms of extending this project, I find it very useful being used along with IBM's STAF/STAX since im involved with huge deployment testings for enterprise applications. In addition I have planned an implementation of a UI management system. Im imagining a webservice where you can do hot code push to autoit clients in multiple deployments only consuming the ACTION and SETVARS and perhaps a few for ASSERTIONS. A centralized log can then be produced regarding status of Automated Test Run.

For some projects i really think this project has put autoit as a good and free alternative against qtp and codedui. 

I'm really looking forward to all the progress and plans you guys have made, to really make this project shine a Source Control, Issue Tracking and Wiki system would be very helpful. 

Hoping you continue your great work. Thanks.

PS. I really wanted to post this on github but that would be disrespectful to the original authors and avid contributors. 

Edited September 11, 2014 by Bangonkali

[junkew]

My ambition is to be going to the moon and extending it to reach mars

My vision on a testframework

• Testscripts understandable by business people but still flexible for IT personell under the hood
• Testrepository would be nice but most business people only understand excelsheets (although when I shared hundreds of them in the past that also becomes hard to share and understand)
• Supports Linux, Windows, Android, iOS, TN3270 emulation
• Supports UIA, iAccessible, iAccessible2, ATK/AT-SPI, 
• A spy should be able to deliver the standard code to interact with an element
• A spy should be able to do a record and playback and on top of that deliver maintainable code
• GUI tooling around it is nice but under the hood stuff should be flexible
• Flexible enough to graps concepts of other testframeworks

So to reach above with AutoIT will overtime be hard but as AutoIT has so many flexible UDF solutions for

• remote scripting (TCP/IP, netsend, .....) on other computers
• very good forum with support
• ....

 

Sourcecontrol

Whats the benefit in putting it on GitHUB or sourceforge compared to this forum besides some versioncontrol benefits

• For the moment this thread and maybe when I come to version 0.8 or something time to move in a shared source environment
• STAF/STAX is not my favorite (to complex for learning and understanding although the concepts are fine)
• ...

My goal for the coming weeks

• Cleanup
• Compare with IE.UDF and FF.AU3 and CHROME.AU3 actions to be on same level
• Standardize for UDF guidelines (please comments are welcome on current source to become more UDF compliant)
• Document the actions provided so far
  
  "leftclick", "left", "click", "leftdoubleclick", "leftdouble", "doubleclick", _
  "rightclick", "right", "rightdoubleclick", "rightdouble", _
  "middleclick", "middle", "middledoubleclick", "middledouble", "mousemove", "movemouse"
  "setvalue","settextvalue"
  "setvalue using keys"
  "getvalue"
  "sendkeys", "enterstring", "type", "typetext"
  "invoke"
  "focus", "setfocus", "activate"
  "close"
  "move","setposition"
  "resize"
  "minimize", "maximize", "normal", "close", "exist", "exists"
   "searchcontext", "context"
  "highlight"
  "getobject","object"
  "attach"
  "capture","screenshot"

[Bangonkali]

Awesome work man. 

As for me, I've been looking for alternative free frameworks as ldtp and what you're working on. My orgs current setup is significantly hugely expensive since we're using Microsoft Test Manager, Microsoft System Center/SCVMM and Microsoft CodedUI that is with Premium & Ultimate versions of Visual Studio. Yes I think we've already got all your notes/vision covered but at a very huge cost comparable to our huge deployments involved. That is why I've been actively looking forward to open source projects in this area. 

It was great to know your planned uptake in this area. Really looking forward to further improvements. Thanks!