IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

your second window is a different popup (or a hidden one you do not see) so thats why the event fires twice (2 windows are closing)

I am not sure but this if
If $iEventId <> $UIA_Window_WindowClosedEventId Then

and later this if
If _UIA_getPropertyValue($oSender, $UIA_ControlTypePropertyId) = "50032" Then ;50032 popup

seems to be in the wrong order.

adding sleep 5000 will not help in any way

not sure if this $TreeScope_Subtree should be $TreeScope_Children

Copied your example and works on my system for Class=#32770 and X top right has name Sluiten (is in dutch)

Triggered popup in IE with javascript:alert("hello"); in addressbar

PopUp Visible - Windows Name: Bericht van webpagina
Title: Bericht van webpagina Pos: 879;484;179;171 Center = Class=#32770
Title: OK Pos: 954;610;88;26 Center = Class=Button
Title: Pos: 912;540;32;32 Center = Class=Static
Title: hello Pos: 952;548;61;17 Center = Class=Static
Title: Pos: 887;487;163;27 Center = Class=
Title: Sluiten Pos: 1003;485;49;20 Center = Class=
Closing Now

expandcollapse popup

Func oIUIAutomationEventHandler_HandleAutomationEvent($pSelf, $pSender, $iEventId) ; Ret: long  Par: ptr;int
    Dim $t
    If $iEventId <> $UIA_Window_WindowClosedEventId Then
        Local $oSender = ObjCreateInterface($pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        $oSender.AddRef()
;~         If _UIA_getPropertyValue($oSender, $UIA_ControlTypePropertyId) = "50032" Then ;50032 popup
        if _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId )= "#32770" then
            ConsoleWrite("PopUp Visible - Windows Name: " & _UIA_getPropertyValue($oSender, $UIA_NamePropertyId) & @CRLF)
            findThemAll($oSender, $TreeScope_Subtree)
            ; sleep(5000) ; used to avoid 2nd loop
        EndIf
    EndIf
    Return $S_OK
EndFunc   ;==>oIUIAutomationEventHandler_HandleAutomationEvent

Func findThemAll($oElementStart, $TreeScope)
    Local $oCondition, $oAutomationElementArray, $oUIElement
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength
    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)
    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Pos: " & _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId) & @TAB & "Center =" & _UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
            if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Sluiten" Then
            ConsoleWrite("Closing Now" & @CRLF)
            Sleep(500)
;~             dim $t2
;~             $t2=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId),";")
;~             MouseClick("left",$t2[1],$t2[2],1,0)

            _UIA_Action($oUIElement,"left")

;~             Sleep(5000)
            ExitLoop
        EndIf
    Next
EndFunc   ;==>findThemAll

and i just use _UIA_Action($oUIElement,"left")

[LarsJ]

marko001, How many event handlers have you created? If you are only watching for window open events, you should not create event handlers to handle window close, menu open or menu close. If a window is opened and the window contains a menu, you'll probably get a window open event and a menu open event. That could be the reason why you are getting two events.

[junkew]

@Marko001: I think you have the buttons top right beeing minimize, maximize, CLOSE

and a button CLOSE

I suspect when closing the popup gets hidden and as such

the 2nd button CLOSE window you receive the closing event (but then is already hidden)

[marko001]

@larsJ, @junkew

Hi guys and sorry for delay in  answer, i was out with family.

i removed all unuseful handlers:

expandcollapse popup

Func MainFunc()

    $oIUIAutomationEventHandler = ObjectFromTag("oIUIAutomationEventHandler_", $dtagIUIAutomationEventHandler, $tIUIAutomationEventHandler, True)

    If Not IsObj($oIUIAutomationEventHandler) Then Return



    $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)

    If Not IsObj($oUIAutomation) Then Return



    Local $pUIElement

    $oUIAutomation.GetRootElement($pUIElement) ; Desktop

    If Not $pUIElement Then Return



    ; Window open/close events

    If $oUIAutomation.AddAutomationEventHandler($UIA_Window_WindowOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler) Then Exit



    HotKeySet("{ESC}", "Quit")



    While Sleep(10)

    WEnd



EndFunc   ;==>MainFunc

There are no others button (visible or not) except

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow  <---- WHY IT CALLS IT TWICE??

Title:     Pos: 257;424;24;24    Center =269;436    Class=Image

Title: Network Message    Pos: 286;428;69;15    Center =320;435    Class=TextBlock

Title: (private)    Pos: 365;428;43;15    Center =386;435    Class=TextBlock

Title: Minimize    Pos: 425;424;24;24    Center =437;436    Class=Button

Title: Maximize    Pos: 449;424;24;24    Center =461;436    Class=Button

Title: Close    Pos: 473;424;24;24    Center =485;436    Class=Button

my class is

_UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId )= "ToolWindow"

so I had to change it accordingly.

If I use _UIA_Action()

expandcollapse popup

Func findThemAll($oElementStart, $TreeScope)

    Local $oCondition, $oAutomationElementArray, $oUIElement

    Dim $pCondition, $pTrueCondition

    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)

    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)

    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)

    For $i = 0 To $iLength - 1; it's zero based

        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)

        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

        ConsoleWrite("Title: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Pos: " & _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId) & @TAB & "Center =" & _UIA_getPropertyValue($oUIElement, $UIA_ClickablePointPropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)

          if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) = "Close" Then

            ConsoleWrite("Closing Now" & @CRLF)

            Sleep(1000)

            _UIA_Action($oUIElement,"left")

             Sleep(5000)

            ExitLoop

        EndIf

    Next

EndFunc   ;==>findThemAll

I get the following:

expandcollapse popup

PopUp Visible - Windows Name: Shiny.Chat.ViewModels.PrivateChatSessionViewModel

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow

Title:     Pos: 257;424;24;24    Center =269;436    Class=Image

Title: Network Message    Pos: 286;428;69;15    Center =320;435    Class=TextBlock

Title: (private)    Pos: 365;428;43;15    Center =386;435    Class=TextBlock

Title: Minimize    Pos: 425;424;24;24    Center =437;436    Class=Button

Title: Maximize    Pos: 449;424;24;24    Center =461;436    Class=Button

Title: Close    Pos: 473;424;24;24    Center =485;436    Class=Button

Closing Now

F:\AUTO-IT Projects\___Health Software Manager\_________________________2.0 __\ex3\UIAWrappers.au3 (1020) : ==> Variable used without being declared.:

$controlType=_UIA_getPropertyValue($UIA_oUIElement,$UIA_ControlTypePropertyId)

^ ERROR

PopUp Visible - Windows Name: Shiny.Chat.ViewModels.PrivateChatSessionViewModel

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow

Title: Shiny.Chat.ViewModels.PrivateChatSessionViewModel    Pos: 249;423;250;550    Center =374;698    Class=ToolWindow

Title:     Pos: 257;424;24;24    Center =269;436    Class=Image

Title: Network Message    Pos: 286;428;69;15    Center =320;435    Class=TextBlock

Title: (private)    Pos: 365;428;43;15    Center =386;435    Class=TextBlock

Title: Minimize    Pos: 425;424;24;24    Center =437;436    Class=Button

Title: Maximize    Pos: 449;424;24;24    Center =461;436    Class=Button

Title: Close    Pos: 473;424;24;24    Center =485;436    Class=Button

Closing Now

So it still replicates the loop twice and set this error using UIA_Action().

Moreover it doesn't click and it loose focus

[EDIT]:

Found that Local $controlType needed to be declared in UAIWrappers.au3.

Now it works correctly and clicks.

No matter for the 2nd pop-up. It's not a problem since when called for the 2nd time it now looks like:

Title:     Pos: 262;413;250;550    Center =387;688    Class=ToolWindow
Title:     Pos: 262;413;250;550    Center =387;688    Class=ToolWindow
Title:     Pos: 305;496;17;17    Center =313;504    Class=Image
Title: Minimize    Pos: 423;496;17;17    Center =431;504    Class=Button
Title: Maximize    Pos: 0;0;0;0    Center =-2147483648;-2147483648    Class=Button
Title: Close    Pos: 0;0;0;0    Center =-2147483648;-2147483648    Class=Button

So consider that completely solved!

I will create a template .au3 and put it as a new thread as help. I believe this could be very useful for people always looking trough "_imagesearch()" that could now use these solutions to achieve their goals.

Thanks again guys for the support.

Marco

Edited July 7, 2014 by marko001

[junkew]

please try the same for another popup with entering in addressbar

javascript:alert('Hello world');

Difficult to see what happens. Could be a program error in your application if it sends twice a WM_CLOSE message in the system.

[marko001]

Infact is what I did, I tried and it didn't duplicate. That's for now I consider it closed, probably is due to what you said some posts before

your second window is a different popup (or a hidden one you do not see) so thats why the event fires twice (2 windows are closing)

Thanks,

Marco

[marko001]

Hi all again guys,

i'm trying now to update my applications with this technology just used as seen before.

I have an event I can catch:

        if StringInStr(_UIA_getPropertyValue($oSender, $UIA_NamePropertyId),"Diagnostic: ") <> 0 Then
            ConsoleWrite("Diagnostic Window: " & _UIA_getPropertyValue($oSender, $UIA_NamePropertyId) & @CRLF)
            findThemAll($oSender, $TreeScope_Subtree)
        EndIf

 

Among all datas collected I need to focus on a grid, containing samples to be analyzed.

This grid, 6 rows visible (up to 400), at beginning (i.e. now, as soon as I catch the event) is empty and can be filled in real time by analysis lab  personnel. So I can see the rows filling the grid. I always can see 6 rows so i need to scrolldown to see the rest.

Using simplespy.au3 I see, clicking in the middle of the grid (still empty)

*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=DetailsViewDataGrid
UIA_BoundingRectanglePropertyId :=10;564;491;181
UIA_ClassNamePropertyId :=ReqExamListViewDataGrid

 

If I now click on the filled grid, clicking on an item I see:

*** Parent Information ***
Title is: <Exam_Req: Toxoplasmosis >    Class   := <DataGridRow>    controltype:= <UIA_DataItemControlTypeId>    ,<50029>    , (0000C36D)    
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=50;588;126;19
UIA_ClassNamePropertyId :=DataGridCell

 

And I see arrows to go further 6th row.

Question is: is there a way to catch when the list is updated (rows added/removed/modified) and get the datas? (also the one hidden)?

Every row contains (of my interest)

1) Number of dosis

2) Exame requested

3) Lot #

Thanks,

Marco

[LarsJ]

May be you can catch updates with the StructureChangedEventHandler. Example 3 in post 112. The example is looking for events in all windows in the Desktop. You should only be looking for events in your own window. But for a test you can probably use the example as it is.

[marko001]

@LarsJ

yes, probably I could use it.

How can I embeed the code in the previous one so I can get the event for the specific window *only* once the window is active?

(or maybe I can first check if winexists($mywindowpartialname) and then check for it?)

[Edit] I see I can update event but, for example, how can I catch text sent in the pop-up network message? (similar to netsend, just to understand)

Thanks,

M.

Edited July 8, 2014 by marko001

[LarsJ]

To get an automation element for the entire desktop use this code:

Local $pDesktop, $oDesktop
$oUIAutomation.GetRootElement( $pDesktop )
$oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "Creating Desktop automation element failed" & @CRLF )

To get an automation element for a window use this code:

Local $pWindow, $oWindow
$hWindow = WinGetHandle( "String to identify window" )
$oUIAutomation.ElementFromHandle( $hWindow, $pWindow )
$oWindow = ObjCreateInterface( $pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oWindow ) Then Return ConsoleWrite( "Creating Window automation element failed" & @CRLF )

You can only get a pop-up network message (with the UI automation framework) from the pop-up window. And you would probably use $oDesktop to identify the window.

Edited July 9, 2014 by LarsJ

[marko001]

@LarsJ

Ok I got it, now I can focus on that window, when available, to monitor events.

As I wrote in post 247, there is another window(*) i need to work with. That contains several rows, one for every exam:

Every row is read, from simplespy.au3, as

Title is: <Element: ExamSlot: Toxoplasmosis , index visualization column: 6>    Class   := <DataGridCell>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    

*** Parent Information ***

Title is: <Element: ExamSlot: Toxoplasmosis >    Class   := <DataGridRow>    controltype:= <UIA_DataItemControlTypeId>    ,<50029>    , (0000C36D)    

*** Detailed properties of the highlighted element ***

UIA_AcceleratorKeyPropertyId :=

UIA_AccessKeyPropertyId :=

UIA_AriaPropertiesPropertyId :=

UIA_AriaRolePropertyId :=

UIA_AutomationIdPropertyId :=

UIA_BoundingRectanglePropertyId :=822;626;137;20

UIA_ClassNamePropertyId :=DataGridCell

Every row has (exactly like Excel) 3 interesting cells I need to monitor (not necessary visibile in the grid (may be over the 6th row, or over the width of the window (that means scrollbar...))):

1) Number of dosis

2) Exam requested (Exam Name)

3) Lot #

I can't get with simplespy.au3 the 1) and 3). I mean, every mouse pos between the row give me same result (the one posted) except the different mouse position and index visualization column:

Is there a way to catch all the info for every cell of every row? (at least the ones i need to store in a var?)

Thanks,

Marco

(*): honestly this PART of a main window, but I can get it's infos by just monitoring that part with

     If _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) = "ExamListViewDataGrid" Then

[LarsJ]

This code should find all rows in your data grid. What does it print in the console?

Local $pCondition
$oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_DataItemControlTypeId, $pCondition )
If Not $pCondition Then Return ConsoleWrite( "CreatePropertyCondition Err" & @CRLF )

Local $pUIElementArray, $oUIElementArray, $iElements
$oWindow.FindAll( $TreeScope_Descendants, $pCondition, $pUIElementArray )
$oUIElementArray = ObjCreateInterface( $pUIElementArray, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
$oUIElementArray.Length( $iElements )
If Not $iElements Then Return ConsoleWrite( "FindAll Err" & @CRLF )
ConsoleWrite( "$iElements = " & $iElements & @CRLF )

Local $pUIElement, $oUIElement, $name
For $i = 0 To $iElements - 1
  $oUIElementArray.GetElement( $i, $pUIElement )
  $oUIElement = ObjCreateInterface( $pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oUIElement.GetCurrentPropertyValue( $UIA_NamePropertyId, $name )
  ConsoleWrite( $name & @CRLF )
Next

$oWindow is the window that contains the data grid. Edited July 10, 2014 by LarsJ

[marko001]

PM

[zovudewey]

Thanks for this!

I'm trying to automate clicking on a string in Chrome, can't make it work.

The thing is that it is not a link, as in that you can't see a link to a different HTML file. It just sends a different parameter via GET.

Anyone knows if there's a way to find a text and then click there?

This is a bit advanced for me, so any guidance would be greatly apreciated.

[junkew]

chrome://accessibility

Gives you the main info that can be retrieved

[dilipped]

This example prints the items of the Windows Explorer listview in Scite console. The selected state of the items is printed too.

expandcollapse popup

#include "..\Include\CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global $oUIAutomation

MainFunc()


Func MainFunc()

  ; Be sure to use the right class if you are on Vista or Windows 8
  Local $hWindow = WinGetHandle( "[CLASS:CabinetWClass]", "" )  ; Windows Explorer, Windows 7
  ;Local $hWindow = WinGetHandle( "[CLASS:ExploreWClass]", "" ) ; Windows Explorer, Windows XP
  If Not $hWindow Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pWindow
  $oUIAutomation.ElementFromHandle( $hWindow, $pWindow )
  If Not $pWindow Then Return

  Local $oWindow = ObjCreateInterface( $pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oWindow ) Then Return

  ListAllItems( $oWindow, 50007 ) ; 50007 = List view element
  ; Run the code in post #71 to get this value

EndFunc


Func ListAllItems( $oWindow, $iCtrlType )

  Local $pCondition
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $iCtrlType, $pCondition )
  If Not $pCondition Then Return

  Local $pUIElementArray, $oUIElementArray, $iElements
  $oWindow.FindAll( $TreeScope_Descendants, $pCondition, $pUIElementArray )
  $oUIElementArray = ObjCreateInterface( $pUIElementArray, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
  $oUIElementArray.Length( $iElements )
  If Not $iElements Then Return

  Local $pUIElement, $oUIElement, $name, $sel
  For $i = 0 To $iElements - 1
    $oUIElementArray.GetElement( $i, $pUIElement )
    $oUIElement = ObjCreateInterface( $pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    $oUIElement.GetCurrentPropertyValue( $UIA_NamePropertyId, $name )
    $oUIElement.GetCurrentPropertyValue( $UIA_SelectionItemIsSelectedPropertyId, $sel )
    ConsoleWrite( $name & "  " & $sel & @CRLF )
  Next

EndFunc

This code works for me. But I am facing one problem. Suppose I have many files such that I have to scroll to see all files. In that case $iElements will have the count of files visible only in the window and not of those beyond the scroll. How can I have a list of all the files i.e including files existing below the scroll

[jvanegmond]

Latest simplespy and UIAWrappers conflict. 

simplyspy defines AutoItSetOption("MustDeclareVars", 1)

UIAWrappers does:

func _UIA_loadCFGFile($strFname )
    Local $var
    $sections=IniReadSectionNames($strFName)

[LarsJ]

dilipped, You are absolutely right. I'll update the example within a few days to fix the issue. Edited July 18, 2014 by LarsJ

[AutID]

Nvm, found the solution

Edited July 20, 2014 by AutID

[LarsJ]

dilipped, The example you are referring to (post 78, example 15 in the list in first post), does not handle virtual items in Windows Explorer listview.

To handle virtual items see post 94, example 18 in the list in first post. I have updated the code. This example also prints the items in Windows Explorer listview and the selected state.

You should take a look at this example: Automating Windows Explorer.