IUIAutomation Parent Obj

[MichaelHB]

Hello,

First let me apologize if i state something wrong, english is not my native language. I'm using IUIAutomation to get the parent data of a element in the Google Chrome. I can't use IE/FF (for compatibility problems). I know i can use the function _UIA_action($Element,"getvalue"), but this will send a CTRL+C, and i'm avoiding using keystrokes. I found another approach, to get the data (text actually) that i want, i have adapted a function of the simplespy and it's working, but is there a simple method or small function than the one i'm using?

To summarize what i trying to do: in this website, above each post there is the information when it was posted ("Posted 26 March" for exemple), the expression "Posted" is static, so with that info i user to get to the parent element and retrieve the complete data (Posted 26 March). If you put your mouse exactly on top of the word "Posted" and the use the simplespy you will notice that only the word "Posted" will be highlighted. Let me show my adaptation:

#include "UIAWrappers.au3"

Global $oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Chrome_WidgetWin_1", $treescope_children) ; this is a generic approach, grabbing the Chrome and then looking for the static data (you can see below that in this case i'm using "title:=Time = " as my static object).

$FullData = GetData()

Func GetData()
    Local $i, $parentCount, $cUIA_MAXDEPTH = 25
    Local $oParentHandle[$cUIA_MAXDEPTH] ; Max number of (grand)parents

    Local $oP = _UIA_getObjectByFindAll($oChrome, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
    Local $oUIElement = _UIA_getObjectByFindAll($oP, "title:=Time = ;ControlType:=UIA_TextControlTypeId", $treescope_subtree)

    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    Local $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(0, "UI automation treewalker failed", "UI Automation failed failed")
    EndIf

    ;;~  at least 1 assumed (assuming we are not spying the desktop)
    $i=0
    $oTW.GetParentElement($oUIElement, $oParentHandle[$i])
    $oParentHandle[$i] = ObjCreateInterface($oParentHandle[$i], $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If IsObj($oParentHandle[$i]) = 0 Then
        MsgBox(0, "No parent", "UI Automation failed could be you spy the desktop")
    Else
        While ($i <= $cUIA_MAXDEPTH-1) And (IsObj($oParentHandle[$i]) = True)
            $i = $i+1
            $oTW.GetParentElement($oParentHandle[$i-1], $oParentHandle[$i])
            $oParentHandle[$i] = ObjCreateInterface($oParentHandle[$i], $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        WEnd
        $parentCount = $i-1
    EndIf

    Return _UIA_getPropertyValue($oParentHandle[0], $UIA_NamePropertyId)
EndFunc

As i still don't understand these actions/function completely, i don't know if this is the best way to do this without using send keystrokes. The reason why I put only part of the code is because of the size and because the website I work requires a login.

Thank you in advance for any help / opnion. 

[junkew]

You seem to be referring to an html page. Most likely a bookmarklet in your addressbar can most easily do what you want.

Simple example that shows you all text of your page. You can make this as advanced as you want with oneliners of javascript

javascript:alert(document.body.innerText);

Scraping popupbox should not be difficult

[MichaelHB]

Hi junkew,

 

Ty for your interest.  I thought of that possibility, but then I would have to use the commands to click in the address bar and then use send to send the java command. I'm trying to avoid send because for some reason that I still could not understand in Chrome sometimes i keys get virtually stuck (i try many things to fix this; using down and up with the keys; change delay; _WinAPI_Keybd_Event; general unstuck method in the wiki) nothing worked.

 

My intent in the first post was clear? I dont't know if i explained it right. Let me give you a working example in Chrome.

expandcollapse popup

#include "UIAWrappers.au3"

Global $Data = "title:=Posted " ; with only "Posted " we will retrive all

Global $oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Chrome_WidgetWin_1", $treescope_children) ; this is a generic approach, grabbing the Chrome and then looking for the static data (you can see below that in this case i'm using "title:=Time = " as my static object).

If IsObj($oChrome) Then
    $oChrome.SetFocus()
    Sleep(500)

    $oChromeTabs = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
    If not isobj($oChromeTabs) Then
        _UIA_DumpThemAll($oChrome, $treescope_subtree)
    EndIf

    $oChromeYukonTab = _UIA_getObjectByFindAll($oChromeTabs, "name:=IUIAutomation Parent Obj - AutoIt General Help and Support - AutoIt Forums", $treescope_subtree)
    If not isobj($oChromeYukonTab) Then
        _UIA_DumpThemAll($oChromeTabs, $treescope_subtree)
    EndIf
    _UIA_action($oChromeYukonTab,"leftclick")
    sleep(500)
EndIf


$FullData = MsgBox(0, "Full DATA", GetData())

Func GetData()
    Local $i, $parentCount, $cUIA_MAXDEPTH = 25
    Local $oParentHandle[$cUIA_MAXDEPTH] ; Max number of (grand)parents

    Local $oP = _UIA_getObjectByFindAll($oChrome, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
    Local $oUIElement = _UIA_getObjectByFindAll($oP, $Data&";ControlType:=UIA_TextControlTypeId", $treescope_subtree)

    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    Local $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(0, "UI automation treewalker failed", "UI Automation failed failed")
    EndIf

    ;;~  at least 1 assumed (assuming we are not spying the desktop)
    $i=0
    $oTW.GetParentElement($oUIElement, $oParentHandle[$i])
    $oParentHandle[$i] = ObjCreateInterface($oParentHandle[$i], $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If IsObj($oParentHandle[$i]) = 0 Then
        MsgBox(0, "No parent", "UI Automation failed could be you spy the desktop")
    Else
        While ($i <= $cUIA_MAXDEPTH-1) And (IsObj($oParentHandle[$i]) = True)
            $i = $i+1
            $oTW.GetParentElement($oParentHandle[$i-1], $oParentHandle[$i])
            $oParentHandle[$i] = ObjCreateInterface($oParentHandle[$i], $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        WEnd
        $parentCount = $i-1
    EndIf

    Return _UIA_getPropertyValue($oParentHandle[0], $UIA_NamePropertyId)
EndFunc

Exit

 

For learning purposes junkew, how can i read all the data in the popupbox? Can you point me to a generic exemple?

 

Ty again 

[junkew]

Within chrome you can see whole uia accessibility tree by typing chrome://accessibility. That information you can get with iuiautomation using find and treewalker functions. As thats frequently limited you can get everything thru short javascript in adressbar. By

A.injecting websocket javascript or

B.textbox adding within html page and filling with information you can read with iuiautomation or

C. Javascript:alert(.....);void(0); and read the info from the popupbox.

B and C work with most browsers and A is more difficult as you sometimes need certificates with https websites.

[MichaelHB]

Ty for the reply junkew.

May I abuse little more of your good will ? 

1º - Can you show me or point to an exemple on how to read the popup in Chrome? (is there any generic method to grab the popup content? i was looking with simplespy but i get a specific popup like "www.autoitscript.com says:", i was able to find and click, but not grab the all the content)

2º - Which variable holds all the data i can see in the chrome://accessibility? (is it in a string format by the way? maybe i can easly read that data and then use other methods to grab the info i want)

 

Ty once more.

[junkew]

1. Not behind my pc so wait a little longer

A. Can be done with regular autoit functions

B. Just with the uiawrapper functions

2. You have access to full dom of html. See

http://www.w3schools.com/jsref/prop_html_innerhtml.asp

But many more properties and frames can complicate the answer.

[junkew]

1.A. I was wrong chrome popup seems not to be readable with AutoIT functions 

B. Advice is first to use highlight functions and work per object in the hierarchy
If stuff seems to be to difficult even after using inspect.exe to see full tree just start using $treescope_subtree if you are 100% sure your tree is not huge anymore.

Working source with UIAWrappers below (click is commented out), you can try by just typing javascript:alert("Hello world");void(0); in your chrome addressbar

;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=.*Google Chrome;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
Local $oP3=_UIA_getObjectByFindAll($oP4, "controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
;~ Local $oP2=_UIA_getObjectByFindAll($oP3, Title:=;controltype:=UIA_CustomControlTypeId;class:=, $treescope_children)
;~ Local $oP1=_UIA_getObjectByFindAll($oP2, Title:=;controltype:=UIA_CustomControlTypeId;class:=, $treescope_children)
;~ Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=OK;controltype:=UIA_ButtonControlTypeId", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("OK.mainwindow", "title:=OK;ControlType:=UIA_TextControlTypeId", $treescope_subtree)
;~ Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=OK;ControlType:=UIA_TextControlTypeId", $treescope_subtree)

;~ Local $oUIElement=_UIA_getObjectByFindAll($oP4, "Title:=OK;controltype:=UIA_ButtonControlTypeId", $treescope_children)

Local $oUIElement=_UIA_getObjectByFindAll($oP4, "Title:=OK;controltype:=UIA_ButtonControlTypeId", $treescope_subtree)


_UIA_Action($oP4,"setfocus")
;~ _UIA_action($oP4,"highlight")
_UIA_Action($oP3,"setfocus")
_UIA_action($oP3,"highlight")

_UIA_action($oUIElement,"highlight")

_UIA_Action($oP4,"setfocus")
_UIA_Action($oP3,"setfocus")
;~ _UIA_action($oUIElement,"click")

Steps are basically

1. Find the main window of the application under the desktop $treescope_children

2. Find the subwindow within your application $treescope_children and sometimes $treescope_subtree

3. Find the element you need and as this one is sometimes in a deep difficult treehierarchy you can use $treescope_subtree

 

So maybe cleaned for chrome it can look like this

 

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oChrome=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=.*Google Chrome;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
;~ Local $oChromeDocument=_UIA_getObjectByFindAll($oChrome, "Title:=Accessibility;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND, $treescope_children)
Local $oChromeDocument=_UIA_getObjectByFindAll($oChrome, "controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)


;~ Local $oP0=_UIA_getObjectByFindAll($oP1, Title:=;controltype:=UIA_CustomControlTypeId;class:=, $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("Globalaccessibilitymode:.mainwindow", "title:=Global accessibility mode: ;ControlType:=UIA_TextControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oChromeDocument, "title:=Global accessibility mode: ;ControlType:=UIA_TextControlTypeId", $treescope_subtree)



_UIA_Action($oChrome,"setfocus")
_UIA_Action($oChromeDocument,"setfocus")

_UIA_action($oChrome,"highlight")
_UIA_action($oChromeDocument,"highlight")
_UIA_action($oUIElement,"highlight")


;~ _UIA_action($oUIElement,"click")

 

Edited March 28, 2016 by junkew
edit: chrome example added

[MichaelHB]

Ty for the patience junkew 

In fact I just realized that it is possible to easily copy text from the alert popup, all you have to do is send CTRL + C ( CTRL + INSERT does not work), you can't select the text, but you can copy it.