FF.au3 (V0.6.0.1b-10)

frank10 · October 30, 2013

This is working:

local $elem = _FFCmd(".elementFromPoint(977,832);")

Now I need to detect the correct x, y points from mouse in CSS coordinates relative to document, to insert them into .elementFromPoint.

I think it could be correct to use:

event.pageY

https://developer.mozilla.org/en-US/docs/Web/API/event.pageY

but I can't make it work pageY , neither clientY:

local $y = _FFCmd("window.event.pageY;")

or

local $y = _FFCmd("content.wrappedJSObject.window.event.pageY;")

It seems only window.scrollY works:

local $scrollY = _FFCmd("content.wrappedJSObject.window.scrollY;")

Any ideas?

Danp2 · October 30, 2013

Is it possible to detect which element is under the mouse on a page?

For example a specific img or link.

Have you reviewed the code >here?

frank10 · October 30, 2013

Thank you Danp2, I had just seen your code.

But I can't make it work.

For example, let's take this site page

Let's position the mouse over the first Img's text: "Above City Skyscrapers Aerial":

with this HTML:

<li class="video-preview ">
<a href="http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/">
<img class="preview" rel="/preview/LA-AERIAL-21-2012-5222013/" src="http://misc.videoblocks.com/vbpreviews/thumbs3/LA-AERIAL-21-2012-5222013.jpg" height="83" width="148" alt="Above City Skyscrapers Aerial" title="Above City Skyscrapers Aerial" />
<span class="title">Above City Skyscrapers Aerial </span>
</a>
<div class="grey-button not-logged-join" id="LA-AERIAL-21-2012-5222013" style="display:none;"></div>
</li>

Without scrolling the page and fullscreen at 1920x1080, the absolute position relative to the content area of the client win is:

local $oTitle =  _FFXpath("//li[contains(@class,'video-preview')]/a/span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
local $aPos = _FFGetPosition($oTitle)

x,y  are 478,633

This is correct result as we can test:

local $elem = _FFCmd(".elementFromPoint(478,633);")

MsgBox(0,'', $elem ) --> "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...}"

With your code I get a screen mousepoint of x,y 762,984 and as returned from

_FFGetMouseElement($pos) I obtain x,y 762,899

but if you put these x,y into .elementFromPoint:

local $elem = _FFCmd(".elementFromPoint(762,899);")

MsgBox(0,'', $elem ) --> "" that is, it can't get the element from the x,y

If I force the x,y as in the first case, I get the correct LI element nodeName.

So the problem is in the conversion of x,y from screen coords to correct CSS windows coord related.

Edited October 30, 2013 by frank10

Danp2 · October 30, 2013

When I run this:

#include <FF.au3>

$pos = MouseGetPos()

ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF)

_FFConnect(Default, Default, 3000)

__FFSetTopDocument()

_FFGetMouseElement($pos)

MsgBox(0,'', _FFCmd("FFau3.xpath") )

ConsoleWrite("NodeName = " & _FFCmd("FFau3.xpath.nodeName") & @CRLF)

it appears that the the code is working as expected. However, it appears that the Span associated with the link is being returned instead of the Div.

frank10 · October 30, 2013

Weird.

Try with this:

#include <FF.au3>

_FFConnect(Default, Default, 3000)

local $oTitle =  _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
local $aPos = _FFGetPosition($oTitle)

MsgBox(0,'','now position the mouse over text of first img text Above city...')

Sleep(2000)

$pos = MouseGetPos()

ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF)

 __FFSetTopDocument()

_FFGetMouseElement($pos)



Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])


    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )


    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

In the msgbox "compare pos", I get:

485,564 for GetPosition

versus

764,900 for FFGetMouseElement

So it can't get the correct element with elementFromPoint.

Do you get the same x,y with the two methods?

Danp2 · October 30, 2013

It's possible that the results are being affected because of the javascript that his hiding / showing the Img / Span combination when you mouse over the area.

Also, make sure that you are using the latest version of FF.au3 as there was an issue with _FFGetPosition in an earlier version.

Edited October 30, 2013 by Danp2

frank10 · October 30, 2013

No, it's not.

For example put in the background Firefox and a foreground window of Scite with focus on this script, put mouse over the link "Aerial" (in the middle Home > Aerial > City views) and hit F5 (with no mouseover javascript):

#include <FF.au3>

_FFConnect(Default, Default, 3000)

local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9)
local $aPos = _FFGetPosition($oHref)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])


    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )


    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

I get different x,y in the Msgbox... so it's not javascript.

What do you get?

Edited October 30, 2013 by frank10

Danp2 · October 31, 2013

You removed the line that adjusts the vertical position of the mouse. What do you get when you run the following?

#include <FF.au3>

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9)

MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))

local $aPos = _FFGetPosition($oHref)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )

    MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

FYI, the x/y values will never match exactly unless you position mouse cursor at the very top left of the desired element.

frank10 · October 31, 2013

No, it doesn't work:

first _FFCMD('FFau3.xpath') from _FFGetPosition:

http://www.videoblocks.com/videos/aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

second _FFCMD('FFau3.xpath') from _FFGetMouseElement (with mouse over "Aerial" link):

[object XrayWrapper [object HTMLDivElement]] - {align: "", click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", ...}

different x,y:

GetPosition:495,281 _FFGetMouseElement:767  473.666664123535

I know the x,y can't be exactly equal,but here we talk about a very different couple!

Edited October 31, 2013 by frank10

Danp2 · October 31, 2013

Did you run the code exactly as posted, including the setting for MouseCoordMode?

What version of FF.au3 are you running?

Edited October 31, 2013 by Danp2

frank10 · October 31, 2013

Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings)

W7 64bit @ 1920x1080

I run the FF 0.6.0.1b-11

Autoit 3.3.6.1

Firefox 24.0

Mozrepl 1.1.2

But it's not a problem only with this page, I tried for example another one:

https://wiki.mozilla.org/Accessibility/AT-Windows-API#How_to_Find_the_Content_Window_and_Load_the_Document

with this code, leaving mouse over the link "Back to Accessibility Home Page":

#include <FF.au3>

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oA =  _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9)

MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))
ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf)

local $aPos = _FFGetPosition($oA)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )
    ConsoleWrite("--" & "GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] & @crlf)


    MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))
    ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf)
EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

results:

first _FFCMD('FFau3.xpath') from _FFGetPosition:

https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

second _FFCMD('FFau3.xpath') from _FFGetMouseElement (with mouse over "Aerial" link):

[object XrayWrapper [object HTMLSpanElement]] - {click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", dataset: {...}, ...}

different x,y:

GetPosition:308,383 _FFGetMouseElement:435  578.716667175293

But, you didn't answer which values do you get in any of these examples I gave you?

I would like if anyone test these on their platform to see if it's a problem on my setup or...

Danp2 · October 31, 2013

Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings)

W7 64bit @ 1920x1080

I run the FF 0.6.0.1b-11

Autoit 3.3.6.1

Firefox 24.0

Mozrepl 1.1.2

Not sure if that's a typo, but the latest release version of Autoit is 3.3.8.1.

Also, my testing has been with FF 0.6.0.1b-9, which was released back in September. I haven't seen the version that you are using.

Danp2 · October 31, 2013

I'm still seeing different results from you as well.

--GetPosition:273,345 _FFGetMouseElement:359  351

and the same element was returned in both instances.

-https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

frank10 · October 31, 2013

Thank you for the tests. Weird.

I have tried with both 3.3.6.1 and the beta 3.3.9.1 (anyway I will install the latest, but that's not the problem)

Here you get the 1.b12 version (when I downloaded it was .11):

http://thorsten-willert.de/Themen/FFau3/Testversion/files

So it's a problem with my setup!

I don't know what, maybe some firefox addon? Now I will try disabling them...

Edit:

I tried FF .1b-9 and it's not that

Edited October 31, 2013 by frank10

frank10 · October 31, 2013

I disabled all addon but mozrepl and no change!

Now, I don't know what to think...

I tried to look at every step in changing coordinates:

original mouse:

459 , 734

user.dll ScreenToClient:

459 , 712

mozInnerScreenY:

459 , 667

real window coordinates from _FFGetPosition (the only correct to use with .elementFromPoint):

267 , 357

So I guess the problem is in user.dll because it's the only that can change significantly the coordinates, instead they remain quite the same.

Edited October 31, 2013 by frank10

frank10 · October 31, 2013

But It seems good also the user.dll...

For example if I put Firefox not fullscreen and scroll the link to match the left margin window and toward up just under the tabs

I get correct coord, like 4, 174 (it calculates from tabs not the menu).

But how can these coord relative to window's client became relative to the absolute CSS document?

FFGetPosition refers to absolute doc positioning (without scrolls and so on).

After you have these user32.dll relative to client window how can they became relative to document? that's the step it lacks.

There is Y correction (little px) but in every case there isn't X correction, so?

I don't understand how you get correct result.

Could you post the coord before and after the user32.dll, after mozInnerScreenY and FFGetposition?

Thanks.

Danp2 · October 31, 2013

;#include <FF.au3>
#include "FF V0.6.0.1b-12.au3"

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oA =  _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9)

ConsoleWrite("--_FFXpath = " & _FFCMD('FFau3.xpath') & @crlf)

local $aPos = _FFGetPosition($oA)

ConsoleWrite("--_FFGetPosition = " & $aPos[0] & ","&$aPos[1] & @crlf)

$pos = MouseGetPos()

ConsoleWrite("--MouseGetPos = " & $pos[0] & ","&$pos[1] & @crlf)

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

ConsoleWrite("--_ScreenToClient = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf)

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

ConsoleWrite("--AdjustedPosition = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf)

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

ConsoleWrite("--_FFXpath #2 = " & _FFCMD('FFau3.xpath') & @crlf)

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

Here's the output:

--_FFXpath = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}
--_FFGetPosition = 272,344
--MouseGetPos = 1599,1156
--_ScreenToClient = 394,444
--AdjustedPosition = 394,351
--_FFXpath #2 = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

frank10 · October 31, 2013

Hmm interesting: it seems your user32.dll trasforms the screen coords to the document coords, not the windows client coords.

Mine instead doesn't!

I don't know why.

Is there a formula how to make this calc apart user32.dll?

Or maybe another approach, if someone knows how to get info from event.pageY in FF_Cmd...

It should fire when there is a mouse event, like:

local $oTitle =  _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
If StringLeft($oTitle, 7) = "OBJECT|" Then $oTitle = StringMid($oTitle, 8)
_FFCmd("FFau3.simulateEvent(" & $oTitle &",'MouseEvents','mouseover')")

but doesn't work...

Danp2 · October 31, 2013

Perhaps a difference between XP and W7? <shrug>

You could try using _WinAPI_GetMousePos() instead of MouseGetPos(). Not sure what else to suggest at this point.

frank10 · November 2, 2013

I studied a bit more events and mozrepl, so I solved this way:

test page:

https://github.com/bard/mozrepl/wiki/Custom-commands-library

#include <FF.au3>

_FFConnect(Default, Default, 3000)

_FFCmd("repl.enter(content.wrappedJSObject)")  ; without this, 'e' event doesn't work!
_FFCmd("var posx; var posy")
_FFCMD("function coord(e) {  posx = e.clientX + document.body.scrollLeft+ document.documentElement.scrollLeft;  posy = e.clientY + document.body.scrollTop + document.documentElement.scrollTop;}");  posX = posx ; posY = posy ;}
_FFCmd("var WCD = window.content.document; var element = WCD.getElementsByTagName('A');for (var i = 0; i < element.length; i++) { element[i].addEventListener('mouseover',  coord , false);}; ")
_FFCmd("repl.home()")  


local $x = 0, $y = 0
while 1
    $x = _FFCmd("content.wrappedJSObject.posx")
    $y = _FFCmd("content.wrappedJSObject.posy")

    local $elem = _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint("&$x&","&$y&")")

        if _FFCmd("FFau3.xpath") = '' Then ContinueLoop

    local $aPos = _FFGetPosition("FFau3.xpath")


    Consolewrite("--CustomFuncPos: " & $x & "," & $y & "  FFGetPosition: " & $aPos[0] & "," & $aPos[1] &@crlf)
    MsgBox(0,'',"--CustomFuncPos: " & $x & "," & $y & "  FFGetPosition: " & $aPos[0] & "," & $aPos[1] & " nodo:" & _FFCMD('FFau3.xpath.nodeName') & " text:"&  _FFCMD('FFau3.xpath.text')  )
    Sleep(1000)
wend

Some links with 'span' tag don't give text with _FFCMD('FFau3.xpath.text') (?) but anyway with normal 'a' elements I get text and correct coord.

For example "bard" or "Clone URL" don't give text.

So, solved with DOM method.

Edited November 2, 2013 by frank10

Sign In

FF.au3 (V0.6.0.1b-10)

Recommended Posts

frank10

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Stilgar

Danp2

Danp2

Posted Images

Danp2

frank10

Danp2

frank10

Danp2

frank10

Danp2

frank10

Danp2

frank10

Danp2

Danp2

frank10

frank10

frank10

Danp2

frank10

Danp2

frank10

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta