Jump to content

FF.au3 (V0.6.0.1b-10)


Stilgar
 Share

Recommended Posts

This is working:

local $elem = _FFCmd(".elementFromPoint(977,832);")

Now I need to detect the correct x, y points from mouse in CSS coordinates relative to document, to insert them into .elementFromPoint.

I think it could be correct to use:

event.pageY

https://developer.mozilla.org/en-US/docs/Web/API/event.pageY

but I can't make it work  pageY   , neither      clientY:

local $y = _FFCmd("window.event.pageY;")

or

local $y = _FFCmd("content.wrappedJSObject.window.event.pageY;")

It seems only window.scrollY works:

local $scrollY = _FFCmd("content.wrappedJSObject.window.scrollY;")

Any ideas?

Link to comment
Share on other sites

Thank you Danp2, I had just seen your code.

But I can't make it work.

For example, let's take this site page

Let's position the mouse over the first Img's text: "Above City Skyscrapers Aerial":

with this HTML:

<li class="video-preview ">
<a href="http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/">
<img class="preview" rel="/preview/LA-AERIAL-21-2012-5222013/" src="http://misc.videoblocks.com/vbpreviews/thumbs3/LA-AERIAL-21-2012-5222013.jpg" height="83" width="148" alt="Above City Skyscrapers Aerial" title="Above City Skyscrapers Aerial" />
<span class="title">Above City Skyscrapers Aerial </span>
</a>
<div class="grey-button not-logged-join" id="LA-AERIAL-21-2012-5222013" style="display:none;"></div>
</li>

Without scrolling the page and fullscreen at 1920x1080, the absolute position relative to the content area of the client win is:

local $oTitle =  _FFXpath("//li[contains(@class,'video-preview')]/a/span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
local $aPos = _FFGetPosition($oTitle)

x,y  are 478,633

This is correct result as we can test:

local $elem = _FFCmd(".elementFromPoint(478,633);")

MsgBox(0,'', $elem ) --> "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...}"

With your code I get a screen mousepoint of x,y 762,984 and as returned from

_FFGetMouseElement($pos)  I obtain x,y 762,899

but if you put these x,y into .elementFromPoint:

local $elem = _FFCmd(".elementFromPoint(762,899);")

MsgBox(0,'', $elem ) --> "" that is, it can't get the element from the x,y

If I force the x,y as in the first case, I get the correct LI element nodeName.

So the problem is in the conversion of x,y from screen coords to correct CSS windows coord related.

Edited by frank10
Link to comment
Share on other sites

When I run this:

#include <FF.au3>

$pos = MouseGetPos()

ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF)

_FFConnect(Default, Default, 3000)

__FFSetTopDocument()

_FFGetMouseElement($pos)

MsgBox(0,'', _FFCmd("FFau3.xpath") )

ConsoleWrite("NodeName = " & _FFCmd("FFau3.xpath.nodeName") & @CRLF)

it appears that the the code is working as expected. However, it appears that the Span associated with the link is being returned instead of the Div.

Link to comment
Share on other sites

Weird.

Try with this:

#include <FF.au3>

_FFConnect(Default, Default, 3000)

local $oTitle =  _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
local $aPos = _FFGetPosition($oTitle)

MsgBox(0,'','now position the mouse over text of first img text Above city...')

Sleep(2000)

$pos = MouseGetPos()

ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF)

 __FFSetTopDocument()

_FFGetMouseElement($pos)



Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])


    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )


    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

In the msgbox "compare pos", I get:

485,564  for GetPosition

versus

764,900 for FFGetMouseElement

So it can't get the correct element with elementFromPoint.

Do you get the same x,y with the two methods?

Link to comment
Share on other sites

It's possible that the results are being affected because of the javascript that his hiding / showing the Img / Span combination when you mouse over the area.

Also, make sure that you are using the latest version of FF.au3 as there was an issue with _FFGetPosition in an earlier version.

Edited by Danp2
Link to comment
Share on other sites

No, it's not.

For example put in the background Firefox and a foreground window of Scite with focus on this script, put mouse over the link "Aerial" (in the middle Home > Aerial > City views) and hit F5 (with no mouseover javascript):

#include <FF.au3>

_FFConnect(Default, Default, 3000)

local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9)
local $aPos = _FFGetPosition($oHref)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])


    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )


    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

I get different x,y in the Msgbox...  so it's not javascript.

What do you get?

Edited by frank10
Link to comment
Share on other sites

You removed the line that adjusts the vertical position of the mouse. What do you get when you run the following?

#include <FF.au3>

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9)

MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))

local $aPos = _FFGetPosition($oHref)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )

    MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

FYI, the x/y values will never match exactly unless you position mouse cursor at the very top left of the desired element.

Link to comment
Share on other sites

No, it doesn't work:

first _FFCMD('FFau3.xpath') from _FFGetPosition:

http://www.videoblocks.com/videos/aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

second  _FFCMD('FFau3.xpath') from _FFGetMouseElement   (with mouse over "Aerial" link):

[object XrayWrapper [object HTMLDivElement]] - {align: "", click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", ...}

different x,y:

GetPosition:495,281 _FFGetMouseElement:767  473.666664123535

I know the x,y can't be exactly equal,but here we talk about a very different couple!

Edited by frank10
Link to comment
Share on other sites

Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings)

W7 64bit @ 1920x1080

I run the FF 0.6.0.1b-11

Autoit 3.3.6.1

Firefox 24.0

Mozrepl 1.1.2

But it's not a problem only with this page, I tried for example another one:

https://wiki.mozilla.org/Accessibility/AT-Windows-API#How_to_Find_the_Content_Window_and_Load_the_Document

with this code, leaving mouse over the link "Back to Accessibility Home Page":

#include <FF.au3>

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oA =  _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9)

MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))
ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf)

local $aPos = _FFGetPosition($oA)

$pos = MouseGetPos()

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

    MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] )
    ConsoleWrite("--" & "GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & "  "& $aFFPos[1] & @crlf)


    MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath'))
    ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf)
EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

results:

first _FFCMD('FFau3.xpath') from _FFGetPosition:

https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}

second  _FFCMD('FFau3.xpath') from _FFGetMouseElement   (with mouse over "Aerial" link):

[object XrayWrapper [object HTMLSpanElement]] - {click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", dataset: {...}, ...}

different x,y:

GetPosition:308,383 _FFGetMouseElement:435  578.716667175293

But, you didn't answer which values do you get in any of these examples I gave you?

I would like if anyone test these on their platform to see if it's a problem on my setup or...

Link to comment
Share on other sites

Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings)

W7 64bit @ 1920x1080

I run the FF 0.6.0.1b-11

Autoit 3.3.6.1

Firefox 24.0

Mozrepl 1.1.2

 

Not sure if that's a typo, but the latest release version of Autoit is 3.3.8.1.

Also, my testing has been with FF 0.6.0.1b-9, which was released back in September. I haven't seen the version that you are using.

Link to comment
Share on other sites

I'm still seeing different results from you as well.

--GetPosition:273,345 _FFGetMouseElement:359  351

and the same element was returned in both instances.

-https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}
Link to comment
Share on other sites

Thank you for the tests. Weird.

 

I have tried with both 3.3.6.1 and the beta 3.3.9.1 (anyway I will install the latest, but that's not the problem)

Here you get the  1.b12 version (when I downloaded it was .11):

http://thorsten-willert.de/Themen/FFau3/Testversion/files

So it's a problem with my setup!

I don't know what, maybe some firefox addon? Now I will try disabling them...

Edit:

I tried FF .1b-9 and it's not that

Edited by frank10
Link to comment
Share on other sites

I disabled all addon but mozrepl and no change!

Now, I don't know what to think...

I tried to look at every step in changing coordinates:

original mouse:               

459 , 734

user.dll ScreenToClient:

459 , 712

mozInnerScreenY:

459 , 667

real window coordinates from _FFGetPosition     (the only correct to use with .elementFromPoint):

267 , 357

So I guess the problem is in user.dll because it's the only that can change significantly the coordinates, instead they remain quite the same.

Edited by frank10
Link to comment
Share on other sites

But It seems good also the user.dll...

For example if I put Firefox not fullscreen and scroll the link to match the left margin window and toward up just under the tabs

I get correct coord, like   4, 174 (it calculates from tabs not the menu).

But how can these coord relative to window's client became relative to the absolute CSS document?

FFGetPosition refers to absolute doc positioning (without scrolls and so on).

After you have these user32.dll relative to client window how can they became relative to document? that's the step it lacks.

There is Y correction (little px) but in every case there isn't X correction, so?

I don't understand how you get correct result.

Could you post the coord before and after the user32.dll, after mozInnerScreenY and FFGetposition?

Thanks.

Link to comment
Share on other sites

;#include <FF.au3>
#include "FF V0.6.0.1b-12.au3"

AutoItSetOption ( "MouseCoordMode", 1)

_FFConnect(Default, Default, 3000)

local $oA =  _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9)

ConsoleWrite("--_FFXpath = " & _FFCMD('FFau3.xpath') & @crlf)

local $aPos = _FFGetPosition($oA)

ConsoleWrite("--_FFGetPosition = " & $aPos[0] & ","&$aPos[1] & @crlf)

$pos = MouseGetPos()

ConsoleWrite("--MouseGetPos = " & $pos[0] & ","&$pos[1] & @crlf)

__FFSetTopDocument()

_FFGetMouseElement($pos)

Func _FFGetMouseElement($aPoint)
    Local $aFFPos, $hWindow

    $hWindow = WinGetHandle(_FFCmd("document.title"))

    $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1])

ConsoleWrite("--_ScreenToClient = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf)

    $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

ConsoleWrite("--AdjustedPosition = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf)

    _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")

ConsoleWrite("--_FFXpath #2 = " & _FFCMD('FFau3.xpath') & @crlf)

EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
    If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

    Local $t_xy_coords = DllStructCreate("int x; int y")
    DllStructSetData($t_xy_coords, "x", $i_x)
    DllStructSetData($t_xy_coords, "y", $i_y)

    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

    Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
    Return $a_ret_coords
EndFunc

Here's the output:

--_FFXpath = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}
--_FFGetPosition = 272,344
--MouseGetPos = 1599,1156
--_ScreenToClient = 394,444
--AdjustedPosition = 394,351
--_FFXpath #2 = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...}
Link to comment
Share on other sites

Hmm interesting: it seems your user32.dll trasforms the screen coords to the document coords, not the windows client coords.

Mine instead doesn't!

I don't know why.

Is there a formula how to make this calc apart user32.dll?

Or maybe another approach, if someone knows how to get  info from event.pageY in FF_Cmd...

It should fire when there is a mouse event, like:

local $oTitle =  _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9)
If StringLeft($oTitle, 7) = "OBJECT|" Then $oTitle = StringMid($oTitle, 8)
_FFCmd("FFau3.simulateEvent(" & $oTitle &",'MouseEvents','mouseover')")

but doesn't work...

Link to comment
Share on other sites

I studied a bit more events and mozrepl, so I solved this way:

test page:

https://github.com/bard/mozrepl/wiki/Custom-commands-library

#include <FF.au3>

_FFConnect(Default, Default, 3000)

_FFCmd("repl.enter(content.wrappedJSObject)")  ; without this, 'e' event doesn't work!
_FFCmd("var posx; var posy")
_FFCMD("function coord(e) {  posx = e.clientX + document.body.scrollLeft+ document.documentElement.scrollLeft;  posy = e.clientY + document.body.scrollTop + document.documentElement.scrollTop;}");  posX = posx ; posY = posy ;}
_FFCmd("var WCD = window.content.document; var element = WCD.getElementsByTagName('A');for (var i = 0; i < element.length; i++) { element[i].addEventListener('mouseover',  coord , false);}; ")
_FFCmd("repl.home()")  


local $x = 0, $y = 0
while 1
    $x = _FFCmd("content.wrappedJSObject.posx")
    $y = _FFCmd("content.wrappedJSObject.posy")

    local $elem = _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint("&$x&","&$y&")")

        if _FFCmd("FFau3.xpath") = '' Then ContinueLoop

    local $aPos = _FFGetPosition("FFau3.xpath")


    Consolewrite("--CustomFuncPos: " & $x & "," & $y & "  FFGetPosition: " & $aPos[0] & "," & $aPos[1] &@crlf)
    MsgBox(0,'',"--CustomFuncPos: " & $x & "," & $y & "  FFGetPosition: " & $aPos[0] & "," & $aPos[1] & " nodo:" & _FFCMD('FFau3.xpath.nodeName') & " text:"&  _FFCMD('FFau3.xpath.text')  )
    Sleep(1000)
wend

Some links with 'span' tag don't give text with  _FFCMD('FFau3.xpath.text') (?) but anyway with normal 'a' elements I get text and correct coord.

For example "bard" or "Clone URL" don't give text.

So, solved with DOM method.

Edited by frank10
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...