frank10 Posted October 30, 2013 Share Posted October 30, 2013 This is working: local $elem = _FFCmd(".elementFromPoint(977,832);") Now I need to detect the correct x, y points from mouse in CSS coordinates relative to document, to insert them into .elementFromPoint. I think it could be correct to use: event.pageY https://developer.mozilla.org/en-US/docs/Web/API/event.pageY but I can't make it work pageY , neither clientY: local $y = _FFCmd("window.event.pageY;") or local $y = _FFCmd("content.wrappedJSObject.window.event.pageY;") It seems only window.scrollY works: local $scrollY = _FFCmd("content.wrappedJSObject.window.scrollY;") Any ideas? Link to comment Share on other sites More sharing options...
Danp2 Posted October 30, 2013 Share Posted October 30, 2013 Is it possible to detect which element is under the mouse on a page? For example a specific img or link. Have you reviewed the code >here? Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 30, 2013 Share Posted October 30, 2013 (edited) Thank you Danp2, I had just seen your code. But I can't make it work. For example, let's take this site page Let's position the mouse over the first Img's text: "Above City Skyscrapers Aerial": with this HTML: <li class="video-preview "> <a href="http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/"> <img class="preview" rel="/preview/LA-AERIAL-21-2012-5222013/" src="http://misc.videoblocks.com/vbpreviews/thumbs3/LA-AERIAL-21-2012-5222013.jpg" height="83" width="148" alt="Above City Skyscrapers Aerial" title="Above City Skyscrapers Aerial" /> <span class="title">Above City Skyscrapers Aerial </span> </a> <div class="grey-button not-logged-join" id="LA-AERIAL-21-2012-5222013" style="display:none;"></div> </li> Without scrolling the page and fullscreen at 1920x1080, the absolute position relative to the content area of the client win is: local $oTitle = _FFXpath("//li[contains(@class,'video-preview')]/a/span[contains(.,'Above City Skyscrapers Aerial')]", "", 9) local $aPos = _FFGetPosition($oTitle) x,y are 478,633 This is correct result as we can test: local $elem = _FFCmd(".elementFromPoint(478,633);") MsgBox(0,'', $elem ) --> "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/details/above-city-skyscrapers-aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...}" With your code I get a screen mousepoint of x,y 762,984 and as returned from _FFGetMouseElement($pos) I obtain x,y 762,899 but if you put these x,y into .elementFromPoint: local $elem = _FFCmd(".elementFromPoint(762,899);") MsgBox(0,'', $elem ) --> "" that is, it can't get the element from the x,y If I force the x,y as in the first case, I get the correct LI element nodeName. So the problem is in the conversion of x,y from screen coords to correct CSS windows coord related. Edited October 30, 2013 by frank10 Link to comment Share on other sites More sharing options...
Danp2 Posted October 30, 2013 Share Posted October 30, 2013 When I run this: #include <FF.au3> $pos = MouseGetPos() ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF) _FFConnect(Default, Default, 3000) __FFSetTopDocument() _FFGetMouseElement($pos) MsgBox(0,'', _FFCmd("FFau3.xpath") ) ConsoleWrite("NodeName = " & _FFCmd("FFau3.xpath.nodeName") & @CRLF) it appears that the the code is working as expected. However, it appears that the Span associated with the link is being returned instead of the Div. Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 30, 2013 Share Posted October 30, 2013 Weird. Try with this: expandcollapse popup#include <FF.au3> _FFConnect(Default, Default, 3000) local $oTitle = _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9) local $aPos = _FFGetPosition($oTitle) MsgBox(0,'','now position the mouse over text of first img text Above city...') Sleep(2000) $pos = MouseGetPos() ConsoleWrite("pos = " & $pos[0] & ' , ' & $pos[1] &@CRLF) __FFSetTopDocument() _FFGetMouseElement($pos) Func _FFGetMouseElement($aPoint) Local $aFFPos, $hWindow $hWindow = WinGetHandle(_FFCmd("document.title")) $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1]) MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & " "& $aFFPos[1] ) _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");") EndFunc Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd) Local $t_xy_coords = DllStructCreate("int x; int y") DllStructSetData($t_xy_coords, "x", $i_x) DllStructSetData($t_xy_coords, "y", $i_y) DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords)) Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")] Return $a_ret_coords EndFunc In the msgbox "compare pos", I get: 485,564 for GetPosition versus 764,900 for FFGetMouseElement So it can't get the correct element with elementFromPoint. Do you get the same x,y with the two methods? Link to comment Share on other sites More sharing options...
Danp2 Posted October 30, 2013 Share Posted October 30, 2013 (edited) It's possible that the results are being affected because of the javascript that his hiding / showing the Img / Span combination when you mouse over the area. Also, make sure that you are using the latest version of FF.au3 as there was an issue with _FFGetPosition in an earlier version. Edited October 30, 2013 by Danp2 Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 30, 2013 Share Posted October 30, 2013 (edited) No, it's not. For example put in the background Firefox and a foreground window of Scite with focus on this script, put mouse over the link "Aerial" (in the middle Home > Aerial > City views) and hit F5 (with no mouseover javascript): expandcollapse popup#include <FF.au3> _FFConnect(Default, Default, 3000) local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9) local $aPos = _FFGetPosition($oHref) $pos = MouseGetPos() __FFSetTopDocument() _FFGetMouseElement($pos) Func _FFGetMouseElement($aPoint) Local $aFFPos, $hWindow $hWindow = WinGetHandle(_FFCmd("document.title")) $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1]) MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & " "& $aFFPos[1] ) _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");") EndFunc Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd) Local $t_xy_coords = DllStructCreate("int x; int y") DllStructSetData($t_xy_coords, "x", $i_x) DllStructSetData($t_xy_coords, "y", $i_y) DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords)) Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")] Return $a_ret_coords EndFunc I get different x,y in the Msgbox... so it's not javascript. What do you get? Edited October 30, 2013 by frank10 Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 You removed the line that adjusts the vertical position of the mouse. What do you get when you run the following? expandcollapse popup#include <FF.au3> AutoItSetOption ( "MouseCoordMode", 1) _FFConnect(Default, Default, 3000) local $oHref = _FFXpath("//*[@class='breadcrumbs']/li[2]/a", "", 9) MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath')) local $aPos = _FFGetPosition($oHref) $pos = MouseGetPos() __FFSetTopDocument() _FFGetMouseElement($pos) Func _FFGetMouseElement($aPoint) Local $aFFPos, $hWindow $hWindow = WinGetHandle(_FFCmd("document.title")) $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1]) $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY") _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");") MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & " "& $aFFPos[1] ) MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath')) EndFunc Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd) Local $t_xy_coords = DllStructCreate("int x; int y") DllStructSetData($t_xy_coords, "x", $i_x) DllStructSetData($t_xy_coords, "y", $i_y) DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords)) Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")] Return $a_ret_coords EndFunc FYI, the x/y values will never match exactly unless you position mouse cursor at the very top left of the desired element. Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 (edited) No, it doesn't work: first _FFCMD('FFau3.xpath') from _FFGetPosition: http://www.videoblocks.com/videos/aerial/ - {toString: function() {...}, href: "http://www.videoblocks.com/videos/aerial/", target: "", download: "", ping: "", rel: "", hreflang: "", ...} second _FFCMD('FFau3.xpath') from _FFGetMouseElement (with mouse over "Aerial" link): [object XrayWrapper [object HTMLDivElement]] - {align: "", click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", ...} different x,y: GetPosition:495,281 _FFGetMouseElement:767 473.666664123535 I know the x,y can't be exactly equal,but here we talk about a very different couple! Edited October 31, 2013 by frank10 Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 (edited) Did you run the code exactly as posted, including the setting for MouseCoordMode? What version of FF.au3 are you running? Edited October 31, 2013 by Danp2 Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings) W7 64bit @ 1920x1080 I run the FF 0.6.0.1b-11 Autoit 3.3.6.1 Firefox 24.0 Mozrepl 1.1.2 But it's not a problem only with this page, I tried for example another one: https://wiki.mozilla.org/Accessibility/AT-Windows-API#How_to_Find_the_Content_Window_and_Load_the_Document with this code, leaving mouse over the link "Back to Accessibility Home Page": expandcollapse popup#include <FF.au3> AutoItSetOption ( "MouseCoordMode", 1) _FFConnect(Default, Default, 3000) local $oA = _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9) MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath')) ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf) local $aPos = _FFGetPosition($oA) $pos = MouseGetPos() __FFSetTopDocument() _FFGetMouseElement($pos) Func _FFGetMouseElement($aPoint) Local $aFFPos, $hWindow $hWindow = WinGetHandle(_FFCmd("document.title")) $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1]) $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY") _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");") MsgBox(0,'compare pos',"GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & " "& $aFFPos[1] ) ConsoleWrite("--" & "GetPosition:" & $aPos[0] & ","&$aPos[1] & " _FFGetMouseElement:" & $aFFPos[0] & " "& $aFFPos[1] & @crlf) MsgBox(0, 'FFau3.xpath', _FFCMD('FFau3.xpath')) ConsoleWrite("-" & _FFCMD('FFau3.xpath') & @crlf) EndFunc Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd) Local $t_xy_coords = DllStructCreate("int x; int y") DllStructSetData($t_xy_coords, "x", $i_x) DllStructSetData($t_xy_coords, "y", $i_y) DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords)) Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")] Return $a_ret_coords EndFunc results: first _FFCMD('FFau3.xpath') from _FFGetPosition: https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...} second _FFCMD('FFau3.xpath') from _FFGetMouseElement (with mouse over "Aerial" link): [object XrayWrapper [object HTMLSpanElement]] - {click: function() {...}, focus: function() {...}, blur: function() {...}, title: "", lang: "", dir: "", dataset: {...}, ...} different x,y: GetPosition:308,383 _FFGetMouseElement:435 578.716667175293 But, you didn't answer which values do you get in any of these examples I gave you? I would like if anyone test these on their platform to see if it's a problem on my setup or... Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 Yes, I run the code exactly including MouseCoord (added only 3 ConsoleWrite). (I tried also all the other MouseCoord settings) W7 64bit @ 1920x1080 I run the FF 0.6.0.1b-11 Autoit 3.3.6.1 Firefox 24.0 Mozrepl 1.1.2 Not sure if that's a typo, but the latest release version of Autoit is 3.3.8.1. Also, my testing has been with FF 0.6.0.1b-9, which was released back in September. I haven't seen the version that you are using. Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 I'm still seeing different results from you as well. --GetPosition:273,345 _FFGetMouseElement:359 351 and the same element was returned in both instances. -https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...} Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 (edited) Thank you for the tests. Weird. I have tried with both 3.3.6.1 and the beta 3.3.9.1 (anyway I will install the latest, but that's not the problem) Here you get the 1.b12 version (when I downloaded it was .11): http://thorsten-willert.de/Themen/FFau3/Testversion/files So it's a problem with my setup! I don't know what, maybe some firefox addon? Now I will try disabling them... Edit: I tried FF .1b-9 and it's not that Edited October 31, 2013 by frank10 Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 (edited) I disabled all addon but mozrepl and no change! Now, I don't know what to think... I tried to look at every step in changing coordinates: original mouse: 459 , 734 user.dll ScreenToClient: 459 , 712 mozInnerScreenY: 459 , 667 real window coordinates from _FFGetPosition (the only correct to use with .elementFromPoint): 267 , 357 So I guess the problem is in user.dll because it's the only that can change significantly the coordinates, instead they remain quite the same. Edited October 31, 2013 by frank10 Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 But It seems good also the user.dll... For example if I put Firefox not fullscreen and scroll the link to match the left margin window and toward up just under the tabs I get correct coord, like 4, 174 (it calculates from tabs not the menu). But how can these coord relative to window's client became relative to the absolute CSS document? FFGetPosition refers to absolute doc positioning (without scrolls and so on). After you have these user32.dll relative to client window how can they became relative to document? that's the step it lacks. There is Y correction (little px) but in every case there isn't X correction, so? I don't understand how you get correct result. Could you post the coord before and after the user32.dll, after mozInnerScreenY and FFGetposition? Thanks. Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 expandcollapse popup;#include <FF.au3> #include "FF V0.6.0.1b-12.au3" AutoItSetOption ( "MouseCoordMode", 1) _FFConnect(Default, Default, 3000) local $oA = _FFXpath("//a[contains(.,'Back to Accessibility Home')]", "", 9) ConsoleWrite("--_FFXpath = " & _FFCMD('FFau3.xpath') & @crlf) local $aPos = _FFGetPosition($oA) ConsoleWrite("--_FFGetPosition = " & $aPos[0] & ","&$aPos[1] & @crlf) $pos = MouseGetPos() ConsoleWrite("--MouseGetPos = " & $pos[0] & ","&$pos[1] & @crlf) __FFSetTopDocument() _FFGetMouseElement($pos) Func _FFGetMouseElement($aPoint) Local $aFFPos, $hWindow $hWindow = WinGetHandle(_FFCmd("document.title")) $aFFPos = _ScreenToClient($hWindow, $aPoint[0], $aPoint[1]) ConsoleWrite("--_ScreenToClient = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf) $aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY") ConsoleWrite("--AdjustedPosition = " & $aFFPos[0] & ","&$aFFPos[1] & @crlf) _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");") ConsoleWrite("--_FFXpath #2 = " & _FFCMD('FFau3.xpath') & @crlf) EndFunc Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd) Local $t_xy_coords = DllStructCreate("int x; int y") DllStructSetData($t_xy_coords, "x", $i_x) DllStructSetData($t_xy_coords, "y", $i_y) DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords)) Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")] Return $a_ret_coords EndFunc Here's the output: --_FFXpath = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...} --_FFGetPosition = 272,344 --MouseGetPos = 1599,1156 --_ScreenToClient = 394,444 --AdjustedPosition = 394,351 --_FFXpath #2 = https://wiki.mozilla.org/Accessibility - {toString: function() {...}, href: "https://wiki.mozilla.org/Accessibility", target: "", download: "", ping: "", rel: "", hreflang: "", ...} Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted October 31, 2013 Share Posted October 31, 2013 Hmm interesting: it seems your user32.dll trasforms the screen coords to the document coords, not the windows client coords. Mine instead doesn't! I don't know why. Is there a formula how to make this calc apart user32.dll? Or maybe another approach, if someone knows how to get info from event.pageY in FF_Cmd... It should fire when there is a mouse event, like: local $oTitle = _FFXpath("//span[contains(.,'Above City Skyscrapers Aerial')]", "", 9) If StringLeft($oTitle, 7) = "OBJECT|" Then $oTitle = StringMid($oTitle, 8) _FFCmd("FFau3.simulateEvent(" & $oTitle &",'MouseEvents','mouseover')") but doesn't work... Link to comment Share on other sites More sharing options...
Danp2 Posted October 31, 2013 Share Posted October 31, 2013 Perhaps a difference between XP and W7? <shrug> You could try using _WinAPI_GetMousePos() instead of MouseGetPos(). Not sure what else to suggest at this point. Latest Webdriver UDF Release Webdriver Wiki FAQs Link to comment Share on other sites More sharing options...
frank10 Posted November 2, 2013 Share Posted November 2, 2013 (edited) I studied a bit more events and mozrepl, so I solved this way: test page: https://github.com/bard/mozrepl/wiki/Custom-commands-library #include <FF.au3> _FFConnect(Default, Default, 3000) _FFCmd("repl.enter(content.wrappedJSObject)") ; without this, 'e' event doesn't work! _FFCmd("var posx; var posy") _FFCMD("function coord(e) { posx = e.clientX + document.body.scrollLeft+ document.documentElement.scrollLeft; posy = e.clientY + document.body.scrollTop + document.documentElement.scrollTop;}"); posX = posx ; posY = posy ;} _FFCmd("var WCD = window.content.document; var element = WCD.getElementsByTagName('A');for (var i = 0; i < element.length; i++) { element[i].addEventListener('mouseover', coord , false);}; ") _FFCmd("repl.home()") local $x = 0, $y = 0 while 1 $x = _FFCmd("content.wrappedJSObject.posx") $y = _FFCmd("content.wrappedJSObject.posy") local $elem = _FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint("&$x&","&$y&")") if _FFCmd("FFau3.xpath") = '' Then ContinueLoop local $aPos = _FFGetPosition("FFau3.xpath") Consolewrite("--CustomFuncPos: " & $x & "," & $y & " FFGetPosition: " & $aPos[0] & "," & $aPos[1] &@crlf) MsgBox(0,'',"--CustomFuncPos: " & $x & "," & $y & " FFGetPosition: " & $aPos[0] & "," & $aPos[1] & " nodo:" & _FFCMD('FFau3.xpath.nodeName') & " text:"& _FFCMD('FFau3.xpath.text') ) Sleep(1000) wend Some links with 'span' tag don't give text with _FFCMD('FFau3.xpath.text') (?) but anyway with normal 'a' elements I get text and correct coord. For example "bard" or "Clone URL" don't give text. So, solved with DOM method. Edited November 2, 2013 by frank10 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now