Danp2 Posted July 9, 2020 Author Posted July 9, 2020 @nooneclose This is Troubleshooting 101 -- Do you have an existing instance of Firefox open? Was it launched with Marionette enabled? Is the correct Marionette port being used? You can check the last two items under the About:Config page. Also, I just noticed this in your post -- Quote _WDStartup: WinHTTP: 1.6.4.1 See troubleshooting section in the Wiki -- https://www.autoitscript.com/wiki/WebDriver#Troubleshooting Also this -- Quote _WDStartup: WD.au3: 0.2.0.9 (Update available) You are multiple versions behind the latest release. 🙄 Latest Webdriver UDF Release Webdriver Wiki FAQs
nooneclose Posted July 9, 2020 Posted July 9, 2020 @Danp2 how do I check if Marionette is enabled? yes I did have an existing instance of Firefox open. I just finished updating both the Webdriver UDF and the WinHTTP UDF
Danp2 Posted July 9, 2020 Author Posted July 9, 2020 @nooneclose Perhaps you missed this from my earlier post -- Quote You can check the last two items under the About:Config page Go to About:Config, click the Accept button, then filter by "Marionette" Latest Webdriver UDF Release Webdriver Wiki FAQs
nooneclose Posted July 9, 2020 Posted July 9, 2020 (edited) @Danp2 It works now with the existing Firefox browser. Though it doesn't work every time? It acts like it doesn't get the session every run. To better test that statement I tried to run the code 15 times back to back: Worked 11 times, Did not work: 4 times. When it doesn't work the output says: _WDStartup: Existing instance of geckodriver.exe detected! When it doesn't say that it works. Thank you very much for your help. Edited July 9, 2020 by nooneclose
Danp2 Posted July 9, 2020 Author Posted July 9, 2020 @nooneclose Glad you got it working, even if it's only intermittently. FWIW, I don't recall having this fail, so it's likely something on your end. 😛 I don't think geckodriver supports multiple sessions, so maybe it's failing because you haven't completed the prior session. No way for me to know for sure without more details. nooneclose 1 Latest Webdriver UDF Release Webdriver Wiki FAQs
nooneclose Posted July 9, 2020 Posted July 9, 2020 @Danp2 maybe its do to my code messing up 😝. Im fixing another issue with excel opening a file so I keep killing my code and running it. Most likely the brakes are my fault. Thanks again though. you were a huge help.
crazycrash Posted July 11, 2020 Posted July 11, 2020 (edited) On 7/9/2020 at 12:11 AM, Danp2 said: @crazycrash Here's a revised version that works AFAICS -- expandcollapse popup#include "wd_core.au3" #include "wd_helper.au3" Local $sDesiredCapabilities, $sSession, $sElement Local $sImageXpath = "//a[@rel='theater' and @data-ploi]" Local $sContentXpath = "//ancestor::div[@class='_3x-2']/preceding-sibling::div[contains(@class,'userContent')]" SetupChrome() _WD_Startup() $sSession = _WD_CreateSession($sDesiredCapabilities) If @error = $_WD_ERROR_Success Then ConsoleWrite("Starting extraction: " & @CRLF) _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/") _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sImageXpath, 2000) $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sImageXpath, Default, True) ; get image elements For $i = 0 To UBound($aElements, 1) - 1 $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath) ; get user content elements If @error = $_WD_ERROR_Success Then $sComment = _WD_ElementAction($sSession, $sElement, 'attribute', 'innerText') ConsoleWrite($sComment & @crlf & "===============" & @CRLF) Sleep(350) EndIf Next EndIf _WD_DeleteSession($sSession) _WD_Shutdown() Func SetupChrome() _WD_Option('Driver', 'chromedriver.exe') _WD_Option('Port', 9515) _WD_Option('DriverParams', '--verbose --log-path="' & @ScriptDir & '\chrome.log"') $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}' EndFunc I can confirm that on Chrome everything works just fine. It seems the "innertext" issue is then definitely related to Firefox. I also tried "textContent" without luck on Firefox. "textContent" also works on Chrome. Also, could you explain how the relative Xpath to the previous element works? Can one specify the element to which the xpath should be relative to? Maybe it's possible that this doesn't behave the same way as it does in Firefox? For Example: //ancestor::div[@class='_5pcr userContentWrapper']//span[contains(@class,'timestampContent')] always returns the same timestamp, the first one in the blog. Edit: I just realized that also in Chrome it always returns the same text and date. Hence it seems the relative path is not related to the current element being processed. Cheers Edited July 11, 2020 by crazycrash
Danp2 Posted July 11, 2020 Author Posted July 11, 2020 @crazycrash Seems like innerText is a property, not an attribute. Try this and let me know how it works for you -- expandcollapse popup#include "wd_core.au3" #include "wd_helper.au3" Local $sDesiredCapabilities, $sSession, $sElement ;~ Local $sImageXpath = "//a[@rel='theater' and @data-ploi]" ;~ Local $sContentXpath = "//ancestor::div[@class='_3x-2']/preceding-sibling::div[contains(@class,'userContent')]" Local $sContentXpath2 = "//div[@data-testid='post_message' and contains(@class, '_5pbx')]" Local $sJavascript = "window.scrollTo(0, document.body.scrollHeight)" SetupGecko() _WD_Startup() $sSession = _WD_CreateSession($sDesiredCapabilities) If @error = $_WD_ERROR_Success Then ConsoleWrite("Starting extraction: " & @CRLF) _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/") ; force page to load dynamic content For $i = 1 To 3 _WD_ExecuteScript($sSession, $sJavascript) Next _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath2, 2000) $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath2, Default, True) For $i = 0 To UBound($aElements, 1) - 1 $sComment = _WD_ElementAction($sSession, $aElements[$i], 'property', 'innerText') ConsoleWrite($sComment & @crlf & "===============" & @CRLF) Sleep(350) Next EndIf _WD_DeleteSession($sSession) _WD_Shutdown() Func SetupChrome() _WD_Option('Driver', 'chromedriver.exe') _WD_Option('Port', 9515) _WD_Option('DriverParams', '--verbose --log-path="' & @ScriptDir & '\chrome.log"') $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}' EndFunc Func SetupGecko() _WD_Option('Driver', 'geckodriver.exe') _WD_Option('DriverParams', '--log trace') _WD_Option('Port', 4444) $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts":true}}}' EndFunc crazycrash 1 Latest Webdriver UDF Release Webdriver Wiki FAQs
crazycrash Posted July 12, 2020 Posted July 12, 2020 Thanks a lot for the guidance. Everything works beautifully on any browser.
crazycrash Posted July 12, 2020 Posted July 12, 2020 (edited) It seems I still fail to find an element within or related to a previously found element. Could you show me an example of how to use _WD_FindElement to search within another or from that element? I tried the following to get the images related to the comments in the posts: Local $sXpathPosts = "//div[@data-testid='post_message' and contains(@class, '_5pbx')]" Local $sJavascript = "window.scrollTo(0, document.body.scrollHeight)" Local $XpathPostTimeStamp = "//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]" Local $XpathPostImages = "//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]" $sSession = _WD_CreateSession($sDesiredCapabilities) If @error = $_WD_ERROR_Success Then ConsoleWrite("Starting extraction: " & @CRLF) _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/") ; force page to load dynamic content For $i = 1 To 3 _WD_ExecuteScript($sSession, $sJavascript) Next _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, 2000) Sleep(2000) $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, Default, True) For $i = 0 To UBound($aElements, 1) - 1 $sComment = _WD_ElementAction($sSession, $aElements[$i], 'property', 'innerText') ConsoleWrite($sComment & @CRLF) $sTime = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $XpathPostTimeStamp, $aElements[$i], False) $sUtimestamp = _WD_ElementAction($sSession, $sTime, 'attribute', 'data-utime') ConsoleWrite("Timestamp extracted: " & $sUtimestamp & @CRLF) $aImages = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $XpathPostImages, $aElements[$i], True) ConsoleWrite("Images: " & UBound($aImages, 1) - 1 & @CRLF) ConsoleWrite(@CRLF & "==============Done with current post ================" & @CRLF) Sleep(350) Next EndIf _WD_DeleteSession($sSession) _WD_Shutdown() Console out: expandcollapse popup__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/url; $sData={"url":"https://www.facebook.com/AstrophysicsAndAstronomy/"} __WD_Post: StatusCode=200; ResponseText={"value":null}... _WD_Navigate: {"value":null} __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]} __WD_Post: StatusCode=200; ResponseText={"value":null}... _WD_ExecuteScript: {"value":null}... __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]} __WD_Post: StatusCode=200; ResponseText={"value":null}... _WD_ExecuteScript: {"value":null}... __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]} __WD_Post: StatusCode=200; ResponseText={"value":null}... _WD_ExecuteScript: {"value":null}... __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element; $sData={"using":"xpath","value":"//div[@data-testid='post_message' and contains(@class, '_5pbx')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"}} _WD_WaitElement ==> Success __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//div[@data-testid='post_message' and contains(@class, '_5pbx')]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"},{"element-6066-11e4-a52e-4f735466cecf":"32260115-9a73-4e25-bc49-8366caa8f00d"},{"element-6066-11e4-a52e-4f735466cecf":"2f8952dc-a35c-49b8-97f4-3e4254a04b3b"},{"element-6066-11e4-a52e-4f735466cecf":"19be5389-6d9e-47e7-9807-052ffaeb214d"},{"element-6066-11e4-a52e-4f735466cecf":"a99ae5ca-7045-4bc1-9dbd-d45580e2a22f"}]} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/8d023ade-5572-457a-813f-fbc2d9613fbb/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"I do have to admit that although I have seen the published data plots, a true understandin... _WD_ElementAction: {"value":"I do have to admit that although I have seen the published data plots, a true understandin... I do have to admit that although I have seen the published data plots, a true understanding of the Higgs boson and some of the high energy particle physics done at CERN has eluded me, so I'm listening to and sharing this series on the discovery of the Higgs boson this week. This first episode describes the utility of colliding things at high energy in order to generate new particles and exchange energy for mass in the process. __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/8d023ade-5572-457a-813f-fbc2d9613fbb/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}... _WD_ElementAction: {"value":"1594561734"}... Timestamp extracted: 1594561734 __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]} Images: 4 ==============Done with current post ================ __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/32260115-9a73-4e25-bc49-8366caa8f00d/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"Comparing the Gas Giants\n\nOur solar system has 4 large planets made up mostly of gas: Ju... _WD_ElementAction: {"value":"Comparing the Gas Giants\n\nOur solar system has 4 large planets made up mostly of gas: Ju... Comparing the Gas Giants Our solar system has 4 large planets made up mostly of gas: Jupiter, Saturn, Uranus, and Neptune. Each of them have some things in common; large numbers of moons, rings present, gaseous outer layers in their atmosphere, and it is thought each likely has some sort of rocky core around which the gases originally were collected as the planets formed. However, the type of material in the planet’s interior depends on what it is made of and how big the plan... See More __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/32260115-9a73-4e25-bc49-8366caa8f00d/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}... _WD_ElementAction: {"value":"1594561734"}... Timestamp extracted: 1594561734 __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]} Images: 4 ==============Done with current post ================ __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/2f8952dc-a35c-49b8-97f4-3e4254a04b3b/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Cen... _WD_ElementAction: {"value":"Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Cen... Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Centauri A and B, 4.2, 4.35, and 4.37 light years away, respectively. At least one planet is thought to exist there. What else might we find out about this solar system? __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/2f8952dc-a35c-49b8-97f4-3e4254a04b3b/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}... _WD_ElementAction: {"value":"1594561734"}... Timestamp extracted: 1594561734 __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]} Images: 4 ==============Done with current post ================ __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/19be5389-6d9e-47e7-9807-052ffaeb214d/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"An incredible view\n\nThis image was taken at 3060m above sea level (10,000ft), on the mou... _WD_ElementAction: {"value":"An incredible view\n\nThis image was taken at 3060m above sea level (10,000ft), on the mou... An incredible view This image was taken at 3060m above sea level (10,000ft), on the mountain peak of Cerro Armazones in Chile. The high number of star trails is testament to the dark, clear site. This is one of the reasons why this mountain top will host the upcoming European Extremely Large Telescope (E-ELT). The E-ELT, nicknamed the “world's biggest eye on the sky” will have a 39m (128ft) primary mirror and will constitute the largest optical/near-infrared telescope on Eart... See More __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/19be5389-6d9e-47e7-9807-052ffaeb214d/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}... _WD_ElementAction: {"value":"1594561734"}... Timestamp extracted: 1594561734 __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]} Images: 4 ==============Done with current post ================ __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/a99ae5ca-7045-4bc1-9dbd-d45580e2a22f/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"This is what it is like to start off with an overview of our night sky and zoom in on the ... _WD_ElementAction: {"value":"This is what it is like to start off with an overview of our night sky and zoom in on the ... This is what it is like to start off with an overview of our night sky and zoom in on the Andromeda Galaxy. __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/a99ae5ca-7045-4bc1-9dbd-d45580e2a22f/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}... _WD_ElementAction: {"value":"1594561734"}... Timestamp extracted: 1594561734 __WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]} Images: 4 ==============Done with current post ================ __WD_Delete: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803 __WD_Delete: StatusCode=200; ResponseText={"value":null}... AS you can see I always get the 4 Images, same elements. An example to only search within another element in java taken from seleniumjava: WebElement result = driver.findElement(RESULT_BOX_XPATH); WebElement title = result.findElement(TITLE_XPATH); I'm going to try to search for a child element like that afterwards, will post an update in a few hours. Thanks Edit: Added xpath's and console out 😃 Edited July 12, 2020 by crazycrash
Danp2 Posted July 12, 2020 Author Posted July 12, 2020 @crazycrash It would help if you includ the xpath strings needed to run your code. 😃 Also, should your results from Scite output panel. Latest Webdriver UDF Release Webdriver Wiki FAQs
crazycrash Posted July 12, 2020 Posted July 12, 2020 (edited) I fixed the post above trying to get the xpath relative to another previously found element. Below my attempt at getting a child element of a previously found element. (Similar to the javascribt above). However, this is also failing. Local $sXpathPosts = "//div[@class='_5pcr userContentWrapper']" Local $sXpathUserContent = "//div[contains(@class, 'userContent')]" ;xpath within element $PostContent to usercontent Local $sXpathPostTimeStamp = "//div[contains(@class, '_6a')]//abbr[contains(@class, '5ptz')]" ;xpath within element $PostContent to timestamp Local $sXpathPostImages = "//a[@rel='theater' and @data-ploi]" ;xpath within element $PostContent to images $sSession = _WD_CreateSession($sDesiredCapabilities) If @error = $_WD_ERROR_Success Then ConsoleWrite("Starting extraction: " & @CRLF) _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/") ; force page to load dynamic content For $i = 1 To 3 _WD_ExecuteScript($sSession, $sJavascript) Next _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, 2000) Sleep(2000) $ePosts = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, Default, True) For $i = 0 To UBound($ePosts, 1) - 1 $eUserContent = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathUserContent, $ePosts[$i], False) If @error = $_WD_ERROR_Success Then ;if there is a user content $sUserContent = _WD_ElementAction($sSession, $eUserContent, 'property', 'innerText') ConsoleWrite($ePosts & @CRLF) EndIf ConsoleWrite(@CRLF & "==============Done with current post ================" & @CRLF) Sleep(350) Next EndIf _WD_DeleteSession($sSession) _WD_Shutdown() Console out (always finds the same element for the post content, as where I would like to find the usercontent child element of the post element) expandcollapse popup__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element; $sData={"using":"xpath","value":"//div[@class='_5pcr userContentWrapper']"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} _WD_WaitElement ==> Success __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/elements; $sData={"using":"xpath","value":"//div[@class='_5pcr userContentWrapper']"} __WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"},{"element-6... _WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"},{"element-6066-11e4-a52e-4f735466cecf":"cbc1ba94-029a-49cd-80c4-3cf5718859b2"},{"element-6066-11e4-a52e-4f735466cecf":"13fb39a9-629b-449c-b30b-a3cd6497d659"},{"element-6066-11e4-a52e-4f735466cecf":"8f37bd3e-0e7e-4a1d-9394-b67e2189a5da"},{"element-6066-11e4-a52e-4f735466cecf":"0a87bf1b-3c98-43a9-8c88-60eac244bcff"},{"element-6066-11e4-a52e-4f735466cecf":"81360c07-65d4-4f7d-8fbc-befecf52ff26"},{"element-6066-11e4-a52e-4f735466cecf":"f0221fa5-e46f-41cc-8012-2a34bbfea96a"},{"element-6066-11e4-a52e-4f735466cecf":"bdc6a3dd-42ba-4c48-8302-14b081e443e6"}]} __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/cbc1ba94-029a-49cd-80c4-3cf5718859b2/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/13fb39a9-629b-449c-b30b-a3cd6497d659/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/8f37bd3e-0e7e-4a1d-9394-b67e2189a5da/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/0a87bf1b-3c98-43a9-8c88-60eac244bcff/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/81360c07-65d4-4f7d-8fbc-befecf52ff26/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/f0221fa5-e46f-41cc-8012-2a34bbfea96a/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/bdc6a3dd-42ba-4c48-8302-14b081e443e6/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"} __WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}... _WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}} __WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText __WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... _WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul... ==============Done with current post ================ __WD_Delete: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995 __WD_Delete: StatusCode=200; ResponseText={"value":null}... _WD_DeleteSession: {"value":null} So basically I always get the same element for the usercontent element instead of the usercontent child element of the postelement. The reason for all this is that sometimes there is text, sometimes not, sometimes there is one or multiple images. Hence I need the relationship between those elements. Thanks Edited July 13, 2020 by crazycrash
cramaboule Posted July 13, 2020 Posted July 13, 2020 Hi, I have an internal website like: https://mywebsite Chrome, obviously is warning me because of certificate, that obviously I do not have. How to bypass, this warning ? Cramaboule My Autoit programs: MAC Address - - Delete Temp Files - - Ping Test - - Play Video with VLC full screen dual monitors - - Set IP - - Pics Converter - - AutoUpdater - - CPU Usage - - Ending Script Nicely - - GDI+ GUI crossfades (slide transitions) - - Beamer - - Search and Search in Files - - Silent Ninite Others: Export Icons into Dll - - My website
Danp2 Posted July 13, 2020 Author Posted July 13, 2020 @cramaboule Have you tried adding the following to your DesiredCapabilities? "args":["--ignore-certificate-errors"] Latest Webdriver UDF Release Webdriver Wiki FAQs
cramaboule Posted July 13, 2020 Posted July 13, 2020 (edited) 9 minutes ago, Danp2 said: @cramaboule Have you tried adding the following to your DesiredCapabilities? "args":["--ignore-certificate-errors"] __WD_Post: URL=HTTP://127.0.0.1:9515/session; $sData={"capabilities":{"args":["--ignore-certificate-errors"]}} __WD_Post: StatusCode=200; ResponseText={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"83.0.... It dosn't work !!! 😞 C. *EDITED the response code Edited July 13, 2020 by cramaboule My Autoit programs: MAC Address - - Delete Temp Files - - Ping Test - - Play Video with VLC full screen dual monitors - - Set IP - - Pics Converter - - AutoUpdater - - CPU Usage - - Ending Script Nicely - - GDI+ GUI crossfades (slide transitions) - - Beamer - - Search and Search in Files - - Silent Ninite Others: Export Icons into Dll - - My website
Danp2 Posted July 13, 2020 Author Posted July 13, 2020 @cramaboule I don't think you implemented it correctly. It should look like this -- $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}' Note: I didn't actually test this, so YMMV. 😉 If that still doesn't work, then I know for a fact that this does -- $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"acceptInsecureCerts": true, "goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}' Latest Webdriver UDF Release Webdriver Wiki FAQs
cramaboule Posted July 13, 2020 Posted July 13, 2020 1 minute ago, Danp2 said: @cramaboule I don't think you implemented it correctly. It should look like this -- $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}' Note: I didn't actually test this, so YMMV. 😉 If that still doesn't work, then I know for a fact that this does -- $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"acceptInsecureCerts": true, "goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}' Quote $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}' Works fine for me... Thanks a lot Cramaboule. My Autoit programs: MAC Address - - Delete Temp Files - - Ping Test - - Play Video with VLC full screen dual monitors - - Set IP - - Pics Converter - - AutoUpdater - - CPU Usage - - Ending Script Nicely - - GDI+ GUI crossfades (slide transitions) - - Beamer - - Search and Search in Files - - Silent Ninite Others: Export Icons into Dll - - My website
crazycrash Posted July 13, 2020 Posted July 13, 2020 (edited) I found what I d like to do in the official documentation of selenium: selenium.dev -> Find Element(s) From Element If you could show me how to do that in AutoIt it would answer my previous questions 😃 Thanks Edited July 13, 2020 by crazycrash
Danp2 Posted July 13, 2020 Author Posted July 13, 2020 @crazycrash That is what _WD_FindElement already does, but it isn't working correctly for some reason. I'll have to review the code to see if it's something that I've done wrong. crazycrash 1 Latest Webdriver UDF Release Webdriver Wiki FAQs
cal Posted July 14, 2020 Posted July 14, 2020 On 7/3/2020 at 3:55 PM, cal said: On to the actual issue. I just started looking into the webdriver. I've hit a snag that I'm not seeing talked about. On a whim I tried to run the same exact code at work and its fine. All I'm doing is opening chrome, moving to another page and closing. I'm just seeing how things work. When I try and open chrome using my existing profile, two of my extensions are removed by the webdriver. I've been hunting for the issue for a couple days when I tried running it at work and it does not have the same issue Responding to my own post.... A clean test profile had the same issues. It fine at work fine but removes lastpass at home. Same code. Today I decided to look into this deeper. After completely redoing Chrome a couple times and playing around a lot with nothing fixing the problem I found the issue. Turns out the issue was the way that lastpass was installed. At least I think thats the case. I have no issues with lastpass so was not looking at it as the source of the problem. I had one odd behavior that I thought was chrome syncing extensions. Lastpass would be removed by the webdriver. Then after starting chrome normally the extension was automatically put back. I thought this was chrome doing a sync and reinstalling the missing extension. But during my test today I had a clean blank chrome install with no accounts logged in or syncing. .... and the lastpass extension installed itself without me doing so on a fresh blank install. Since I was not logged into chrome and I could see the sync was off, where did it come from? I don't know for sure but I believe its because I had done a full Lastpass install on my home computer and it was running as a service in the background. This service is what reinstalled it. Removing this caused lastpass to no longer self install when I opened chrome. Likely I installed lastpass at work via the extensions and never did the full thing. Its always synced just fine between Firefox/chrome both at home and work. All 4 browsers stayed in sync. My guess is that the full lastpass service was seeing chrome start using the webdriver and stepping in and removing the lastpass extension. Then when chrome was started without the webdriver, it restored the extension. Removing the full lastpass install and using just the extension install method has fixed the issue. Lastpass is no longer being removed when the webdriver starts. Cal.
Recommended Posts