WebDriver UDF - Help & Support (II)

frank10 · May 1, 2020

Yes, I tried both:

HttpSetUserAgent('Mozilla / 5.0')

local $sHTML = _INetGetSource($url, True )

and

local $sHTML = InetRead($url, $INET_FORCERELOAD + $INET_IGNORESSL)

both are over 2.4s for each page

Edited May 1, 2020 by frank10

Danp2 · May 1, 2020

@frank10 In my testing, the ajax method is taking around 1.5s, so the difference is around 1s per call. How many times do you anticipate executing this routine during a script?

frank10 · May 1, 2020

Yes, I discovered this, too. It seems I can't get the page in autoit faster like in plain JS... the script is left free faster after 7ms because the ajax call is made and it is async, but if I wait the text coming to autoit I get longer timing, similar to loading pages in tabs...

Calculating timing into JS with ajax, using chrome console, I get 2 pages loaded in 134ms ( using on success: t0 = performance.now(), t1= performance.now() and t1-t0 )

But in Autoit with

$answer = _WD_ExecuteScript($sSession, "return myPage;")

as you, I get more than 1.5s to get the text of a single page! Let's say I could load 60 pages sometimes in a single cycle...

Any idea why ajax on JS page is SO faster (and if we can fast it a bit optimizing autoit)?

EDIT:

I mean, 70ms vs 1500ms is a very BIG difference!! This is plain after JS success func called.

Instead, in JS, timing after putting text into a JS var, becomes 1300ms for 2 pages, that is 650ms per page.

EDIT2

Anyway, making a regExp in JS after the ajax calls, I get 867ms for 2 pages, that is 433ms per page having also some data I need (without assigning all the text page to another global var, directly using the var of the success func).

Edited May 1, 2020 by frank10

Danp2 · May 1, 2020

I imagine the time different can be chalked up to the overhead of the additional layer when using webdriver to communicate with the browser.

Here's a simpler implementation that performs a synchronous call --

$jqueryCommand = "return jQuery.get('www.amazon.com/dp/B07H71XXGY', function(data){ return data.responseText; });"
$answer = _WD_ExecuteScript($sSession, $jqueryCommand)

frank10 · May 1, 2020

Ok, but we'll remain about 2.1s per page... too much.

But it remains strange that making inet or winhttp, that is direct receiving data from the net, instead of communicating with the browser, you get slower results than an ajax call from the browser, that is always a http request.

Maybe I must make all my call and parse inside JS and at the end pass one short var to autoit.

Danp2 · May 1, 2020

Yes, that appears to be the best solution if you require extra fast HTTP requests.

frank10 · May 2, 2020

I get different behaviour on this page

1) https://www.amazon.com/s?k=guide+to+investing+crypto&i=digital-text&page=2&qid=1587374511&ref=sr_pg_2

vs

2) https://www.amazon.com/dp/B087PFN6CT/

after injecting

_WD_jQuerify($sSession)

in 1) you can use both jQuery('div') or $('div')

in 2) you can't use $('div'), you must use only jQuery('div')

Why?

Danp2 · May 2, 2020

See here. Essentially, $ may not be equivalent to jQuery, so your safest bet is to just use the jQuery object directly.

frank10 · May 4, 2020

This does not work:

local $jqueryCommand = " jQuery.parseHTML( '<b>aaa</b>' );"
    _WD_ExecuteScript($sSession, $jqueryCommand )

__WD_Post: URL=HTTP://127.0.0.1:9515/session/1a0dc618d5228451c569f4b78860e3bd/execute/sync; $sData={"script":" jQuery.parseHTML( '<b>a</b>' );", "args":[[]]}
__WD_Post: StatusCode=500; ResponseText={"value":{"error":"javascript error","message":"javascript error: jQuery.parseHTML is not a function\n (Session info:....

Instead in Chrome console it works (after manually inserting jQuery script... why? Shouldn't the page already have the <script> jquery injected from autoit _WD_jQuerify ?).

Edited May 4, 2020 by frank10

Danp2 · May 4, 2020

@frank10 Yes, it should work after you've injected jQuery. My guess is that you navigated to a new page and failed to reinject the script, but I have no way of knowing for sure since you didn't post a reproducer script.

frank10 · May 4, 2020

No, it's simple script like before where you load a page, add _WD_jQuerify and then that line with .parseHTML and does not work.

EDIT

Like this:

#include "wd_helper.au3"
#include "wd_core.au3"

OnAutoItExitRegister("_onExit")

Local $sDesiredCapabilities

$_WD_DEBUG =  True ; You could also use $_WD_DEBUG_Error
;~ You can also control the visibility of the console with the function _WD_ConsoleVisible.

_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--log-path=' & @ScriptDir & '\chrome.log')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--user-data-dir=C:\\Users\\' & @UserName & '\\AppData\\Local\\Google\\Chrome\\User Data\\", "--profile-directory=Default"]}}}}'

_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)

If @error = $_WD_ERROR_Success Then
    _WD_Navigate($sSession, "https://www.amazon.com/dp/B07H71XXGY/")
    _WD_LoadWait($sSession)
    _WD_jQuerify($sSession)

    local $jqueryCommand = " jQuery.parseHTML( '<b>aaa</b>' );"
    _WD_ExecuteScript($sSession, $jqueryCommand )
    
    MsgBox(0,'',"")
EndIf

Func _onExit()
    _WD_DeleteSession($sSession)
    _WD_Shutdown()
EndFunc

Edited May 4, 2020 by frank10

Danp2 · May 4, 2020

If you examine that page in a browser, you'll find that it already has an older version of jQuery loaded. You can determine the exact version by issuing the command "jQuery.fn.jquery" in the browser console, which for me returns 1.6.4. Since the ParseHTML method wasn't added until v1.8, this explains why it isn't working in this case.

You can force the newer version of JQuery to load by running the following command before calling _WD_jQuerify --

_WD_ExecuteScript($sSession, "jQuery = undefined;")

frank10 · May 4, 2020

Thank you, that did it.

HJL · May 4, 2020

On 4/30/2020 at 7:40 PM, Danp2 said:

I'm glad that you found something that works for you. However, it's still unclear which commands that you tried before that resulted in a long pause. To be honest, I don't see why _WD_Timeouts would be faster that other commands, such as _WD_Status, _WD_Action, etc.

Also, you would be better off checking the value of @error since the value returned during an error condition is subject to change in the future.

Good idea to take @error!
I tested WD_Timeouts and WD_Action both working fine and no difference in speed, WD_Status however does not function at all. So my function to test if a session no more exists and I have to open a new session (because I don't want and I am not able to start a second session with FF and same profile).

Func _WD_SessionNotExists($sSession)  ; Check if session still exists
    _WD_Action($sSession,"refresh")    ; or _WD_Timeouts($sSession)  
    Return @error
EndFunc   ; ==>_WD_SessionExists

If you have an idea to improve, please...

HJL · May 4, 2020

Is it possible to reconnect to the old session after closing a script, if you remember the $sSession? For example, prog1.au3 opens a Web page, saves $sSession to a file, and exits. Prog2.au3 starts, reads $sSession from the file, and connects to the web page opened by prog1.au3.

Danp2 · May 5, 2020

11 hours ago, HJL said:

WD_Status however does not function at all.

Can you elaborate on this?

12 hours ago, HJL said:

Is it possible to reconnect to the old session after closing a script, if you remember the $sSession?

Yes. If the chromedriver console is still active, you should be able to continue interaction with it from an additional script. You would perform your normal steps, but skip the calls to _WD_Startup and _WD_CreateSession.

HJL · May 5, 2020

3 hours ago, Danp2 said:

Can you elaborate on this?

image.png.42eac979eebff52fb05d6ed34d158b8f.png

WD_Status returns @extended = 200 even if the $sSession ist changed or the session is closed. But there must be an error in my test script to measure the time because @error is always reported as 0 and in reality all except wd_status report @error <> 0.

#include "wd_core.au3"
#include "wd_helper.au3"
#include <MsgBoxConstants.au3>
;MsgBox($MB_TOPMOST,"Errors", "@error:" & @error & @CRLF & "@extended:" & @extended & @CRLF)

;$_WD_DEBUG = False

Local $sDesiredCapabilities, $sSession, $sURL
Local $sE1 = "WD_Timeout:         @error/                  Time" & @CRLF & @tab & @TAB & "@extended" & @CRLF
Local $sE2 = "WD_Status:"   & @CRLF
Local $sE3 = "WD_Action:"   & @CRLF
Local $sE4 = "WD_Navigate:" & @CRLF

Test()  ; Exit with "Esc"

Func Test()
    SetUpGecko()
    _WD_Startup()
    $sSession = _WD_CreateSession($sDesiredCapabilities)
    _WD_LoadWait($sSession, 3000)
    _WD_Navigate($sSession, "http://wdr.de")
    _WD_LoadWait($sSession, 3000)
    TestTime("Normal:      ")
    _WD_LoadWait($sSession, 3000)
    Local $sSessionSave = $sSession
    $sSession = " / Dummy"
    TestTime("Session=Dummy: ")
    _WD_DeleteSession($sSessionSave)
    TestTime("SessionDeleted: ")
    MsgBox($MB_SYSTEMMODAL, "Time Differences", $sE1 & @CRLF & $sE2 & @CRLF  & $sE3 & @CRLF & $sE4 & @CRLF)
    _WD_ShutDown()
EndFunc  ;==> ReadLoop

Func TestTime($sText)
    Local $hTimer = TimerInit()
    _WD_Timeouts($sSession)
    $sE1 = $sE1 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Status()
    $sE2 = $sE2 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Action($sSession, "refresh")
    $sE3 = $sE3 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Navigate($sSession, "http:sdr.de")
    $sE4 = $sE4 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

EndFunc

Func SetupGecko()
    _WD_Option('Driver', 'geckodriver.exe')
    _WD_Option('DriverParams', '--log trace --marionette-port 2828')
    _WD_Option('Port', 4444)
    $sDesiredCapabilities = '{"capabilities":{"alwaysMatch": {"moz:firefoxOptions": {"args": ["-profile", "C:\\webdriver\\FF_WD_Profile"]}}}}'
EndFunc

Edited May 5, 2020 by HJL

Danp2 · May 5, 2020

There are some issues with your code, as can be seen in the console output --

__WD_Get: URL=HTTP://127.0.0.1:4444/session/ / Dummy/timeouts
__WD_Get: StatusCode=405; $iResult = 0; $sResponseText=HTTP method not allowed...

I'll review the functionality of _WD_Status, but you can see that it does return useful info --

__WD_Get: URL=HTTP://127.0.0.1:4444/status
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":{"message":"Session already started","ready":false}}...

__WD_Get: URL=HTTP://127.0.0.1:4444/status
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":{"message":"","ready":true}}...

Here's a modified version of your code --

#include "wd_core.au3"
#include "wd_helper.au3"
#include <MsgBoxConstants.au3>

Local $sDesiredCapabilities, $sSession, $sURL
Local $sE1 = "WD_Timeout:         @error/                  Time" & @CRLF & @tab & @TAB & "@extended" & @CRLF
Local $sE2 = "WD_Status:"   & @CRLF
Local $sE3 = "WD_Action:"   & @CRLF
Local $sE4 = "WD_Navigate:" & @CRLF

Test()  ; Exit with "Esc"

Func Test()
    SetUpGecko()
    _WD_Startup()
    $sSession = _WD_CreateSession($sDesiredCapabilities)
    _WD_LoadWait($sSession, 3000)
    _WD_Navigate($sSession, "http://wdr.de")
    _WD_LoadWait($sSession, 3000)
    TestTime("Normal:      ")
    _WD_LoadWait($sSession, 3000)
    Local $sSessionSave = $sSession
    $sSession = "Dummy"
    TestTime("Session=Dummy: ")
    _WD_DeleteSession($sSessionSave)
    $sSession = $sSessionSave
    TestTime("SessionDeleted: ")
    MsgBox($MB_SYSTEMMODAL, "Time Differences", $sE1 & @CRLF & $sE2 & @CRLF  & $sE3 & @CRLF & $sE4 & @CRLF)
    _WD_ShutDown()
EndFunc  ;==> ReadLoop

Func TestTime($sText)
    Local $hTimer = TimerInit()
    _WD_Timeouts($sSession)
    $sE1 = $sE1 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Status()
    $sE2 = $sE2 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Action($sSession, "refresh")
    $sE3 = $sE3 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

    Local $hTimer = TimerInit()
    _WD_Navigate($sSession, "http:sdr.de")
    $sE4 = $sE4 & $sText & @TAB & @error&"/"&@extended & @TAB & @TAB & TimerDiff($hTimer) & @CRLF

EndFunc

Func SetupGecko()
    _WD_Option('Driver', 'geckodriver.exe')
    _WD_Option('DriverParams', '--log trace --marionette-port 2828')
    _WD_Option('Port', 4444)
    $sDesiredCapabilities = '{"capabilities":{"alwaysMatch": {"moz:firefoxOptions": {"args": ["-profile", "C:\\webdriver\\FF_WD_Profile"]}}}}'
;   $sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts":true}}}'
EndFunc

seadoggie01 · May 6, 2020

I'm attempting to force PDFs to download in Chrome instead of viewing them. There are already lots of questions about this, where the answer is usually to add something to the capabilities. I've attempted to do this, but it seems it isn't supported anymore. This is the format of my capabilities string:

Spoiler

; I test with one option un-commented and the other removed
    Local $sDesiredCapabilities = _
        '{"capabilities":{' & _
            '"alwaysMatch": {' & _
                '"goog:chromeOptions": {' & _
;~                  '"plugins.plugins_disabled": ["Chrome PDF Viewer"],' & _
;~                  '"plugins.always_open_pdf_externally": true,' & _
                    '"w3c": true, ' & _
                    '"args":[' & _
;~                      '"--disable-extensions",' & _
                        '"--profile-directory=Default"' & _
                    ']' & _
                '}' & _
            '}' & _
        '}' & _
    '}'

Un-commenting either of the first two lines makes the session not be created (_WD_SessionCreate returns an exception). The command line argument does nothing and PDFs are displayed in the browser

Instead, I'm trying to load chrome://settings/content/pdfDocuments and change the setting every session with a click on the toggle. Of course, this isn't simple either, as it deals with ShadowDOM elements. It was a bit tricky at first, but I thought I got it down. After hitting the 3rd ShadowRoot object, however, it seems that JavaScript isn't loaded anymore as _WD_GetShadowRoot times out with this error in the console:

Spoiler

__WD_Post: URL=HTTP://127.0.0.1:9515/session/373186151c5b32925a77dfc38463c9d3/element/1cd884b1-c260-4496-97cc-e25fd0c60f13/element; $sData={"using":"tag name","value":"settings-basic-page"}
__WD_Post: StatusCode=500; ResponseText={"value":{"error":"javascript error","message":"javascript error: b.getElementsByTagName is not a function\n  (Session info: chrome=80.0.3987.100)","stacktrace":"Backtrace:\n\tOrdinal0 [0x00B10C83+1707139]\n\tOrdinal0 [0x00A768F1+1075441]\n\tOrdinal0 [0x009EDFC9+516041]\n\tOrdinal0 [0x009EFC75+523381]\n\tOrdinal0 [0x009EFB64+523108]\n\tOrdinal0 [0x009EFD20+523552]\n\tOrdinal0 [0x00988A82+100994]\n\tOrdinal0 [0x00983991+80273]\n\tOrdinal0 [0x00999ED0+171728]\n\tOrdinal0 [0x00983916+80150]\n\tOrdinal0 [0x0099A0E1+172257]\n\tOrdinal0 [0x009A30F4+209140]\n\tOrdinal0 [0x00999D4B+171339]\n\tOrdinal0 [0x00981D4A+73034]\n\tOrdinal0 [0x00982DC0+77248]\n\tOrdinal0 [0x00982D59+77145]\n\tOrdinal0 [0x00A8BB67+1162087]\n\tGetHandleVerifier [0x00BAA966+508998]\n\tGetHandleVerifier [0x00BAA6A4+508292]\n\tGetHandleVerifier [0x00BBF7B7+594583]\n\tGetHandleVerifier [0x00BAB1D6+511158]\n\tOrdinal0 [0x00A8402C+1130540]\n\tOrdinal0 [0x00A8D4CB+1168587]\n\tOrdinal0 [0x00A8D633+1168947]\n\tOrdinal0 [0x00AA5B35+1268533]\n\tBaseThreadInitThunk [0x76830419+25]\n\tRtlGetAppContainerNamedObjectPath [0x771C66DD+237]\n\tRtlGetAppContainerNamedObjectPath [0x771C66AD+189]\n"}}
__WD_Post ==> Timeout: {"value":{"error":"javascript error","message":"javascript error: b.getElementsByTagName is not a function\n  (Session info: chrome=80.0.3987.100)","stacktrace":"Backtrace:\n\tOrdinal0 [0x00B10C83+1707139]\n\tOrdinal0 [0x00A768F1+1075441]\n\tOrdinal0 [0x009EDFC9+516041]\n\tOrdinal0 [0x009EFC75+523381]\n\tOrdinal0 [0x009EFB64+523108]\n\tOrdinal0 [0x009EFD20+523552]\n\tOrdinal0 [0x00988A82+100994]\n\tOrdinal0 [0x00983991+80273]\n\tOrdinal0 [0x00999ED0+171728]\n\tOrdinal0 [0x00983916+80150]\n\tOrdinal0 [0x0099A0E1+172257]\n\tOrdinal0 [0x009A30F4+209140]\n\tOrdinal0 [0x00999D4B+171339]\n\tOrdinal0 [0x00981D4A+73034]\n\tOrdinal0 [0x00982DC0+77248]\n\tOrdinal0 [0x00982D59+77145]\n\tOrdinal0 [0x00A8BB67+1162087]\n\tGetHandleVerifier [0x00BAA966+508998]\n\tGetHandleVerifier [0x00BAA6A4+508292]\n\tGetHandleVerifier [0x00BBF7B7+594583]\n\tGetHandleVerifier [0x00BAB1D6+511158]\n\tOrdinal0 [0x00A8402C+1130540]\n\tOrdinal0 [0x00A8D4CB+1168587]\n\tOrdinal0 [0x00A8D633+1168947]\n\tOrdinal0 [0x00AA5B35+1268533]\n\tBaseThreadInitThunk [0x76830419+25]\n\tRtlGetAppContainerNamedObjectPath [0x771C66DD+237]\n\tRtlGetAppContainerNamedObjectPath [0x771C66AD+189]\n"}}
_WD_FindElement: {"value":{"error":"javascript error","message":"javascript error: b.getElementsByTagName is not a function\n  (Session info: chrome=80.0.3987.100)","stacktrace":"Backtrace:\n\tOrdinal0 [0x00B10C83+1707139]\n\tOrdinal0 [0x00A768F1+1075441]\n\tOrdinal0 [0x009EDFC9+516041]\n\tOrdinal0 [0x009EFC75+523381]\n\tOrdinal0 [0x009EFB64+523108]\n\tOrdinal0 [0x009EFD20+523552]\n\tOrdinal0 [0x00988A82+100994]\n\tOrdinal0 [0x00983991+80273]\n\tOrdinal0 [0x00999ED0+171728]\n\tOrdinal0 [0x00983916+80150]\n\tOrdinal0 [0x0099A0E1+172257]\n\tOrdinal0 [0x009A30F4+209140]\n\tOrdinal0 [0x00999D4B+171339]\n\tOrdinal0 [0x00981D4A+73034]\n\tOrdinal0 [0x00982DC0+77248]\n\tOrdinal0 [0x00982D59+77145]\n\tOrdinal0 [0x00A8BB67+1162087]\n\tGetHandleVerifier [0x00BAA966+508998]\n\tGetHandleVerifier [0x00BAA6A4+508292]\n\tGetHandleVerifier [0x00BBF7B7+594583]\n\tGetHandleVerifier [0x00BAB1D6+511158]\n\tOrdinal0 [0x00A8402C+1130540]\n\tOrdinal0 [0x00A8D4CB+1168587]\n\tOrdinal0 [0x00A8D633+1168947]\n\tOrdinal0 [0x00AA5B35+1268533]\n\tBaseThreadInitThunk [0x76830419+25]\n\tRtlGetAppContainerNamedObjectPath [0x771C66DD+237]\n\tRtlGetAppContainerNamedObjectPath [0x771C66AD+189]\n"}}
_WD_FindElement ==> Timeout: HTTP status = 500
_WD_GetShadowRoot: 
_WD_GetShadowRoot ==> Timeout

Danp2 · May 6, 2020

@seadoggie01 Can you post a short script that we can run to reproduce the issue?

WebDriver UDF - Help & Support (II)

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

SamsonSlice

Danp2

Jos

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Recently Browsing 0 members