Webdriver Firefox: Login to a website and get data from it. Learning Webdriver

[Danp2]

@SOLVE-SMARTYour first option is essentially what _WD_ElementActionEx with $sCommand = 'hide'  is doing already, correct?

@Blaxxun We could add a "remove" option to _WD_ElementActionEx. Until then, use _WD_ExecuteScript as indicated above.

[SOLVE-SMART]

4 minutes ago, Danp2 said:

@SOLVE-SMARTYour first option is essentially what _WD_ElementActionEx with $sCommand = 'hide'  is doing already, correct?

I don't know, Dan. I have to check this. But you knows best I guess 😅 .

Best regards
Sven

Update @Danp2: Not exactly but essentially the same, yes.

Case 'hide'
    $iActionType = 2
    $sJavaScript = "arguments[0].style='display: none'; return true;"

 

Edited February 16, 2023 by SOLVE-SMART

[SOLVE-SMART]

There shouldn't be a difference between:

Local Const $sJavaScript = _
    'var element = arguments[0]; ' & _
    'element.setAttribute("display", "none");'

and:

$sJavaScript = "arguments[0].style='display: none'; return true;"

but without proper testing, it's only guessing, I think.

 

11 minutes ago, Danp2 said:

@Blaxxun We could add a "remove" option to _WD_ElementActionEx. Until then, use _WD_ExecuteScript as indicated above.

Nevertheless, a remove option would be helpful. Although I personally don't like these functions with many, many variants. I like it more individually. But as you and @mLipok  started this way, I guess you should continue in the way to be consistent 🤝 .

Edited February 16, 2023 by SOLVE-SMART

[mLipok]

1 hour ago, Blaxxun said:

The "problem" is more of an cosmetic nature.
In the left block there are only shown customer number at the very top but the rest is empty wasted page space.
In the right block is the chart plus the actual table which iam interested in.

The 'hide' works well with an xpath. But it does not make the right block jump to the left.
If i delete the left div block manually, the ight block fills the whole width of the page as desired.

Why you do this ? Is this because you try to clean up view area before export to pdf ?

[Blaxxun]

Hello and thanks!

It is working with:

Func Clean()
    Local $sXp = "/html/body/app-root/div/div/app-consumption/div[3]/div[1]"
    Local $sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sXp)
    Local $sJavaScript = 'var element = arguments[0]; ' & 'element.remove();'
    _WD_ExecuteScript($sSession, $sJavaScript, __WD_JsonElement($sElement), Default, Default)
EndFunc   ;==>Clean

Ok. i get it now. Hard coded xPath's are not the way to go.
Better use "keywords" with " _WD_FindElement"

[Blaxxun]

Hello ladys and gents,

Just wanted to fire up my little table scraper for my power providers statistics site and suddenly ran into an error for an unknown reason.
It is usually my fault because of some crappy code i wrote. But in this case something must have changed since the error accures already in the init phase of
webdriver.

Minimal code:

#include "wd_core.au3"
#include "wd_helper.au3"

Global $sDesiredCapabilities, $sSession
Global $sDriver = "C:\Users\Chris\Documents\Autoit_Programs\Firefox_Autoit_Webdriver\geckodriver.exe" ; <== Please modify this statement to your environment.
; Setup Firefox
_WD_Option('Driver', $sDriver)
_WD_Option('Port', 4444)
$sDesiredCapabilities = '{"desiredCapabilities":{"javascriptEnabled":true,"nativeEvents":true,"acceptInsecureCerts":true}}'
_WD_Startup()
$sSession = _WD_CreateSession($sDesiredCapabilities)
Exit

And the Error(s):

_WD_Option ==>  Success [0] : Parameters:   Option=Driver   Value=C:\Users\Chris\Documents\Autoit_Programs\Firefox_Autoit_Webdriver\geckodriver.exe
_WD_Option ==>  Success [0] : Parameters:   Option=Port   Value=4444
_WD_GetFreePort ==> Success [0] : 4444
_WD_IsLatestRelease ==> Success [0] : False
_WD_Startup: OS:    WIN_7 WIN32_NT 7601 Service Pack 1
_WD_Startup: AutoIt:    3.3.16.1
_WD_Startup: Webdriver UDF: 0.11.0 (Update available)
_WD_Startup: WinHTTP:   1.6.4.1 (Download latest source at <https://raw.githubusercontent.com/dragana-r/autoit-winhttp/master/WinHttp.au3>)
_WD_Startup: Driver:    C:\Users\Chris\Documents\Autoit_Programs\Firefox_Autoit_Webdriver\geckodriver.exe (32 Bit)
_WD_Startup: Params:    
_WD_Startup: Port:  4444
_WD_Startup: Command:   "C:\Users\Chris\Documents\Autoit_Programs\Firefox_Autoit_Webdriver\geckodriver.exe"  
_WD_Startup ==> Success [0]
__WD_Post ==> Webdriver Exception [10] : HTTP status = 500
_WD_CreateSession ==> Webdriver Exception [10]

 

I actually run Win7 64bit here.
No idea what HTTP status = 500 and Webdriver Exception [10] means.

The code ran well since my previous post. Dont know what changed except of some firefox updates.

Thanks in advance,
Chris

 

Edited March 29, 2023 by Blaxxun
typo

[mLipok]

try to use:

$_WD_DEBUG = $_WD_DEBUG_Full
ConsoleWrite($_WD_HTTPRESPONSE & @CRLF)
ConsoleWrite($_WD_SESSION_DETAILS & @CRLF)

check this:

#include "wd_core.au3"
#include "wd_helper.au3"

Global $sDesiredCapabilities, $sSession
Global $sDriver = "C:\Users\Chris\Documents\Autoit_Programs\Firefox_Autoit_Webdriver\geckodriver.exe" ; <== Please modify this statement to your environment.
; Setup Firefox
_WD_Option('Driver', $sDriver)
_WD_Option('Port', 4444)
$sDesiredCapabilities = '{"desiredCapabilities":{"javascriptEnabled":true,"nativeEvents":true,"acceptInsecureCerts":true}}'
$_WD_DEBUG = $_WD_DEBUG_Full
_WD_Startup()
$sSession = _WD_CreateSession($sDesiredCapabilities)
ConsoleWrite($_WD_HTTPRESPONSE & @CRLF)
ConsoleWrite($_WD_SESSION_DETAILS & @CRLF)
Exit

show console output

Edited March 29, 2023 by mLipok

[Danp2]

1 hour ago, Blaxxun said:

_WD_Startup: Webdriver UDF: 0.11.0 (Update available)
_WD_Startup: WinHTTP:   1.6.4.1 (Download latest source at <https://raw.githubusercontent.com/dragana-r/autoit-winhttp/master/WinHttp.au3>)

There are two updates available. The WinHTTP one shouldn't be ignored.

Quote

I actually run Win7 64bit here.
No idea what HTTP status = 500 and Webdriver Exception [10] means.

Probably because you are running the 32 bit version of geckodriver. You can resolve by either --

• switching to 64 bit version of geckodriver
• Add the "binary" option to your capabilities string (you should be able to find examples of this on the forum / wiki).

P.S. Upgrading to the newest UDF release won't fix this issue, but it should give better error logging that helps with diagnosing issues such as this.

[Blaxxun]

Hello,

Okay, so i updated:

geckodriver.exe to   x64 v0.32.2
Webdriver UDF to   v0.13.0
WinHTTP to   1.6.4.2

I switched to the new capabilities format since the one that i used was deprecated. ($sDesiredCapabilities)
Also i added "binary" and the path to the firefox.exe in the capabilities.

I believe there must be something wrong with the capabilities JSON that i generate.
Got most of it from "wd_demo.au3" and the _WD_CapabilitiesAdd('binary', "C:\Program Files\Mozilla Firefox\firefox.exe") i got from the forum here.

I also cant add 'javascriptEnabled' and 'nativeEvents' for some reason.

It already worked so nice and suddenly it does not run anymore. A bit weird

 

Minimal code:

#AutoIt3Wrapper_UseX64=y
#include "wd_core.au3"
#include "wd_helper.au3"
#include "wd_capabilities.au3"

Global $sSession
Global $sDriver = "geckodriver.exe" ; x64 v0.32.2

_WD_Option('Driver', $sDriver)
_WD_Option('Port', 4444)

_WD_CapabilitiesStartup()
_WD_CapabilitiesAdd('alwaysMatch', 'firefox')
_WD_CapabilitiesAdd('browserName', 'firefox')
;_WD_CapabilitiesAdd('javascriptEnabled', True)  // Error: Not supported KEY parameter ( must be defined in $_WD_KEYS__*** )
;_WD_CapabilitiesAdd('nativeEvents', True)       // Error: Not supported KEY parameter ( must be defined in $_WD_KEYS__*** )
_WD_CapabilitiesAdd('acceptInsecureCerts', True)
_WD_CapabilitiesAdd('binary', "C:\Program Files\Mozilla Firefox\firefox.exe")
_WD_CapabilitiesDump()

$_WD_DEBUG = $_WD_DEBUG_Full
_WD_Startup()
$sSession = _WD_CreateSession(_WD_CapabilitiesGet())

ConsoleWrite("HTTPRESPONSE:    " & $_WD_HTTPRESPONSE & @CRLF)
ConsoleWrite("SESSION_DETAILS: " & $_WD_SESSION_DETAILS & @CRLF)

Exit

 

The Errors:

expandcollapse popup

_WD_Option ==> Success [0] : Parameters:   Option=Driver   Value=geckodriver.exe
_WD_Option ==> Success [0] : Parameters:   Option=Port   Value=4444
_WD_CapabilitiesAdd ==> Success [0] : Successfully used [alwaysMatch]  with specified browser: firefox
_WD_CapabilitiesAdd ==> Success [0] : Successfully added capability
_WD_CapabilitiesAdd ==> Success [0] : Successfully added capability
_WD_CapabilitiesAdd ==> Success [0] : Successfully added capability
_WD_CapabilitiesDump: JSON structure starts below: 
{
    "capabilities":{
        "alwaysMatch":{
            "browserName":"firefox",
            "acceptInsecureCerts":true,
            "moz:firefoxOptions":{
                "binary":"C:\\Program Files\\Mozilla Firefox\\firefox.exe"
            }
        }
    }
}
_WD_CapabilitiesDump: JSON structure ends above.
_WD_GetFreePort ==> Success [0] : 4444
_WD_IsLatestRelease ==> Success [0] : True
_WD_Startup: OS:    WIN_7 WIN32_NT 7601 Service Pack 1
_WD_Startup: AutoIt:    3.3.16.1
_WD_Startup: Webdriver UDF: 0.13.0 (Up to date)
_WD_Startup: WinHTTP:   1.6.4.2
_WD_Startup: Driver:    geckodriver.exe
_WD_Startup: Params:    
_WD_Startup: Port:  4444
_WD_Startup: Command:   "geckodriver.exe"  
_WD_Startup ==> General Error [1] : Error launching WebDriver!
__WD_Post: URL=HTTP://127.0.0.1:4444/session; Data={
    "capabilities":{
        "alwaysMatch":{
            "browserName":"firefox",
            "acceptInsecureCerts":true,
            "moz:firefoxOptions":{
                "binary":"C:\\Program Files\\Mozilla Firefox\\firefox.exe"
            }
        }
    }
}
__WD_Post ==> Send / Recv error [6] : HTTP status = 0 ResponseText=WinHTTP request timed out before Webdriver
_WD_CreateSession ==> Webdriver Exception [10]
HTTPRESPONSE:    0
SESSION_DETAILS:

 

[Blaxxun]

Also i noticed:

_WD_Startup: OS:    WIN_7 WIN32_NT 7601 Service Pack 1

This is wrong because my OS is Win_7 64bit

[Danp2]

Make sure your pathing is correct. Is geckodriver in the script directory? If that isn't the issue, then I suggest manually running geckodriver from a cmd prompt and posting the resulting output here.

[Blaxxun]

geckodriver.exe IS in the script directory.

When i run it with CMD i get an error: geckodriver.exe is not a valid Win32 application

What the ... ?

Yes it is the 64bit version of geckodriver.exe but why the error?

[Blaxxun]

Okay, i replaced the 64bit version with the 32bit one of geckodriver.exe and now Firefox fires up!

[Danp2]

47 minutes ago, Blaxxun said:

I also cant add 'javascriptEnabled' and 'nativeEvents' for some reason.

I don't believe these are needed / supported any longer by the webdrivers. If you still have a need, you could manually enable them using _WD_CapabilitiesDefine.

[mLipok]

2 hours ago, Blaxxun said:

__WD_Post ==> Send / Recv error [6] : HTTP status = 0 ResponseText=WinHTTP request timed out before Webdriver

AV / Firewall issue ? 

[mLipok]

1 hour ago, Blaxxun said:

Okay, i replaced the 64bit version with the 32bit one of geckodriver.exe and now Firefox fires up!

Ah here yourproblem was.

So is it works now or do you have other issues ?

[mLipok]

1 hour ago, Blaxxun said:

Also i noticed:

_WD_Startup: OS:    WIN_7 WIN32_NT 7601 Service Pack 1

This is wrong because my OS is Win_7 64bit

Are you sure ?

Ps.

I do not know your IT competence, so please do not offend.

[Danp2]

@mLipokThat information comes directly from the @OS macros, so I suspect that @Blaxxunis mistaken about his OS being 64 bit. This would also explain the "not a valid Win32 application" error.

[Blaxxun]

Hello!

YES! It works now, even without 'javascriptEnabled' and 'nativeEvents'.

I was running geckodriver.exe 32bit before it stopped working. Then was updating everything and replaced the 32bit gecko with the 64bit one.
I also changed the capabilities format into the new one + binarys.

But nothing worked until i switched back to geckodriver.exe 32bit

SO...
My conclusion is that the old capabilities format was no longer supported by firefox or at least it needed the binarys added.

I work since 1992 (windows v3.1) with windows and i'm aware of how to identify it's version. No worrys.
Im just not familiar with webdriver yet.

You can see my windows version in the attached screenshot.

Thanks for your help!

Edited April 2, 2023 by Blaxxun

[Danp2]

@BlaxxunInteresting. So I think we should be displaying @OSArch instead of @OSType. Then it would display

    _WD_Startup: OS:   WIN_11 X64 22621

instead of

    _WD_Startup: OS:    WIN_11 WIN32_NT 22621

Thanks and sorry for the confusion. 🙂

Edited April 2, 2023 by Danp2