Webdriver Firefox: Login to a website and get data from it. Learning Webdriver

[Blaxxun]

Hello,

I open a new thread here since i dont want to hijack the thread of @water  called "I'm creating a WebDriver tutorial".
As recommended i started to take a deeper look into the "GDPR fines.au3" example.

I also fount that "au3WebDriver-0.11.0" comes with a nice Help file "Webdriver.chm"

When running GDPR fines.au3 i noticed that it uses a function called:

_WD_GetTable($sSession, $sBaseElement, $iPage)

It has 3 function parameters and is IN the "GDPR fines.au3"

So i got an error: wrong number of args.
Further investigation showed me that this function is already defined in the "wd_helper.au3" which is loaded at the beginnings #include but has only 2 parameters.
So i commented out the function in the "wd_helper.au3"

The _WD_GetTable function ran error free now but the next stop was:

_WD_ElementOptionSelect($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']//option[contains(text(),'" & $aOptions[UBound($aOptions, 1) - 2] & "')]")

Variable subscript badly formatted (Error at -> UBound)
 

I'm not really sure why this happens because i dont really know how to traverse a website yet.
Have to dig deeper and read into it more.

[water]

The GDPR fines.au3 was written long time ago. The WebDriver UDF didn't have a _wd_gettable function at that time. Now it has, but I didn't have the time to update the GDPR example script.
I suggest to not modify the WebDriver UDF as it constantly gets enhanced. Drop my _WD_GetTable and use the one provided with the WebDriver UDF.

[Blaxxun]

Okay, i understand. I usually never modify UDF's. But in this case i got confused why there is one version of the _WD_GetTable with 3 and one in the UDF with 2 parameters.
I was able to follow along in the example GDPR until the error.

Is there some similar example that is working and that is not old?

Thanks!

[Danp2]

@BlaxxunHere's a modified version of the GDPR fines example that should work once adjusted for your environment --

expandcollapse popup

; #AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7
#include "wd_core.au3"
#include "wd_helper.au3"
#include "_HtmlTable2Array.au3" ; <== This can be commented out, but the table extraction will be much slower

Global $sDesiredCapabilities, $sSession, $sScriptName = "GDPR fines", $aResult_All
$_WD_DEBUG = $_WD_DEBUG_Error ; Set debug level: $_WD_DEBUG_None (0) = No logging to console, $_WD_DEBUG_Error (1) = Error logging to console, $_WD_DEBUG_Info (2) = Full logging to console (Default)
Global $hSplash = SplashTextOn($sScriptName, "Running ... please be patient!", Default, 45, 0, 0)

; (1, 2) ----------------------------------------------------------------------
; Automate the website using FireFox. Start FF from a different location
; -----------------------------------------------------------------------------
Global $sDriver = "geckodriver.exe" ; <== Please modify this statement to your environment. Must be a local drive, network drives don't work!
; Setup Firefox
_WD_Option("Driver", $sDriver)
If @error Then Exit SetError(1, @error)
_WD_Option('DriverParams', '--log trace')
_WD_Option('Port', 4444)
$sDesiredCapabilities = '{"desiredCapabilities":{"javascriptEnabled":true,"nativeEvents":true,"acceptInsecureCerts":true}}'
; (1, 2) End ------------------------------------------------------------------

_WD_Startup()
If @error Then Exit SetError(2, @error)
$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error Then Exit SetError(3, @error)

; (3) -------------------------------------------------------------------------
; Maximize browser window
; -----------------------------------------------------------------------------
_WD_Window($sSession, "Maximize")
If @error Then Exit SetError(4, @error)
; (3) End ---------------------------------------------------------------------

RetrieveTable($aResult_All)
If @error Then Exit SetError(5, @error)

ControlSetText($hSplash, "", "Static1", "Shutting down WebDriver automation")
_WD_DeleteSession($sSession)
If @error Then Exit SetError(6, @error)
_WD_Shutdown()
If @error Then Exit SetError(7, @error)
SplashOff()
_ArrayDisplay($aResult_All)
MsgBox($MB_ICONINFORMATION, $sScriptName, "Finished!", 5)
Exit

Func RetrieveTable(ByRef $aResult_All)
    Local $aResult, $sElement, $iPage = 0, $sText, $aOptions
    ControlSetText($hSplash, "", "Static1", "Opening website www.enforcementtracker.com")

    ; Open website
    _WD_Navigate($sSession, "http://www.enforcementtracker.com/")
    If @error Then Return SetError(@error)

    ; (4) ---------------------------------------------------------------------
    ; Retrieve the values from the selection list and use the second to last (50)
    ; -------------------------------------------------------------------------
    $sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']")
    $aOptions = _WD_ElementSelectAction($sSession, $sElement, 'options')
    _WD_ElementOptionSelect($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']//option[contains(text(),'" & $aOptions[UBound($aOptions, 1) - 2][0] & "')]")
    If @error Then Return SetError(@error)
    ; (4) End -----------------------------------------------------------------

    ; Wait for initial appearance of navigation button
    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, "//a[@class='paginate_button next']")

    ; (5) ---------------------------------------------------------------------
    ; Sort table by fine (descending)
    ; -------------------------------------------------------------------------
    $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//th[contains(text(),'Fine [')]") ; Find "Fine" header of the table
    _WD_ElementAction($sSession, $sElement, "click") ; Click twice to sort descending
    _WD_ElementAction($sSession, $sElement, "click")
    ; (5) End -----------------------------------------------------------------

    While 1
        $iPage = $iPage + 1
        ControlSetText($hSplash, "", "Static1", "Loading data from website - page " & $iPage)
        $aResult = _WD_GetTable($sSession, "//table[@id='penalties']")
        If @error Then Return SetError(@error)
        If $iPage = 1 Then
            $aResult_All = $aResult
        Else
            _ArrayConcatenate($aResult_All, $aResult, 2)
        EndIf

        ; (6, 7) --------------------------------------------------------------
        ; Find the "next" button. If found scroll down so the button comes into
        ; view and click the button. Else exit the function
        ; ---------------------------------------------------------------------
        $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//a[@class='paginate_button next']") ; Find "Next" button
        If @error Then ExitLoop
        _WD_ExecuteScript($sSession, "arguments[0].scrollIntoView(true);", '{"' & $_WD_ELEMENT_ID & '":"' & $sElement & '"}') ; Scroll down to the link
        If @error Then
            Return SetError(@error)
        Else
            _WD_ElementAction($sSession, $sElement, "click") ; Click on the "Next" button
        EndIf
        ; (6, 7) End ----------------------------------------------------------

    WEnd
EndFunc   ;==>RetrieveTable

 

[Blaxxun]

@Danp2 Hello Dan, thanks for the working example.

I was able to run it and extract all 33 pages of this website which gave me an array of 1600 rows.
Also the UDF from @Gianni  ( _HtmlTable2Array.au3 ) made the whole process way faster. Thanks for that!
I also made a second version of the _HtmlTable2Array.au3 with the recommended changes from @barbossa .
Works also but i could not feel a speed up. Although i did not measure the time. Maybe it only benefits from larger tables. I dont know.

Cool!

Now i can start to disect and learn about the navigation

Thank you guys!

[SOLVE-SMART]

Just a hint @Blaxxun, you will also find help regarding WebDriver in the german forum. In case you're feeling more comfortable in your native language (mal abgesehen vom "Schwitzerdütsch" 😅). Out of your english sentences and grammar behavior, I guess it does not matter to you, but I just want to mention it 🤝 .

Besides that you're in good hands (@Danp2, @mLipok and other great guys here) regarding WebDriver.

Best regards
Sven

[Blaxxun]

@SOLVE-SMART Hi Sven, vielen Dank für den Hinweis. Bin aber eigentlich seit 2016 dabei

Edited February 11, 2023 by Blaxxun

[Blaxxun]

Hello,

So i made it to the website, through the Cookie Popup and the Login.
Also a Tab and a collapse-box.

Now im at the final table for the current day (still need to solve the date selection), but one step after the other.
I have the xpath for the table but i must be doing something wrong with _WD_GetTable and $sBaseElement.

$sElementSelector = "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[11]/div[2]/div/table" ;                  Table
$sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sElementSelector)
ConsoleWrite("Element : " & $sElement & @LF)
$aResult = _WD_GetTable($sSession, $sElement)
_ArrayDisplay($aResult)
Exit

Element : 45892678-1aae-4f91-b275-61b2290a073f
__WD_Post ==> Invalid Expression [11] : HTTP status = 400
_WD_FindElement ==> Invalid Expression [11] : Parameters:   Strategy=xpath   Selector=45892678-1aae-4f91-b275-61b2290a073f   StartNodeID=Default   Multiple=Default   ShadowRoot=Default
_WD_GetTable ==> Invalid Expression [11]

 

I use

$sElement

for

$sBaseElement

like

_WD_GetTable($sSession, $sBaseElement)

Is that wrong? I guess so...

 

[Danp2]

Try this instead --

$sElementSelector = "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[11]/div[2]/div/table" ;                  Table
$aResult = _WD_GetTable($sSession, $sElementSelector)
_ArrayDisplay($aResult)
Exit

 

[Blaxxun]

@Danp2 Thank you Dan!
Okay... i see my error now.

I passed the Element ID instead of an xpath... 🙄

It works now!

"Those who can read have a clear advantage..."

Edited February 11, 2023 by Blaxxun

[Blaxxun]

So, okay, endgoal.
A 3D array. No problem.

2D=Results of 1 day. 3rd D = every other day.
Pretty simple. Like a book.

So.Now. How do i enter a date into this: "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[6]/div/div/div[4]/div[2]/input"

 

[Blaxxun]

I will also need a next, next, next day function that calculates the next day.
Are there any good UDF's ?

[Blaxxun]

Aside of that. How do i even enter any date at all?

[Danp2]

See _WD_ElementAction and _WD_SetElementValue functions in the help file. There should be examples of usage in wd_demo,

[Blaxxun]

@Danp2
I tryd this

_WD_SetElementValue($sSession, $sElement, "8.2.2023") ; and the next one
_WD_ElementAction($sSession, $sElement, 'value', "8.2.2023")

 

[Blaxxun]

It does not change the actual date. It is not doing anything.
I'm stuck at this point

[Danp2]

You should try _WD_SetElementValue with $_WD_OPTION_Advanced. If you still can't get it to work, then I suggest that you find a website that we can all access for testing.

Hint: Try Googling "ngbdatepicker"

[Blaxxun]

@Danp2 Hi Dan, YES! That did the trick! Thank you!

$sElementSelector = "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[6]/div/div/div[4]/div[2]/input" ;         Date Input
$sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sElementSelector)
ConsoleWrite("Element : " & $sElement & @LF)
$sValue = "1.2.2023"
_WD_SetElementValue($sSession, $sElement, $sValue, $_WD_OPTION_Advanced)

 

[Blaxxun]

Hello, i have another question.

I have this <div> block and i want to delete it.

I just can't find a delete option but only a 'hide' option in

_WD_ElementActionEx($sSession, $sElement, $sCommand)

where $sCommand = 'hide'

<div _ngcontent-spe-c94="" class="col-md-4 d-none d-lg-block">
lalalala...
</div>

The "problem" is more of an cosmetic nature.
In the left block there are only shown customer number at the very top but the rest is empty wasted page space.
In the right block is the chart plus the actual table which iam interested in.

The 'hide' works well with an xpath. But it does not make the right block jump to the left.
If i delete the left div block manually, the ight block fills the whole width of the page as desired.

Thanks!!

 

[SOLVE-SMART]

Hi @Blaxxun,

I think you have (at least) two options. Either use setAttribute by JavaScript like this:

Local Const $sDivElement = '...'
Local Const $sJavaScript = _
    'var element = arguments[0]; ' & _
    'element.setAttribute("display", "none");'

_WD_ExecuteScript($sSession, $sJavaScript, __WD_JsonElement($sDivElement), Default, Default)

which sets the display attribute of your DIV to none. This should be similar to remove.

 

Or in case it does not work, you can use JavaScript a with element.remove() call, like this:

Local Const $sDivElement = '...'
Local Const $sJavaScript = _
    'var element = arguments[0]; ' & _
    'element.remove();'

_WD_ExecuteScript($sSession, $sJavaScript, __WD_JsonElement($sDivElement), Default, Default)

 

💡 I didn't tested it with AutoIt 😅 . But I did it in the past successfully with Selenium and WebdriverIO .

See _WD_ExecuteScript function and a usage example in wd_demo.au3.

Best regards
Sven

Edited February 16, 2023 by SOLVE-SMART