Scrape a Javascript rendered web page - (Moved)

[eschettini]

Hi all,

I'd like to know if it is possible to scrape javascript rendered data from a web page and eventually how.

Thanks in advance,

Enrico

[Melba23]

Moved to the appropriate forum, as the AutoIt Example Scripts forum very clearly states:

Quote

Share your cool AutoIt scripts, UDFs and applications with others.

---

Do not post general support questions here, instead use the AutoIt Help and Support forums.

Moderation Team

[Skeletor]

Could you give an example of what you trying to accomplish?

And how about some code... 
 

[eschettini]

Hi Skeletor,

I need to extract javascript rendered data of a web page like this: https://www.hpba.org/Membership/Organization-Search/Organization-Profile/orgcd/111773

but trying with _IEDocReadHTML or _INetGetSource functions I only get the original source of the page without data that browser's javascript engine process as receive the html from the webserver. Here are a couple of examples:

1)

#include <IE.au3>
#include <MsgBoxConstants.au3>

Local $oIE = _IECreate("https://www.hpba.org/Membership/Organization-Search/Organization-Profile/orgcd/111773")
Local $sHTML = _IEDocReadHTML($oIE)
ConsoleWrite($sHTML)

_IEQuit($oIE)

2)

#include <Inet.au3>

Local $sURL = "https://www.hpba.org/Membership/Organization-Search/Organization-Profile/orgcd/111773"
Local $sXML = StringReplace(_INetGetSource($sURL), @CRLF, "")
ConsoleWrite($SXML&@CRLF)

Is there a way to get these kind of data with Autoit ?

Thanks in advance,

Enrico

[Nine]

Try this :

Opt ("MustDeclareVars", 1)
#include <IE.au3>

Local $oIE = _IECreate("https://www.hpba.org/Membership/Organization-Search/Organization-Profile/orgcd/111773")
Local $cFrames = _IEFrameGetCollection ($oIE)
Local $iNumFrames = @extended
Local $sTxt = $iNumFrames & " frames found" & @CRLF & "Error = " & @error & @CRLF
; MsgBox (0,"",$sTxt)
Local $oFrame = 0
$oFrame = _IEFrameGetCollection($oIE, 0)
Local $cTags = _IETagNameGetCollection ($oFrame, "dd")
Local $sInfo = "", $sTag, $iS
For $oTag in $cTags
  $sTag = $oTag.innerText
  $iS = StringInStr ($sTag,"; }",0,-1)
  if $iS then $sTag = StringMid($sTag,$iS+4)
  $sInfo &= $sTag & @CRLF
Next
MsgBox (0,"",$sInfo)

_IEQuit($oIE)

 

Edited February 19, 2019 by Nine

[eschettini]

Thanks a lot Nine!!! It works fine.

[Nine]