INetGetSource - login only sometimes

[Seminko]

I wrote a script a while back to get the source of a page. The info I'm after only appears if one is logged in.

The strange thing is that when I first wrote the script, it worked. It's like the INetGetSource somehow remembered me being logged in from somewhere.

Now, despite me being logged in in Chrome and in IE, INetGetSource returns the non-logged version of the site.

How does one influence that? Where do I have to login for INetGetSource to recognize me being logged in?

 

[Nine]

Sorry crystal ball at repair shop, and I am quite bad at guessing.  How about, you show us your code with the site being an issue ?

[Seminko]

Hey @Nine,

Thanks for getting back to me.

There is no need for any code, since this is a general question.

"Howcome, INetGetSource sometimes remembers / recognizes me being logged in on a website and sometimes it doesn't. Is there a way to make it recognize that?"

Anyways, I've been on these forums for a few years so I should know that answers for general questions are hard to come by, so here's the story.

 

Me and my gf do geocaching. When we go out together I log the caches on my phone. When we get home, gf looks at my caches and logs hers. Now we got into a situation when there are some caches missing and we need to identify which ones. Now, we both have under 200 caches so manual search is fine, but one has to look to the future. What if that happens further down the line and doing it manually with take a significant amount of time.

So I decided to write a script that would grab her caches and mine, and compare them.

The only relevant piece of code is this one:

$Link = "https://www.geocaching.com/seek/nearest.aspx?ul=Sem%C3%ADnko"
$source = _INetGetSource($Link)
regex stuff etc

Now, you can only see the list if you are logged in at geocaching.com. If you're not, you'll see a login prompt.

The thing is that when I first used the script, it worked perfectly. Now it doesn't.

There are two possibilities, either INetGetSource somehow remembered / recognized me being logged in the whole day I've been using it for OR geocaching.com changed how their site work.

I've been on geocaching.com for a while now and it always required a login to see that data, so I pressume it is the former.

However, I can't find any information regarding that.

[Nine]

I guess you answered you own question.  I would tend to agree with you, it is the site that has made some changes.  Now just use _IE UDF, to login and recuperate the information you need.

[mikell]

You also might use Curl (command line way) to authenticate, manage the cookie if needed and get the infos

[Seminko]

4 hours ago, mikell said:

You also might use Curl (command line way) to authenticate, manage the cookie if needed and get the infos

So I found the singin request and it seems that when signing in it creates a verification token for which I don't seem to be able to find the origin.

Anyways, thanks mikell.

Edited July 24, 2019 by Seminko

[mikell]

Who cares about the origin ? If you can get this token then you can probably send it as an authorization requestheader in your request (using curl or winhttp) 

[Seminko]

That's the thing, I would love to, but when I do, it seems like the token is "expired". I suspect when one wants to log in a verification token is generated and once logged, it is spent and cannot be used again.

EDIT: I'm onto something. Will report back, if it checks out...

Edited July 25, 2019 by Seminko

[Seminko]

Figured it out. It turns out that when you get to the login page, the page generates one token while a different token is generated and saved to a cookie. When you then do curl post, you need to pass both the token as well as the cookie and boom, it works.

 

Thank you very much for the direction, mikell!

[mikell]

@Seminko

It would be super nice if you could post the corresponding piece of code, this will certainly interest many users

[Seminko]

Indeed, here it is:

Global Const $WorkDir = 'C:\curl-7.60.0-win64-mingw\bin' ; PATH TO CURL.EXE

; "FIRST YOU CALL 'CurlInitiate' TO LOG IN AND THEN YOU USE 'CurlThat' THE SAME WAY YOU WOULD USE _INetGetSource"

Func CurlInitiate($login, $pass)
    ; "NAVIGATE TO SIGNIN, SAVE THE COOKIE AND GET THE VERIFICATION TOKEN"
    $Search = 'curl -c cookie-jar.txt https://www.geocaching.com/account/signin'
    $iPID = Run(@ComSpec & " /c " & $Search, $WorkDir, @SW_HIDE, 2)
    ProcessWaitClose($iPID)
    $sOutput = StdoutRead($iPID)
    $aToken = StringRegExp($sOutput, '__RequestVerificationToken\s*" type="hidden" value="([^"]+)', 3)
    If @error Then
        MsgBox(16, "Error", "Token regex error.")
    EndIf
    
    ; "PASS TOKEN, COOKIE AND YOUR LOGIN INFORMATION TO LOG IN TO THE SITE"
    $SearchPre = '__RequestVerificationToken=' & $aToken[0] & '&ReturnUrl=/play/search&UsernameOrEmail=' & $login & '&Password=' & $pass ;
    $Search = 'curl -b cookie-jar.txt -c cookie-jar.txt "https://www.geocaching.com/account/signin" -d "' & $SearchPre & '"'
    $iPID = Run(@ComSpec & " /c " & $Search, $WorkDir, @SW_HIDE, 2)
    ProcessWaitClose($iPID)
    $sOutput = StdoutRead($iPID)
    
    ; "READ $sOutput TO CHECK WHETHER LOGIN SUCCEEDED"
EndFunc

Func CurlThat($sURL)
    $Search = 'curl -b cookie-jar.txt "' & $sURL & '"'
    $iPID = Run(@ComSpec & " /c " & $Search, $WorkDir, @SW_HIDE, 2)
    ProcessWaitClose($iPID)
    $sOutput = StdoutRead($iPID)

    Return $sOutput
EndFunc