InetGet

[Xenobiologist]

Hi,

I'm trying to use InetGet through our companies squid proxy server.

The function

HttpSetProxy(2, 'http://proxy.xxx.de:8080', @UserName, 'xxx')oÝ÷ ÚÇËhÂäN«z«²ÚÞiÈ^²Ø^¦º1ÊÇ«½êËk'«ÝÊ&éâq«b¢pn¶Ø^©e¶ëFzÚ+ ×­ëR¢êÜyû§rبƥÇhp¢¹ÊènW¦ËajÛaz·§¢·©¢Ë"n)b·'è®g­¡×º!jج§
è
C¡ë-çîËb¢{¬zz+©GM3ö¬Û¬zØ^{^®w­{e¢·«­ë(º·³b,Úh²ÈëhëmáÓLúÞ¶êçr^~º&"w­ëS©ä²Ú&èw'讥¦)න¦V¢Z´ÞyÛhz0!©l¢ÚâyÛ(ëax%G­+ºÚ"µÍ[ÈÜÙ][ÞJ ÌÍÜÞK   ÌÍÜÜ
B[Ù]
    ÌÎNÚÜÞIÌÎNË ÌÍÜÞH   [È ÌÎNÎÌÎNÈ  [È ÌÍÜÜ
B[]J
B[[ÈÏOIÝ×ÜÙ][Þ

with the same result.

Mega

P.s.: Till now, I'm using perl or Autoit in combination with wget.exe to download files through proxy (with authentication), but I still would like to switch to pure Autoit.

@Devs/Mods : No problem moving this to support, when you give me a little answer, too.

[Xenobiologist]

Any problems understanding what I want to do?

Can you send me the InetGet function, so that I can have a look myself?

Sorry for being that impatient.

Mega

Edited February 6, 2009 by Xenobiologist

[ProgAndy]

I think, you could use the WinHTTP udfs and setcredentials (included in the udfs). Then you can fetch all information you want.

//Edit: to set the proxy, you have to use WinHTTPSetOption: http://msdn.microsoft.com/en-us/library/aa383912(VS.85).aspx

Edited February 6, 2009 by ProgAndy

[Valik]

It uses the WinInet functions.

Make sure the username and password are correct. Specifically I'm talking about case sensitivity. Since you are using an AutoIt macro you may be passing "Xeno" instead of "xeno" which would obviously fail if the proxy is case sensitive. Try setting up the proxy in IE and make sure that IE works. Then use HTTPSetProxy(0) to use IE's settings and try InetGet() again.

[Xenobiologist]

Thanks to both of you.

I'm one step further, but I've still not reached the goal. All I want to test, is a simple download script for files (binary) with pure Autoit through a squid proxy which needs username and password.

I tested this: (Setting the proxy information in the IE settings) Of course without user and password, cause it is not possible. Then ...

HTTPSetProxy(0)
Global $s_URL = 'myUsername:myPassword@www.autoit.de'
Global $source = _INetGetSource ( $s_URL )
ConsoleWrite($source & @CRLF)oÝ÷ Ù8Z·
+çwÜ­ºÈ§Þ´g­×hzÉ÷öÐ'thpj{m¢Ç­¶©®rwè®f­âÁ©í¶¬zØ^wè®f­ìzا+®*m|¢u欶^²Ü!jج§
èáÇhzÉ÷öÜ(®I覺1ÊÇ­rjëh×6HttpSetProxy(2, 'http://proxy.xxx.de:8080', 'xxx', 'xxx')
Global $s_URL = 'http://www.autoit.de'
Global $source = _INetGetSource ( $s_URL )
ConsoleWrite($source & @CRLF)

Same code with proxy set in IE leads to <TITLE>FEHLER: Cache-Zugriff Verweigert</TITLE>

which means :ERROR cache access denied!!!

@ProgAndy : How to use your nice UDF to get it done? I need to set proxy with credentials and the url.

Mega

P.S: Proxy is not case sensitive.

[weaponx]

I'm wondering if your proxy is looking at the user agent and blocking it. Take a look at this topic about using winhttprequest and changing the user agent:

http://www.autoitscript.com/forum/index.ph...=winhttprequest

You will also need to look into the setcredentials function on MSDN:

http://msdn.microsoft.com/en-us/library/aa384106(VS.85).aspx

Edited February 11, 2009 by weaponx

[ProgAndy]

I think, you could use the WinHTTP udfs and setcredentials (included in the udfs). Then you can fetch all information you want.

//Edit: to set the proxy, you have to use WinHTTPSetOption: http://msdn.microsoft.com/en-us/library/aa383912(VS.85).aspx

@ProgAndy : How to use your nice UDF to get it done? I need to set proxy with credentials and the url.

I think, you have to insert the credential-thing between those lines: (from example in WinHTTP thread)

$h_openRequest = _WinHttpOpenRequest($hw_connect, "GET", "en-us/library/aa384101(VS.85).aspx", "HTTP/1.1")

_WinHttpSendRequest($h_openRequest)

_WinHttpSetCredentials($h_openRequest,$WINHTTP_AUTH_TARGET_PROXY, $WINHTTP_AUTH_SCHEME_BASIC, "USER", "PASS")

//Edit: @weaponx: Xeno said, _InetGetSource works. This func uses "AutoIt v3" as user agent. So this shouldN#t be a problem.

Edited February 11, 2009 by ProgAndy

[Xenobiologist]

Thanks again!

I tried my luck with the WinHTTP.au3 and I was able to get something to work.

#include <WinHTTP.au3>

Global $hw_open = _WinHttpOpen("WinHTTP Example", $WINHTTP_ACCESS_TYPE_NAMED_PROXY, 'proxy.xxx.de:8080')
;~ $hw_connect = _WinHttpConnect($hw_open, "msdn.microsoft.com")
Global $hw_connect = _WinHttpConnect($hw_open, "autoit.de")
If @error = 1 Then ConsoleWrite(1 & @CRLF)
;~ $h_openRequest = _WinHttpOpenRequest($hw_connect, "GET", "en-us/library/aa384101(VS.85).aspx", "HTTP/1.1")
Global $h_openRequest = _WinHttpOpenRequest($hw_connect, "GET", "index.php?page=MembersList", "HTTP/1.1")
If @error = 1 Then ConsoleWrite(2 & @CRLF)
_WinHttpSetCredentials($h_openRequest, $WINHTTP_AUTH_TARGET_PROXY, $WINHTTP_AUTH_SCHEME_BASIC, "xf01145", "sony1234")
If @error = 1 Then ConsoleWrite(3 & @CRLF)
_WinHttpSendRequest($h_openRequest)
_WinHttpReceiveResponse($h_openRequest)

If _WinHttpQueryDataAvailable($h_openRequest) Then
;~     MsgBox(0, "Header", _WinHttpQueryHeaders($h_openRequest))
    ConsoleWrite(_WinHttpQueryHeaders($h_openRequest) & @CRLF)
    ConsoleWrite('! --- ' & @CRLF)
;~     MsgBox(0, 'Text', _WinHttpReadData($h_openRequest, 1, 1000000))    
    ConsoleWrite(_WinHttpReadData($h_openRequest, 1, 1000000) & @CRLF)
EndIf

_WinHttpCloseHandle($h_openRequest)
_WinHttpCloseHandle($hw_connect)
_WinHttpCloseHandle($hw_open)

I received data from both functions.

HEADER

HTTP/1.0 200 OK
Date: Wed, 11 Feb 2009 21:37:51 GMT
Server: Apache/2.2.4 (Linux/SUSE)
X-Powered-By: PHP/5.2.6
Set-Cookie: wcf_cookieHash=2116a2c50364a761116702bd287d5d4d50547c27; HttpOnly
Set-Cookie: wcf_boardLastActivityTime=1234388271; expires=Thu, 11-Feb-2010 21:37:51 GMT; HttpOnly
Content-Type: text/html; charset=UTF-8
X-Cache: MISS from wspprx03.xxx.de
X-Cache: MISS from wspprx07.xxx.de
Via: 1.0 wspprx07.xxx.de:8081 (squid/2.6.STABLE6)
Proxy-Connection: close

Question : How to use it with binary files? Using the func with a download URL and WinHttpReadData($h_openRequest, 2, 10 * 1024 * 1024) didn't work.

And one question back to topic. Does HttpSetProxy really work? Why does it not work like I used it in post one?

Mega

[LarryDalooza]

Question : How to use it with binary files?

I think this has been explored before and runs into a limitation of AutoIt COM and ByRef parameters... but that is just off the top of my head. You can try searching the forums for more.

Lar.

[Xenobiologist]

Hi Larry,

thanks. But using the WinHTTP.au3 was just a try to get it done. Before I start to use COM in Autoit to download stuff through proxy I will use Perl or Java, cause that way I know what happens.

But I have some other stuff around that piece of code, which is much easier to achieve with Autoit and I do not want to combine the languages.

The initial question was, I want to set a proxy with username and password (with Autoit) and then I want to use the "normal" Autoit functions like InetGet und _INetGetSource. Till now I wasn't able to do so.

So, can you help me to investigate why it doesn't work that way?

From my point of view the HttpSetProxy and InetGet functions do not work together.

That is why I asked in developer forum. I do not need any solution for a download through proxy script. I want to use InetGet through proxy, cause there is a SetProxy function available in Autoit.

Mega

[weaponx]

The code I posted above shows how to get binary data.

#551920

[Xenobiologist]

The code I posted above shows how to get binary data.

#551920

Hi,

yes thanks. I will have a look whether I can use your code with proxy configuration.

Anyhow, this was not what I wanted to get by opening this thread.

Mega

[ProgAndy]

To download binary data with WinHTTP, you should use a loop and mode 2:

;~     ConsoleWrite(_WinHttpReadData($h_openRequest, 1, 1000000) & @CRLF)
    Local $data, $hFile = FileOpen(@DesktopDir & "\ReadData.bin",18) ;binary write mode
    While 1
        $data = _WinHttpReadData($h_openRequest, 2) ; read some bytes
        If @error <> 0 Then ExitLoop ; error -1 for EOF or > 0 for other error then ExitLoop
        FileWrite($hFile,$data) ; Write bytes to file
    WEnd
    FileClose($hFile) ;close file

//Edit: _InetGetSource uses the functions from WinInet, but not in UDF-format, just the DLLCalls directly.

WinHTTP is just a simpler Interface for HTTP than WinInet and is provided from Windows, too.

Edited February 12, 2009 by ProgAndy

[Xenobiologist]

Hi ProgAndy,

danke! I've tested your piece of code and it worked perfectly. So, now I'm able to use the WinHTTP.au3 through proxy to download even binary files.

I need to read up the differences between your Http wrapper and the WinInet Wrapper to microsoft API. Maybe this leads me to the solution why HttpSetProxy and InetGet don't work together.

If I understand you correctly, then both "functions" are wrappers to enable the functionality to access internet resources via different kinds of protocols like HTTP.

http://msdn.microsoft.com/en-us/library/aa383630(VS.85).aspx

http://msdn.microsoft.com/en-us/library/aa382925.aspx

WinHTTP is explained to be the better, newer way. Maybe, the Autoit functions should be migrated to WinHTTP.

Thanks again, I learnt a lot!

Mega

[Valik]

This page pretty much sums up why we are using the right thing in AutoIt. In short, we are a client so we should use WinInet. In addition WinInet is available on all systems AutoIt runs on. WinHTTP requires specific minimum service pack versions for Windows 2000 and Windows XP.

Create a new bug report about the proxy issue you are having so I don't forget. I'll see if I can take a look at it sometime in the future.

[Valik]

Mega, is your proxy server HTTP? HttpSetProxy() works *only* with HTTP proxies. With Internet Explore it's possible to get it to use a SOCKS proxy so maybe that's why your proxy works in IE but not in AutoIt?

I'm not sure this is a bug in AutoIt. It's a clear limitation in the WinInet API.

[Xenobiologist]

We are using this one : Wiki EN Squid

Official homepage : Squid Homepage

I'm using this Perl code without any problems.

expandcollapse popup

#!/usr/bin/perl
use warnings;
use strict;
use diagnostics;

use LWP::Simple;
use LWP::UserAgent;

# Variablendeklaration
my $URL                 = "http://www.url.de";
my $fileDestinationPath = "c:\\Temp\\test.txt";
my $user                = "user";
my $password            = "passwort";
my $proxy               = "proxy.xxx.de";
my $port                = "8080";

# Proxy
my $proxyString =
  "http://" . $user . ":" . $password . "\@" . $proxy . ":" . $port;
my $ua = LWP::UserAgent->new();
$ua->proxy( [ 'http', 'ftp' ], $proxyString );

# Datei download
print _getFile( $URL, $fileDestinationPath );

sub _getFile {
    my $url      = shift;
    my $filePath = shift;
    my $request  = HTTP::Request->new( 'GET', $url );
    my $response = $ua->request($request);
    if ( $response->is_success() ) {
        my $content = $response->content();

        # print $content;                    # Test print in die Console
        open( my $DATEI, " > ", $filePath );    # FileHandle zum Schreiben
        binmode($DATEI);                        # Binary Mode aktivieren
        print $DATEI "$content ";               # Datei schreiben
        close($DATEI);                          # FileHandle schliessen
    }
    else {
        return (
            -1, " Error-Code : ",
            $response->code(), " Fehlermeldung : ",
            $response->message()
        );
    }
}

I guess there must be a way to get it done with Windows/Autoit stuff. There are so many tools which do have no problems to go through the proxy when given the credentials like Pidgin, Teamviewer and so on.

Any ideas what I can test or what data I should collect to help you?

Mega

[Valik]

Knowing that it's Squid is enough. I should be able to set it up sometime and test. Or read the documentation and see if I can spot any reason it might not work.

[Xenobiologist]

OKay fine. If you need something I can try, please let me know.

[Valik]

Wow. First, the username and password feature has been broken since day one. Second, how the hell does Microsoft expect those to be set...?