NetSession UDF ~ Per-Process Embedded Browser Settings (Proxy/Agent/Etc.)

[drego]

NetSession UDF 0.9d

(Set embedded browser Internet options on a per-process basis)

 

Welcome to NetSession UDF! With this UDF we provide you with previously unexposed Internet options for your autoit processes which will apply to embedded browser controls created with _IECreateEmbedded. For instance, you can now set a proxy and user-agent for all for your Autoit application's embedded browsers without using the registry settings. This means that you can now have multiple proxy settings for multiple programs/processses, giving each process it's own proxy/agent/other settings for your embedded browser controls and change them within each application as many times as you like. In other words, dynamic per-program (Not shared) Internet settings for any application containing an embedded browser control. There's also a function to clear all of your browser's cookies as well as all flash cookies.

Versions:

1. Version 0.9 includes the ability to set an HTTP/SOCKS4/SOCKS5 proxy and browser agent for individual AutoIt processes that can all be running at the same time with different settings. This is easy to use. This UDF started out as code that I worked on years ago and now we're bringing it back to life as a UDF with new features planned.
2. Version 0.9b now has the new _ClearCookies function which will remove all IE and Flash cookies.
3. Version 0.9c has added the "#include-once" directive as a UDF should and an optimization to the _ClearCookies function thanks to jdelaney. _ClearCookies has also been changed to return TRUE or FALSE rather than strings, not pause script execution when deleting IE cookies, and the hidden window flag added for Its shell command.
4. Version 0.9d has the added function _UseTOR. This release comes just minutes after the last release because it slipped my mind that it was on the Planned Features list and is easy to add. This new function simply envokes _SetProxy with the proper parameters for TOR with the option to change the port if your install of TOR isn't using the default port number; otherwise, no parameter is necessary. This will likely be the last release until we have more updates to the DLL/Windows API calls.

 

At the moment this UDF makes use of urlmon.dll to apply settings to your application executables. Our goal is to expose all functionality of this DLL as well as possibly expand into more functionality using other Windows API elements.

Important: If you wish to change the settings of an IE control more than once during the life if your program you might have to refresh the IE control or GUI of your app before applying the next change. This was the case years ago. I'll do some testing on this and publish an update to the UDF or add an example of a refresh to the Example Usage.

Example usage & notes:

expandcollapse popup

#include <NetSession-0.9d.au3>

; Let's include IE.au3 just for this example assuming you're making an embedded browser.
#include <IE.au3>

; In this example you would create your embedded IE object and use as follows.

_SetProxy('108.247.158.12:8080')

; Now your proxy is set. This would be an HTTP proxy. For SOCKS you would use:

_SetProxy('socks=108.247.158.12:1080')

; Or if you have TOR running and would like to use it as your proxy simply use:

_UseTOR()

; But if your TOR isn't running on default port 9150 (Like if you changed it to 9292) you must specify your TOR port:

_UseTOR(9292)

; The following is an example of setting your user agent:

_SetUserAgent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36')

; At this point any browsing you perform will use the previously set proxy and user agent. Such as:

_IENavigate($ie,"http://www.ipchicken.com")

; If your proxy or SOCKS requires authentication you would add the proxy auth to the URL like this:

_IENavigate($ie,"http://ProxyUser:ProxyPass@www.ipchicken.com")

; Let's clear all browser & Flash cookies

_ClearCookies()

; Note that you can change these settings as many times as you need to in your application and these
; settings will only be applied to the .exe they were set from. This means you don't have to change
; the system-wide proxy or user-agent settings and can have per-.exe embedded browser settings.

I welcome anyone who wants to contribute. Any suggestions will be much appreciated.

The UDF NetSession-0.9d.au3 is attached to this post. The filename will always reflect the version number and this thread post will be kept up to date. I've chosen 0.9 because at this point we only need to figure out one more thing to consider it a complete 1.0. Version 1.0 will include an advanced authentication mechanism for proxies and socks servers (Better than passing proxy auth via the URL). The reason we want to use the authentication capabilities of the DLL is because when you pass authentication via the URL you can't access HTTPS sites through the proxy (You can if no proxy auth is required).

If you can help make this UDF better by improving authentication as just previously mentioned or add any other functionality PLEASE don't hesitate to reply to this thread with some code. Here are the references to how this UDF came about and where you would look to help improve it:

• http://msdn.microsoft.com/en-us/library/ms775125%28v=vs.85%29.aspx
• http://msdn.microsoft.com/en-us/library/aa385148%28v=vs.85%29.aspx
• http://msdn.microsoft.com/en-us/library/aa385328%28v=vs.85%29.aspx
• http://msdn.microsoft.com/en-us/library/aa383630%28v=vs.85%29.aspx
• http://support.microsoft.com/kb/226473

 

Current Functions:

• _SetProxy(string $Proxy) - Sets an HTTP or SOCKS server for embedded IE browsers in the current process.
• _SetUserAgent(string $Agent) - Sets the user/browser agent for embedded IE browsers in the current process.
• _ClearCookies() - Deletes all IE and system Flash cookies. Returns a Boolean value.
• _UseTOR(int $TORPort = 9150) - Sets TOR as your proxy. You must have TOR running. $TORPort is only necessary if your TOR isn't using default 9150. TOR can be downloaded here: https://www.torproject.org/download/download-easy.html

Planned Features:

• Proxy authentication via DLL rather than passing to URL.
• Function to set HTTP referrer.
• Support for other (Non-IE) embedded browsers.
• An ease-of-use function for setting TOR as the proxy server (This can currently be done manually). (Done)

Compatability: All known versions of Windows since 2000/XP (Possibly earlier versions as well). Note: The settings applied by this UDF do not appear to affect functions such as InetGet. For those functions you would want to use HttpSetProxy. This UDF is for browser controls (which HttpSetProxy does not affect). Also, the browser must be embedded into the application as with _IECreateEmbedded. It will not work with a new window created with _IECreate.

System Requirements:

• The existance of urlmon.dll on the system usually located in WindowsSystem32

Releases:

NetSession-0.9.au3 {OLD}

NetSession-0.9b.au3 {OLD}

NetSession-0.9c.au3 {OLD}

NetSession-0.9d.au3 {Newest}

    The changes in each version are detailed near the top of this post.

!!! WE NEED HELP FOR THE NEXT RELEASE !!!

Here's our current (not working) attempt to use a proxy's username & password without passing it in the URL. Without this capability we can't browse to HTTPS sites if proxy auth is required:

 Please help if you can 

Happy coding!

Edited March 25, 2015 by drego

[JohnOne]

Can get user agent working but not proxy.

For example this works...

HttpSetProxy(2, "108.165.33.12:3128") 

But this does not...

_SetProxy("108.165.33.12:3128")

HTTP proxy.

[drego]

Can get user agent working but not proxy.

For example this works...

HttpSetProxy(2, "108.165.33.12:3128") 

But this does not...

_SetProxy("108.165.33.12:3128")

HTTP proxy.

 

Not sure what's going on with your app but I just retested with an HTTP proxy and it worked just fine. First I set the proxy and then navigated to a page using a browser control. I'm not sure about using the other net functions but I assume it would work for them too, though browser controls are my main focus.

Edit: The answer to this is that the UDF only affects embedded browser controls which HttpSetProxy does not.

Edited January 20, 2015 by drego

[JohnOne]

I just used InetRead, I'll try with a browser control.

Fancy posting your code to save me farting about?

Edited January 20, 2015 by JohnOne

[drego]

I just used InetRead, I'll try with a browser control.

Fancy posting your code to save me farting about?

 

The file I'm using it with right now is huge but you've got the usage right. Just call it before navigating.

$ie = _IECreateEmbedded ()
$GUIActiveX = GUICtrlCreateObj($ie, 1, 25, 780, 500)
; set the proxy
_IENavigate($ie,"http:/www.ipchicken.com",1)

[drego]

Please let me know if it doesn't work with the inet functions and only browser instances so I can update the main post. Also, as far as HttpSetProxy goes I don't believe it works with browser controls. I remember that being the original point of coming up with this years ago and this being the only solution for IE controls.

Edited January 20, 2015 by drego

[JohnOne]

Works fine with browser control.

With InetRead it still does not though.

[drego]

Works fine with browser control.

With InetRead it still does not though.

 

That's fine because with inet you already have a function to set the proxy but not for browser controls. Thanks, I'll update the main post with this info which is that the UDF is geared toward browser objects since there's no other way to accomplish it.

[JohnOne]

Agreed, will this udf evolve to cater for third party (child) Processes, such as IE, FF, GC?

[drego]

Agreed, will this udf evolve to cater for third party (child) Processes, such as IE, FF, GC?

 

I'm not sure if it's even possible to use with _IECreate (I'll test), it might only work for embedded IE controls (_IECreateEmbedded). As for other browsers I would say that if they're embedded then yes I would take a shot at them.

[JohnOne]

It does not work for _IECreate, I tested earlier.

Great UDF though by the way gentlemen.

[drego]

Thanks. Now if I could just figure out how to authenticate a username & password without doing it in the URL so that accessing HTTPS sites when having to provide proxy authentication isn't a problem.

Edited January 20, 2015 by drego

[JohnOne]

I don't really know what that means, is it for some sort of registered or paid proxy service?

If so do you have any links?

[drego]

I don't really know what that means, is it for some sort of registered or paid proxy service?

If so do you have any links?

 

It could be any proxy server that requires username/password authentication. There are many paid services and in situations where someone is rotating through a list of proxies it's likely that they're going to be using a paid service that requires authentication. Here is one such service http://www.changemyip.com/ (I don't own it). This isn't going to be an issue for proxies on public proxy lists. I don't use public lists though because the proxies are always dying or are slow.

Maybe I'll setup a proxy for testing auth in this UDF and post it here if I can get some help from you and others figuring out how to do it with the DLL without passing auth in the URLs.

Edited January 20, 2015 by drego

[SmOke_N]

I don't have a proxy that needs user/pass.  You might try this and see if it works as you expect.

expandcollapse popup

#include <WinAPI.au3>

Func _proxySet($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "")

    Local Static $INTERNET_OPTION_PROXY = 38
    Local Static $INTERNET_OPTION_PROXY_USERNAME = 43
    Local Static $INTERNET_OPTION_PROXY_PASSWORD = 44
    Local Static $INTERNET_OPEN_TYPE_PROXY = 3

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    Local $tUsername, $tPassword, $iEnum, $iErr

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        $sUser = _WinAPI_WideCharToMultiByte($sUser)
        $tUsername = DllStructCreate("char username[" & StringLen($sUser) + 1 & "]")
        DllStructSetData($tUsername, "username", $sUser)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_USERNAME, $tUsername)
        $iErr = @error
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        $sPass = _WinAPI_WideCharToMultiByte($sPass)
        $tPassword = DllStructCreate("char password[" & StringLen($sPass) + 1 & "]")
        DllStructSetData($tPassword, "password", $sPass)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_PASSWORD, $tPassword)
        $iErr = @error
    EndIf

    Local $tProxy
    $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("char proxy[" & StringLen($sProxy) + 1 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
    $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("char proxybypass[" & StringLen($sProxyByPass) + 1 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer)
        $iErr = @error
    EndIf

    If Not $iEnum Then
        Return SetError($iErr, 0, 0)
    EndIf

    Return $iEnum
EndFunc

Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

Edit:

I'll just add.  I could not get this or the original to work for me with embedded IE or "InternetExplorer.Application" object (_IECreate()).  The embedded may be because I'm forcing my embedded browsers to use IWebBrowser2.  Something I'm not willing to change.

However, anything I used to test that uses the wininet.dll other than IE has produced the results I would have expected (eg. _GetIP()).

Edited January 20, 2015 by SmOke_N

[drego]

I don't have a proxy that needs user/pass.  You might try this and see if it works as you expect.

expandcollapse popup

#include <WinAPI.au3>

Func _proxySet($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "")

    Local Static $INTERNET_OPTION_PROXY = 38
    Local Static $INTERNET_OPTION_PROXY_USERNAME = 43
    Local Static $INTERNET_OPTION_PROXY_PASSWORD = 44
    Local Static $INTERNET_OPEN_TYPE_PROXY = 3

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    Local $tUsername, $tPassword, $iEnum, $iErr

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        $sUser = _WinAPI_WideCharToMultiByte($sUser)
        $tUsername = DllStructCreate("char username[" & StringLen($sUser) + 1 & "]")
        DllStructSetData($tUsername, "username", $sUser)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_USERNAME, $tUsername)
        $iErr = @error
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        $sPass = _WinAPI_WideCharToMultiByte($sPass)
        $tPassword = DllStructCreate("char password[" & StringLen($sPass) + 1 & "]")
        DllStructSetData($tPassword, "password", $sPass)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_PASSWORD, $tPassword)
        $iErr = @error
    EndIf

    Local $tProxy
    $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("char proxy[" & StringLen($sProxy) + 1 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
    $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("char proxybypass[" & StringLen($sProxyByPass) + 1 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer)
        $iErr = @error
    EndIf

    If Not $iEnum Then
        Return SetError($iErr, 0, 0)
    EndIf

    Return $iEnum
EndFunc

Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

Edit:

I'll just add.  I could not get this or the original to work for me with embedded IE or "InternetExplorer.Application" object (_IECreate()).  The embedded may be because I'm forcing my embedded browsers to use IWebBrowser2.  Something I'm not willing to change.

However, anything I used to test that uses the wininet.dll other than IE has produced the results I would have expected (eg. _GetIP()).

 

It only affects embedded IE. I've never forced IWebBrowser2 before; however, if you test the original UDF with the default AutoIt  _IECreateEmbedded browser it will work. It won't have any affect on any other net functions that I'm aware of (Though it could be possible). Have you tested the code you posted with a default embed? That would be the best starting point for testing any additions/modifications. Then we can worry about IWebBrowser2 (Which I never knew about until now).

[SmOke_N]

I was pretty clear on what I tested.

1.  It works.  But works with anything that uses the wininet.dll except IE

2.  It does not work for me with embedded IE (the original didn't either)

3.  It does not work with me with standard IE

I merely put the username/password for solution for you to test, as I stated, I do not have a proxy that requires username/password.

Edit:

I take that back, I now have (non-user/pass) working with the version I posted and embedded IE9 version.

Edited January 20, 2015 by SmOke_N

[drego]

I was pretty clear on what I tested.

1.  It works.  But works with anything that uses the wininet.dll except IE

2.  It does not work for me with embedded IE (the original didn't either)

3.  It does not work with me with standard IE

I merely put the username/password for solution for you to test, as I stated, I do not have a proxy that requires username/password.

Edit:

I take that back, I now have (non-user/pass) working with the version I posted and embedded IE9 version.

 

I just tested your code as: _proxySet('user22','pass2','105.190.102.248:64043')

Then tried navigating to a page and was unable to get a connection out. IE threw: Navigation to the webpage was canceled.

Edit: In your 1 2 3 list you were talking about your code I presume? Mine works for embedded IE. There's just that issue of having to pass the login through the URL.

These are great additions though. Like I said I'll see about providing a proper proxy to test auth.

Edited January 20, 2015 by drego

[drego]

I was pretty clear on what I tested.

1.  It works.  But works with anything that uses the wininet.dll except IE

2.  It does not work for me with embedded IE (the original didn't either)

3.  It does not work with me with standard IE

I merely put the username/password for solution for you to test, as I stated, I do not have a proxy that requires username/password.

Edit:

I take that back, I now have (non-user/pass) working with the version I posted and embedded IE9 version.

 

I just tested your code as: _proxySet('user22','pass2','105.190.102.248:64043')

Then tried navigating to a page and was unable to get a connection out. IE threw: Navigation to the webpage was canceled.

Edit: In your 1 2 3 list you were talking about your code I presume? Mine works for embedded IE. There's just that issue of having to pass the login through the URL.

These are great additions though. Like I said I'll see about providing a proper proxy to test auth.

 

I just messaged you a test proxy that takes authentication.

Edit: Looks like sometime tomorrow we could have a new version with better authentication that can manipulate more than just embedded browser objects (Though that isn't nearly as important as better auth). There are also some other features of the Windows API and urlmon.dll that are very interesting that haven't been explored with AutoIt. Good things to come

Edited January 20, 2015 by drego

[drego]

New version 0.9b has just been released. It now includes the _ClearCookies function which clears all of your browser's cookies as well as all Flash cookies.