Powershell Invoke-Webrequest in AutoIt ?

[agivx3]

Is there any similar command in AutoIt or has anyone scripted a function or UDF which does the same job as the powershell invoke-webrequest command ?

[water]

Quote

The Invoke-WebRequest cmdlet sends HTTP and HTTPS requests to a web page or web service. It parses the response and returns collections of links, images, and other significant HTML elements.

AutoIt provides a lot of UDFs. See this section of the UDF-list in the wiki.

[agivx3]

Hi Water,

yeah i know about the UDF´s and in the meantime i nearly have all of them... I am using AutoIt now for many years and created many scripts.
But now i tried to convert a small powershell script and have no idea how to do this. This also may be due to the fact that my knowledge about WinHTTP.au3 is less than basic 😞
The powershell script works fine but i need it in AutoIT script because it will be part of a bigger project. Here is the example i am trying to convert. Maybe you have some ideas about this ?

 

$request="2024-04 21H2 Server"
$outdir = $request.Replace(" ","_")
if (!(Test-Path $outdir)) {
    New-Item -ItemType Directory -Force -Path $outdir | Out-Null
}

$kbObj = Invoke-WebRequest -Uri "https://www.catalog.update.microsoft.com/Search.aspx?q=$request"
$Available_KBIDs = $kbObj.InputFields|Where-Object { $_.type -eq 'Button' -and $_.Value -eq 'Download' } |Select-Object -ExpandProperty ID
$kbObj.Links |Where-Object ID -match '_link' |where-object innertext -NotLike "*4.7*" |select innertext,id
$kbGUIDs = $kbObj.Links |
    Where-Object ID -match '_link' | # where-object innertext -NotLike "*4.7*" |
    ForEach-Object { $_.id.replace('_link', '') } |
    Where-Object { $_ -in $Available_KBIDs }

foreach ($kbGUID in $kbGUIDs) {
    $Post = @{ size = 0; updateID = $kbGUID; uidInfo = $kbGUID } | ConvertTo-Json -Compress
    $PostBody = @{ updateIDs = "[$Post]"
}
Invoke-WebRequest -Uri 'https://www.catalog.update.microsoft.com/DownloadDialog.aspx' -Method Post -Body $postBody|
    Select-Object -ExpandProperty Content |
    Select-String -AllMatches -Pattern "(http?)(:\/\/)([^\s,]+)(?=')" |
    Select-Object -Unique |
    ForEach-Object {
        ForEach ($line in $_.matches.value) {
            switch ($line) { 
                {$_ -match '-x64' } {                                            # 64-bit
                    $line
                    wget\wget.exe $line --no-clobber --directory-prefix=$outdir 
                }
            }
            
        }
    }
}

 

[SOLVE-SMART]

3 hours ago, agivx3 said:

The powershell script works fine but i need it in AutoIT script because it will be part of a bigger project.

Hi @agivx3, does the "bigger project" don't allow to use Powershell? Or could it be possible to stick with Powershell but call the script by AutoIt, get the PS console output and proceed in AutoIt? I mean, why do you want to convert the whole stuff when it's working in PS by default cmdlets?

Best regards
Sven

[agivx3]

I think running the powershell from autoit would work and be much easier. Your'e sure right about this...

But there are a few arguments which speak against taking the ready script. At first the challenge to develop something new. Thats the fun !! And i like challenges. Taking something thats already finished cuts of the whole fun of programming 🙂 And at second i also dont really like powershell scripts. 

[SOLVE-SMART]

On 5/9/2024 at 10:10 AM, agivx3 said:

At first the challenge to develop something new. Thats the fun !! And i like challenges.

Understood, totally agree with that 😅 .

On 5/9/2024 at 10:10 AM, agivx3 said:

And at second i also dont really like powershell scripts. 

Me neither, but only thought it is urgent for you. That's why I suggested to stick with PS.

On 5/8/2024 at 6:37 PM, agivx3 said:

But now i tried to convert a small powershell script and have no idea how to do this. This also may be due to the fact that my knowledge about WinHTTP.au3 is less than basic 😞

I guess the AutoIt script will become a bit longer then the PS, but where are you struggling exactly? First you do you http request (with WinHTTP.au3 like @water already suggested), then you do the parsing stuff like in the PS code. Let's give it a try. I strongly believe we can help you through the steps, but please try first 🤝 .

Best regards
Sven

Edited May 10 by SOLVE-SMART

[SOLVE-SMART]

Here a little starting approach without WinHTTP.au3 for your list of IDs and Names for the server updates:

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-d -w 1 -w 2 -w 4 -w 5 -w 6 -w 7
#AutoIt3Wrapper_AU3Check_Stop_OnWarning=y

#include-once
#include <Array.au3>

_Main()

Func _Main()
    Local Const $sTargetMaschine     = '2024-04 21H2 Server'
    Local Const $sUrlMsUpdateCatalog = 'https://www.catalog.update.microsoft.com/Search.aspx?q=' & $sTargetMaschine
    Local Const $sFile               = 'result.html'

    ; In your case, you can simply get the information by InetGet() instead of using WinHTTP.au3.
    ; When it comes to auth., I recommend WinHTTP.au3.
    InetGet($sUrlMsUpdateCatalog, $sFile)

    Local Const $sContent = _GetFileContent($sFile)
    Local Const $aList    = _GetListOfIdsAndNames($sContent)

    _ArrayDisplay($aList)
EndFunc

Func _GetListOfIdsAndNames($sContent)
    Local Const $sRegExPattern = '(?s)onclick=''goToDetails\("(.+?)".+?>(.+?)</a>'
    Local $aListOfIdsAndNames  = StringRegExp($sContent, $sRegExPattern, 3)

    Local Const $iLeadingTrainlingDoubleFlag = 7

    For $i = 0 To _Length($aListOfIdsAndNames)
        $aListOfIdsAndNames[$i] = StringStripWS($aListOfIdsAndNames[$i], $iLeadingTrainlingDoubleFlag)
    Next

    Return $aListOfIdsAndNames
EndFunc

Func _GetFileContent($sFile)
    Local Const $iUtf8WithoutBomMode = 256

    Local $hFile        = FileOpen($sFile, $iUtf8WithoutBomMode)
    Local $sFileContent = FileRead($hFile)
    FileClose($hFile)

    Return $sFileContent
EndFunc

Func _Length($aList)
    Return UBound($aList) - 1
EndFunc

For the second request, the POST request, you would probably need another approach. But it's only a start - there are several ways to do it.
I personally would use WebDriver (au3WebDriver), but this could be way over the top (regarding the setup) for you needs.

Best regards
Sven

Edited May 10 by SOLVE-SMART

[agivx3]

Hi Sven,

Great work ! I wouldnt have found the RegEx pattern myself but i understand how it works. So for my understanding - the first web request is nothing else than connecting to the website and then parsing the result ? With your script i now have understood how the first powershell commands parse the returned text. So far so good.

I agree with you that the second part is the more difficult part. I see that the string $Post is build somehow with the information we already have or can extract. What i still dont unterstand is what is happing in the "convertTo-Json -Compress" and if there is a possibility in AutoIt for this. The second problem is the web request with the method "Post". I assume the inetget is not working for this. I think the winhttp.au3 can do this somehow but dont know howto.

Analyzing the final download string i can see that we have only parts of the information needed to build it or get its information from the website. The bold text is missing or built somehow.

Example:
https://catalog.s.download.windowsupdate.com/c/msdownload/update/software/secu/2024/04/windows10.0-kb5036909-x64_786040b0b0d000b17d6a727ea93ff77d733d1044.msu

Best regards
Andi
 

[agivx3]

Just have seen that with the json.udf compression of strings to json could be done

; Public Functions:
;   Json_StringEncode($String, $Option = 0)
;   Json_StringDecode($String)
;   Json_IsObject(ByRef $Object)
;   Json_IsNull(ByRef $Null)
;   Json_Encode($Data, $Option = 0, $Indent = Default, $ArraySep = Default, $ObjectSep = Default, $ColonSep = Default)
;   Json_Decode($Json, $InitTokenCount = 1000)
;   Json_ObjCreate()
;   Json_ObjPut(ByRef $Object, $Key, $Value)
;   Json_ObjGet(ByRef $Object, $Key)
;   Json_ObjDelete(ByRef $Object, $Key)
;   Json_ObjExists(ByRef $Object, $Key)
;   Json_ObjGetCount(ByRef $Object)
;   Json_ObjGetKeys(ByRef $Object)
;   Json_ObjGetItems(ByRef $Object)
;   Json_ObjClear(ByRef $Object)
;   Json_Put(ByRef $Var, $Notation, $Data, $CheckExists = False)
;   Json_Get(ByRef $Var, $Notation)
;   Json_Dump($String)
 

[SOLVE-SMART]

Hi Andi ( @agivx3 ),

after a research I realized that there is no API request (no POST) available for https://www.catalog.update.microsoft.com/DownloadDialog.aspx anymore.
If this is the case, then you cannot download the found windows update *.msu packages. Are you sure the powershell script loads the 4 found entries? Not for me and when I debug the PS script, I see it cannot work (at least not in that way which was provided by you in this post).

What do you want exactly? Please describe the bigger goal behind all of this? Is it to download the newest cumulative updates for a specific maschine and that's it?

If yes, then your possibilities to achieve this are less then thought before. You can try UIA or WebDriver to get this automation done. But please, write some code and try something to show your participation. I won't write it all for you 🧐 .

Best regards
Sven

Question: Do you speak german by any chance?
Update: This looks promising ==> https://github.com/ryan-jan/MSCatalog 😁 ==> but it's still Powershell

Edited May 10 by SOLVE-SMART

[agivx3]

Ok, i didn't know this about the API. Thanks for the information. The powershell may have been shortened by a colleague  who may have deleted some neccessary parts. I will investigate this. Maybe then it is better trying the webdriver. I will be back here when i have my first webdriver code done or when i fail with this task 🙂

 

[SOLVE-SMART]

Hi folks 👋 ,

as short note: I was in conversation with @agivx3 to get a better understanding about his requirements. As summary I can say that the goal was to get cumulative updates (windows server etc.) from the microsoft catalog site.

There were two steps to proceed.
The 🥇 first one is getting the IDs (GUIDs) from the page source (search result entries).

👉 Code (feasable due to simply InetGet() usage):

Spoiler

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-d -w 1 -w 2 -w 4 -w 5 -w 6 -w 7
#AutoIt3Wrapper_AU3Check_Stop_OnWarning=y

#include-once
#include <Array.au3>

_Main()

Func _Main()
    Local Const $sTargetMaschine     = '2024-04 21H2 Server'
    Local Const $sUrlMsUpdateCatalog = 'https://www.catalog.update.microsoft.com/Search.aspx?q=' & $sTargetMaschine
    Local Const $sFile               = 'step-one-result.html'

    ; In your case, you can simply get the information by InetGet() instead of using WinHTTP.au3.
    ; When it comes to auth., I recommend WinHTTP.au3.
    InetGet($sUrlMsUpdateCatalog, $sFile)

    Local Const $sContent = _GetFileContent($sFile)
    Local Const $aList    = _GetListOfIdsAndNames($sContent)

    _ArrayDisplay($aList, 'List of IDs (GUIDs) and Names')
EndFunc

Func _GetListOfIdsAndNames($sContent)
    Local Const $sRegExPattern      = '(?s)onclick=''goToDetails\("(.+?)".+?>(.+?)</a>'
    Local Const $sGlobalMatchesFlag = 3
    Local $aListOfIdsAndNames       = StringRegExp($sContent, $sRegExPattern, $sGlobalMatchesFlag)

    Local Const $iLeadingTrailingDoubleFlag = 7

    For $i = 0 To _Length($aListOfIdsAndNames)
        $aListOfIdsAndNames[$i] = StringStripWS($aListOfIdsAndNames[$i], $iLeadingTrailingDoubleFlag)
    Next

    Return $aListOfIdsAndNames
EndFunc

Func _GetFileContent($sFile)
    Local Const $iUtf8WithoutBomMode = 256

    Local $hFile        = FileOpen($sFile, $iUtf8WithoutBomMode)
    Local $sFileContent = FileRead($hFile)
    FileClose($hFile)

    Return $sFileContent
EndFunc

Func _Length($aList)
    Return UBound($aList) - 1
EndFunc

👉 Result:

 

The 🥈 second step was to transfer a PowerShell script to AutoIt.

👉 The short version of the PowerShell script:

Spoiler

# Target ID (GUID)
$guid = "1c09aabd-5134-4b57-9e50-454b6dd63ad0"

# Json body
$post = @{size = 0; updateID = $guid; uidInfo = $guid} | ConvertTo-Json -Compress
$body = @{updateIDs = "[$post]"}

# POST request
$params = @{
    Uri = "https://www.catalog.update.microsoft.com/DownloadDialog.aspx"
    Method = "POST"
    Body = $body
    ContentType = "application/x-www-form-urlencoded"
}
$downloadData = Invoke-WebRequest @params

# Write the response data to file (for debugging purposes).
# This isn't necessary, but to understand to RegEx below it's helpful.
$downloadData.Content > "result-powershell.html"

# Get only the download URL(s) for the cumulative update (based by the GUID on top).
$regExPattern = '(?s)\.files?\[\d+\]\.url(\s+)?=(\s+)?''(.+?.msu)'
$downloadData.Content |
    Select-String -Pattern $regExPattern -AllMatches |
    %{ $_.matches.groups[3].value }

 

👉 The AutoIt counterpart of the PowerShell script above:

Spoiler

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-d -w 1 -w 2 -w 4 -w 5 -w 6 -w 7
#AutoIt3Wrapper_AU3Check_Stop_OnWarning=y

_Main()

Func _Main()
    ; Target ID (GUID)
    Local Const $sGuid = '1c09aabd-5134-4b57-9e50-454b6dd63ad0'

    ; Json body
    Local Const $sJson = 'updateIDs = [{ "uidInfo": "' & $sGuid & '", "updateID": "' & $sGuid & '", "size": 0 }]'

    ; POST request
    Local Const $sResponse = _POST('https://www.catalog.update.microsoft.com/DownloadDialog.aspx', $sJson)

    ; Write the response data to file (for debugging purposes).
    ; This isn't necessary, but to understand to RegEx below it's helpful.
    _WriteFile('result-autoit.html', $sResponse)

    ; Get only the download URL(s) for the cumulative update (based by the GUID on top).
    Local Const $sRegExPattern = '(?s)\.files?\[\d+\]\.url(\s+)?=(\s+)?''(.+?.msu)'
    Local Const $aMatches = StringRegExp($sResponse, $sRegExPattern, 3)
    ConsoleWrite($aMatches[_Length($aMatches)] & @CRLF)
EndFunc

Func _POST($sURL, $sJson)
    Local $oHTTP = ObjCreate('winhttp.winhttprequest.5.1')

    $oHTTP.Open('POST', $sURL, False)
    $oHTTP.SetRequestHeader('Content-Type', 'application/x-www-form-urlencoded')
    $oHTTP.Send(_UrlEncode($sJson))

    Local Const $iStatusCode = $oHTTP.Status

    If $iStatusCode <> 200 Then
        Local Const $iErrorIcon = 16
        Local Const $iTimeout   = 10

        MsgBox($iErrorIcon, 'Error', $iStatusCode, $iTimeout)
        Return
    EndIf

    Return $oHTTP.ResponseText
EndFunc

Func _UrlEncode($sText)
    Local Const $iUtf8Flag          = 4
    Local Const $iCaseSensitiveFlag = 1
    Local Const $sCharacters        = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!$&/()=?+*',;:.-_@"

    Local Const $sUtf8Binary = StringToBinary(StringReplace($sText, ' ', ''), $iUtf8Flag)
    Local Const $sUtf8String = StringReplace($sUtf8Binary, '0x', '', $iCaseSensitiveFlag)

    Local $sChar, $sEncodedText

    For $i = 1 To StringLen($sUtf8String) Step 2
        $sChar = StringMid($sUtf8String, $i, 2)

        If StringInStr($sCharacters, BinaryToString('0x' & $sChar, $iUtf8Flag)) Then
            $sEncodedText &= BinaryToString('0x' & $sChar)
        Else
            $sEncodedText &= '%' & $sChar
        EndIf
    Next

    Return $sEncodedText
EndFunc

Func _WriteFile($sFile, $sText)
    Local Const $iUtf8WithoutBomAndOverwriteCreationMode = 256 + 2 + 8

    Local $hFile = FileOpen($sFile, $iUtf8WithoutBomAndOverwriteCreationMode)
    FileWrite($hFile, $sText)
    FileClose($hFile)
EndFunc

Func _Length($aList)
    Return UBound($aList) - 1
EndFunc

There were three tricky things (at least in my opinion).

• the correct json structure which is expected by the DownloadDialog.aspx site
• the json encoding ("_UrlEncode")
• and the request header "Content-Type" => "application/x-www-form-urlencoded" instead of "application/json"

🔮 The result string for the PowerShell and for the AutoIt-Script is:

https://catalog.s.download.windowsupdate.com/c/msdownload/update/software/secu/2024/04/windows10.0-kb5036909-x64_786040b0b0d000b17d6a727ea93ff77d733d1044.msu

💡 Why do I share this at all, in such kind of detail? Because I learned something and I thought it's worthy enough to share the story 😇 😅 .

Best regards
Sven

Edited May 13 by SOLVE-SMART