DLLCall fails in AutoIt but works in VB6

[TimRude]

8 hours ago, MattyD said:

I'm a bit curious of how returning dllcalls "As String" is supposed to work on the VB side.

I'm curious about that too. I've looked but haven't found a description of how VB does it. I do know that I don't have to manually preallocate a buffer of a specific size, or tell VB how many bytes to read. It just does it.

[TimRude]

19 hours ago, MattyD said:

Does it blindly read from the string pointer until a null character is reached, or is there some other trickery at play where it allocates a buffer before the call?

I can confirm that it isn't a null character that tells VB that it's the end of the string. Some of the encrypted files are data rather than just plain text. Still ANSI, but containing non-printing ASCII characters, including plenty of Chr(0) characters.

So I don't know what kind of magic VB does when it calls the DLL to get the entire string without knowing in advance how big it's gonna be. I know the function does return the number of bytes in the string as the second parameter in the function call, but I don't do anything with that. I just assign my string variable the return result of the function and that's it. It's automagically the right size and contains every byte of the decrypted data, no more and no less.

In looking for information about how VB6 can receive a string as a return value from a DLL function, I found (via the WayBack Machine)  a paper written by Microsoft in 1996 detailing how to write a DLL that could work with VB5 (which still applies to VB6). Here's the link, jumped to the relevant portion:

https://web.archive.org/web/20051215232620/http://vb.mvps.org/tips/vb5dll.asp#:~:text=5%3A Passing and Returning Strings

From my limited understanding of what this says, it looks like the DLL (which was written to work with another flavor of BASIC other than VB but apparently works the same way since it works with VB) returns the string as a BSTR created using SysAllocStringByteLen. This (according to https://learn.microsoft.com/en-us/previous-versions/windows/desktop/automat/bstr ) is a composite data type that consists of a length prefix, a data string, and a terminator.

Beyond that, I'm out of my depth to figure out how to get AutoIt to pass in a BSTR as the first parameter (the name of the file to be decrypted), and then to assign the BSTR returned by the function to an AutoIt variable. But hopefully this bit of info may tell someone who knows a whole lot more than I do what I need to do.

[MattyD]

OK great - there's quite a bit there to digest

So I'm suggesting this with little confidence - but its something different to try I guess.

If the DLL is expecting a string length before the string then perhaps something like this may help... 

edit: length should be in bytes - not number of chars!

#AutoIt3Wrapper_UseX64=N

Local $sFilename = "C:\Test\MyTestFile.xyz"

Local $hDLL = DllOpen("XYZDECRYPT.DLL")

Local $iFilenameLen = StringLen($sFilename)
Local $tBstrFilename = DllStructCreate(StringFormat("long Len;wchar Filename[%d]", $iFilenameLen + 1)) ; Len +1 to include a null terminator

$tBstrFilename.Len = $iFilenameLen * 2
$tBstrFilename.Filename = $sFilename

$aCall = DllCall($hDLL, "ptr", "DECRYPTXYZFILE", "struct*", $tBstrFilename, "long*", 0)

Edited March 2 by MattyD

[Werty]

You're saying it works with VB6, then make a VB6 EXE or DLL that you can channel it through 😛

Your AU3 script sending the request to a VB6 exe/dll, which then calls the xyzdecrypt.dll and then send the result back to your AU3 script.

Just as a last resort ofcourse. however it's done in VB6, dunno VB6.

 

Edited March 2 by Werty

[MattyD]

Ok - May have jumped the gun before, the previous version will likely fail.

When using SysAllocString to create the bstr,  the returned pointer takes us to the character array. The "length" field is in memory before this...

Local $sFilename = "C:\Test\MyTestFile.xyz"
$aCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "wstr", $sFilename) ;Create Bstring
Local $iFilenameLen = StringLen($sFilename)

Local $tBstrFilename = DllStructCreate(StringFormat("long Len;wchar Filename[%d]", $iFilenameLen + 1), $aCall[0] - 4) ;$aCall[0] is a ptr to the wchar array!

ConsoleWrite("Char count = "  & $iFilenameLen & @CRLF)
ConsoleWrite("Byte count = "  & $tBstrFilename.Len & @CRLF) ;should be double $iFilenameLen, as its a wstr
ConsoleWrite($tBstrFilename.Filename & @CRLF)

So to correctly pass a bstr, I think its something like this for unicode...

;Unicode version

Local $sFilename = "C:\Test\MyTestFile.xyz"

$aBstrCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "wstr", $sFilename)
$aCall = DllCall("XYZDECRYPT.DLL", "ptr", "DECRYPTXYZFILE", "ptr", $aBstrCall[0], "long*", 0)

Or this for ANSI...

;ANSI Vesrion

Local $sFilename = "C:\Test\MyTestFile.xyz"

$aBstrCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "str", $sFilename)
$aCall = DllCall("XYZDECRYPT.DLL", "ptr", "DECRYPTXYZFILE", "ptr", $aBstrCall[0], "long*", 0)

 

Edited March 2 by MattyD
Added ANSI Code

[Nine]

Good job again @MattyD.  Not taking any merits here, but for completeness :

Local $hDLL = DllOpen("XYZDECRYPT.DLL")

Local $sFilename = "C:\Test\MyTestFile.xyz"
Local $aCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "wstr", $sFilename)
Local $aCall2 = DllCall($hDLL, "ptr", "DECRYPTXYZFILE", "ptr", $aCall[0], "long*", 0)

Local $tBstr = DllStructCreate(StringFormat("long len;wchar str[%d]", $aCall2[2] + 1), $aCall2[0] - 4)
ConsoleWrite("Char count received = "  & $aCall2[2] & @CRLF)
ConsoleWrite("Byte count received = "  & $tBstr.len & @CRLF)
ConsoleWrite("String received = " & $tBstr.str & @CRLF)

DllCall("OleAut32.dll", "none", "SysFreeString", "ptr", $aCall[0])
DllClose($hDLL)

ps. if there is memory leak, you should also consider freeing $aCall2[0]

Edited March 2 by Nine
small bug

[MattyD]

Thanks for the compliment Nine - but I could still be completely wrong :P

[Nine]

Based on what I have read, I think you are right...but I could be wrong too 

[TimRude]

You guys were so close!

I tried @MattyD's code as completed by @Nine and the first run through I got the same old error message.

But then I adjusted the line that creates the BSTR being passed in (the filename) by changing wstr to str.

With that change, the call to the decryption DLL succeeded and I actually got back 12,677 bytes -- the exact number of bytes of the decrypted file contents.

However, that string was jibberish instead of being the actual text. My guess was that the DLL was returning an ANSI string rather than Unicode, so I changed the $tBstr definition to use char rather than wchar.

And it worked! I got my entire decrypted file contents just as pretty as you please.

So here's the modified version that worked:

Local $hDLL = DllOpen("XYZDECRYPT.DLL")

Local $sFilename = "C:\Test\MyTestFile.xyz"
Local $aCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "str", $sFilename)
Local $aCall2 = DllCall($hDLL, "ptr", "DECRYPTXYZFILE", "ptr", $aCall[0], "long*", 0)

Local $tBstr = DllStructCreate(StringFormat("long len;char str[%d]", $aCall2[2] + 1), $aCall2[0] - 4)
ConsoleWrite("Char count received = "  & $aCall2[2] & @CRLF)
ConsoleWrite("Byte count received = "  & $tBstr.len & @CRLF)
ConsoleWrite("String received = " & $tBstr.str & @CRLF)

DllCall("OleAut32.dll", "none", "SysFreeString", "ptr", $aCall[0])
DllCall("OleAut32.dll", "none", "SysFreeString", "ptr", $aCall2[0])
DllClose($hDLL)

Thanks for the help! And quite impressive work seeing as how you didn't have access to the actual DLL for testing.

Edited March 2 by TimRude

[Nine]

@TimRude  Please give solution to @MattyD.  He has all the credits...

[TimRude]

I did.

[UEZ]

Interesting that the input string must be in that format.

#include <Array.au3>
#include <WinAPIDiag.au3>

Local $sFilename = "c:\Test\MyTestFile.xyž"
Local $aCall = DllCall("OleAut32.dll", "ptr", "SysAllocString", "wstr", $sFilename)
$t = DllStructCreate("wchar p[" & StringLen($sFilename) & "]", $aCall[0])
_WinAPI_DisplayStruct($t)

DllCall("OleAut32.dll", "none", "SysFreeString", "ptr", $aCall[0])

ž = 0000 0001 0111 1110 -> 1 126

Quote

I know. But since it is a decryption DLL and part of a licensed third-party application, I can't share it. Thanks for your suggestions though.

Wondering why there is no API documentation...

[junkew]

To make it complete. The BSTR has a definition: https://learn.microsoft.com/en-us/previous-versions/windows/desktop/automat/bstr?redirectedfrom=MSDN

And some nice topic on many variable types was made by @LarsJ