DLLCall fails in AutoIt but works in VB6

[TimRude]

I have a 3rd party DLL that I use to decrypt certain files that works fine using Visual Basic 6, but I can't figure out how to use it with AutoIt.

In VB6, the DLL function definition is this:

Private Declare Function DECRYPTXYZFILE Lib "XYZDECRYPT.DLL" (ByVal FullFileName As String, dwOutSize As Long) As String

When I want to call the function to actually decrypt a file, I first have to change the working directory to the folder where the XYZDECRYPT.DLL file lives (S:\XYZ) before attempting to call it. It relies on other dependencies within that folder and can't find them (and therefore won't work) unless I do this.

So a sample call to decrypt a file named C:\Test\MyTestFile.xyz in VB6 would be:

Dim FullFileName As String
Dim dwOutSize As Long
Dim FileData As String

FullFileName = "C:\Test\MyTestFile.xyz"

' Change working drive\directory to location of decryption dll
ChDrive "S:"
ChDir "S:\XYZ"

' Call the decryption dll to decrypt the file
FileData = DECRYPTXYZFILE(FullFileName, dwOutSize)

' At this point, FileData contains the decrypted file contents
' and dwOutSize contains the length of the decrypted string

Debug.Print "dwOutSize = " & dwOutSize
Debug.Print "FileData = " & FileData

This works just fine in VB6. Here's how I attempted to the same thing in AutoIt, which doesn't work:

Local $sFullFileName = "C:\Test\MyTestFile.xyz"
Local $dwOutSize, $sFileData

FileChangeDir("S:\XYZ")
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "str", "DECRYPTXYZFILE", "str", $sFullFileName, "LONG*", "$dwOutSize")
ConsoleWrite("DLLCall Completed" & @CRLF)
$sFileData = $aDllCallReturn[0]
$dwOutSize = $aDllCallReturn[2]
ConsoleWrite("$dwOutSize = " & $dwOutSize & @CRLF & "DllCall return value: " & $sFileData & @CRLF)

It crashes in AutoIt with "Exit code: 3221225477    Time: 3.002" after the DLLCall, before it does the ConsoleWrite("DLLCall Completed").

What am I doing wrong here?

Edited February 27 by TimRude

[Nine]

There is one obvious error.  "LONG*" should be followed by 0, not the string "$dwOutSize".

Make sure you can open the dll correctly by using :

Local $sFullFileName = "C:\Test\MyTestFile.xyz"
Local $dwOutSize, $sFileData

FileChangeDir("S:\XYZ")
Local $hDLL = DllOpen("XYZDECRYPT.DLL")
ConsoleWrite("open with " & $hDLL & @CRLF)
Local $aDllCallReturn = DllCall($hDLL, "str", "DECRYPTXYZFILE", "str", $sFullFileName, "LONG*", 0)
ConsoleWrite("DLLCall Completed with " & @error & @CRLF)
$sFileData = $aDllCallReturn[0]
$dwOutSize = $aDllCallReturn[2]
ConsoleWrite("$dwOutSize = " & $dwOutSize & @CRLF & "DllCall return value: " & $sFileData & @CRLF)
DllClose($hDLL)

 

Edited February 27 by Nine

[TimRude]

It opened the DLL correctly with $hDLL being assigned a value of 1.

But still crashed with "Exit code: 3221225477" on the DLLCall before getting to 'ConsoleWrite("DLLCall Completed ...'

I also tried this using ptr instead of str but it still crashed with the same "Exit code: 3221225477":

Local $sFullFileName = "C:\Test\MyTestFile.xyz"
Local $dwOutSize, $sFileData

FileChangeDir("S:\XYZ")
Local $hDLL = DllOpen("XYZDECRYPT.DLL")
ConsoleWrite("open with " & $hDLL & @CRLF)
Local $aDllCallReturn = DllCall($hDLL, "ptr", "DECRYPTXYZFILE", "str", $sFullFileName, "LONG*", 0)
ConsoleWrite("DLLCall Completed with " & @error & @CRLF)
$dwOutSize = $aDllCallReturn[2]
$sFileData = DllStructGetData(DllStructCreate("char[$iOutSize]", DllStructGetData(DllStructCreate("ptr", $aDllCallReturn[0]), 1)), 1)
ConsoleWrite("$dwOutSize = " & $dwOutSize & @CRLF & "DllCall return value: " & $sFileData & @CRLF)
DllClose($hDLL)

 

Edited February 27 by TimRude

[Nine]

You might need to use cdecl.

 

[TimRude]

Thanks for the suggestion. I just tried both of these, but neither worked. Still the same "Exit code: 3221225477" crash.

Local $aDllCallReturn = DllCall($hDLL, "str:cdecl", "DECRYPTXYZFILE", "str", $sFullFileName, "LONG*", 0)

Local $aDllCallReturn = DllCall($hDLL, "ptr:cdecl", "DECRYPTXYZFILE", "str", $sFullFileName, "LONG*", 0)

 

[RTFC]

47 minutes ago, Nine said:

You might need to use cdecl.

Bad suggestion. VB6 uses stdcall by default, and the OP shows that cdecl alternative was not used.

[Nine]

38 minutes ago, RTFC said:

VB6 uses stdcall by default, and the OP shows that cdecl alternative was not used.

I know.  But I also know that informations provided to us are not always exact and precise...

[UEZ]

dwOutSize As Long

should be "long", $dwOutSize

otherwise, if this is a pointer to a struct then try

$dwOutSize = DllStructCreate("long dwOutSize")
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "str", "DECRYPTXYZFILE", "str", $sFullFileName, "struct*", $dwOutSize)

 

Edited February 27 by UEZ

[TimRude]

My posted VB6 code that works is exactly and precisely what works. The only change to it is the actual dll file name and function name. Those have been anonymized since I can't reveal the actual names for security reasons. But in my tests I use the real names.

When I call it from VB6, the dll function call passes in two variables. The first is a string that tells the function what file to decrypt. The second is a Long variable that the function returns the length of the decrypted file contents in. The function itself returns the decrypted file contents as a string.

So I don't think dwOutSize is a pointer to any structure. It just gets returned as a Long value.

In VB6, I don't have to use the dwOutSize variable for anything since once I have the string I can query its length if I need that. VB apparently takes care of whatever magic is needed to get the function's return value into the VB string.

I just don't know how to get AutoIt to do the same. It isn't (so far) a matter of getting the data from the function call, it's a problem even doing the DLLCall without crashing AutoIt.

[UEZ]

Quote

' and dwOutSize contains the length of the decrypted string

When I read the VB code, I understand that the length of the string will be written to dwOutSize variable. Passing $dwOutSize to the dll call make no sense because you pass a value, not the pointer to the variable.

Can you test?

$tLength = DllStructCreate("long dwOutSize")
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "str", "DECRYPTXYZFILE", "str", $sFullFileName, "struct*", $tLength)
ConsoleWrite($tLength.dwOutSize & @CRLF)

 

[Nine]

From some VB documentation  :

Quote

Holds sequences of unsigned 16-bit (2-byte) code points that range in value from 0 through 65535. Each code point, or character code, represents a single Unicode character. A string can contain from 0 to approximately two billion (2 ^ 31) Unicode characters.

Seems is should be wstr

[TimRude]

I just tried:

$tLength = DllStructCreate("long dwOutSize")
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "str", "DECRYPTXYZFILE", "str", $sFullFileName, "struct*", $tLength)
ConsoleWrite($tLength.dwOutSize & @CRLF)

Same "Exit code: 3221225477"

Then I tried with wstr like this:

$tLength = DllStructCreate("long dwOutSize")
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "wstr", "DECRYPTXYZFILE", "str", $sFullFileName, "struct*", $tLength)
ConsoleWrite($tLength.dwOutSize & @CRLF)

Again, same "Exit code: 3221225477"

I even tried tacking a Chr(0) to the end of the $sFullFileName string to see if that would help. It didn't make any difference.

[UEZ]

Error 3221225477 is hex 0xC000005 which is an access violation.

Without the possibility to test it with the DLL, it will be difficult to find the problem.

 

[Nine]

2 hours ago, TimRude said:

Then I tried with wstr like this

It should be :

Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "wstr", "DECRYPTXYZFILE", "wstr", $sFullFileName, "long*", 0)

 

[TimRude]

4 hours ago, UEZ said:

Without the possibility to test it with the DLL, it will be difficult to find the problem.

I know. But since it is a decryption DLL and part of a licensed third-party application, I can't share it. Thanks for your suggestions though.

 

2 hours ago, Nine said:

It should be :

Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "wstr", "DECRYPTXYZFILE", "wstr", $sFullFileName, "long*", 0)

Just tried it with that change. Still the same issue.

FWIW: The decrypted content of the file I'm testing with is only 12,667 bytes of plain ansi text so I don't believe I'm running into an out of memory error or some kind of overflow because of too much data in the returned string.

[TimRude]

As a test, I tried leaving the second parameter off the DLLCall and it no longer crashes but I get error 4 (incorrect number of parameters). So I know AutoIt is talking to the DLL to get back that error, but of course I get no valid data otherwise since the function doesn't ... uh ... function.

[Nine]

Alright my last attempt :

Local $sFullFileName = "C:\Test\MyTestFile.xyz"
Local $tFile = DllStructCreate("wchar string[256]")
$tFile.string = $sFullFileName
Local $aDllCallReturn = DllCall("XYZDECRYPT.DLL", "wstr", "DECRYPTXYZFILE", "ptr", DllStructGetPtr($tFile), "long*", 0)

return type could then be ptr...

Good luck.

[TimRude]

@Nine and @UEZ I appreciate the effort of both of you.

That didn't work, but I'll keep trying various combinations of things. If I can't get it to work, I may just end up having to write my app in VB6 since I can access the DLL function easily from it.

[argumentum]

1 hour ago, TimRude said:

If I can't get it to work, I may just end up having to write my app in VB6

..or, code an IPC of VB6 <-> AutoIt and handle it that way ?, ...so you can do your thing in AutoIt while the VB6 does the DLL stuff ?
If you just gonna call it sporadically for some decryption. No clue of what you're upto. 🤔

[MattyD]

I'm a bit curious of how returning dllcalls "As String" is supposed to work on the VB side.
Does it blindly read from the string pointer until a null character is reached, or is there some other trickery at play where it allocates a buffer before the call?