Problems when returning DllStructGetPtr from function

[emcodem]

Dears,

one more time i would like to reach out for guidance with passing data between autoit and my own dll. 

This simple dll example just prints the string that it gets so we see it in Scite Console. 

Question is if this is some autoit thing that i should not further think about and just workaround, or is it connected to the fact that currently nothing frees any allocated memory or does my dll do something wrong or..

 

Please check out the test.au3 below, there are 2 loops that use the "Echo" function from my C dll.

1. In the first loop, where we do not use the Func WstrPtr to generate the data pointers, everything seems to work, always.
2. The problem is in the second for loop, when we use the Func WstrPtr to generate the Data pointers, it can randomly work but usually it will either crash (the dll? idk, autoit exit code says 0), or output some bogus as if the memory pointer is slightly off, pointing to some nearby memory that it should not point to.

One funny Example Result from Scite console (you see in the au3 code that the content of our string can never be "wstr", so this clearly points to some other memory):

You wrote: this works
You wrote: this works
You wrote: this works
You wrote: this works
You wrote: this works
You wrote: test
You wrote: wstr
You wrote: wstr
You wrote: wstr
You wrote: wstr

As said, in the second for loop it can also crash (not sure if its a crash really but it does not print 5 times), or write empty, or some binary stuff.

FileBridge.dll

mydll.h:

#include <wchar.h>
#define DECLDIR __declspec(dllexport)

extern "C"
{
    DECLDIR wchar_t* Echo(wchar_t* utf16String);  // Function takes a pointer to MyStructure
}

mydll.c:

#include "pch.h"
#include <iostream>
#include <Windows.h> 
#include "FileBridge++.h"

using namespace std;

extern "C"
{

    DECLDIR wchar_t* Echo(wchar_t* utf16String)
    {
        wcout << "You wrote: " << utf16String << endl;
        return utf16String;

    }
}

test.au3:

Global Const $g_sFileDll      = 'C:\dev\FileBridge++_tester\FileBridge++\x64\Debug\FileBridge.dll'
Local $hdll = DllOpen($g_sFileDll)

;~ this seems to work always
For $i = 5 To 1 Step -1
    Local $str = "this works"
    Local $ptr = DllStructCreate("wchar [" & StringLen($str) + 1 & "]") ; +1 for null terminator
    DllStructSetData($ptr, 1, $str)
    Local $bla = DllStructGetPtr($ptr)
    DllCall($hdll, "WSTR", "Echo", "ptr", $bla)
Next


;~ this SHOULD be exactly the same as above but using functions. It usually does not work (longer strings seem to work better funny enough)
Func WstrPtr($str)
    Local $ptr = DllStructCreate("wchar [" & StringLen($str) + 1 & "]") ; +1 for null terminator
    DllStructSetData($ptr, 1, $str)
    Local Const $bla = DllStructGetPtr($ptr)
    return $bla
EndFunc

Func CallEcho($ptr)
    DllCall($hdll, "WSTR", "Echo", "ptr", $ptr) ;the C side prints You wrote: to scite console
EndFunc

Local $dataptr = WstrPtr("test")

For $i = 5 To 1 Step -1
    CallEcho($dataptr)
Next

Same result with

For $i = 5 To 1 Step -1
    Local $dataptr = WstrPtr("test")
    CallEcho($dataptr)
Next

and

For $i = 5 To 1 Step -1
    Local $dataptr = WstrPtr("test")
    DllCall($hdll, "WSTR", "Echo", "ptr", $dataptr)
Next

 

Edited December 29, 2024 by emcodem

[pixelsearch]

I think the memory content used by a structure (created inside a function with DllStructCreate) is released when the function ends (except if you Return the structure variable name to the calling function)

That's why Func WstrPtr() returns a correct string (see "t e s t" in the pic below)

But after you return from Func WstrPtr() and call Func CallEcho() then the memory content at the same location has changed (see pic below where we don't read "t e s t" anymore at the same memory location)

I just found an old discussion about this, between 3 MVP's (PaulIA, Valik, Zedna) in a topic named... DllStructCreate and Scope

Correction : 2 MVP's (PaulIA, Zedna) and a developer (Valik)

Edited December 30, 2024 by pixelsearch

[emcodem]

Fantastic explaination, thank you so much @pixelsearch

Also thanks a lot for the linked topic, it contains very helpful informations for me!

[emcodem]

Sorry but i did not yet fully understand what is the best solution to this topic.

My current solution is this, which seems to work but i still require 2 additional lines and also i use "DllStructGetPtr" in all the DllCalls instead of doing this in the MakeWstrPtr Function.

Func MakeWstrPtr(ByRef $out,$str_data)
    Local $_struct = DllStructCreate("wchar [" & StringLen($str_data) + 1 & "]") ; +1 for null terminator
    DllStructSetData($_struct, 1, $str_data)
    $out = $_struct
EndFunc

As you can see, DllStructGetPtr is not part of MakeWstrPtr, also i believe the shortest way to use it is this:

Func CallEcho($s_data)
    Local $_ptr
    MakeWstrPtr($_ptr,$s_data)
    DllCall($hdll, "WSTR", "Echo", "ptr", DllStructGetPtr($_ptr)) 
EndFunc

I was hoping to get some more hint to further minimize the number of lines in my functions that need MakeWstrPtr, it would make the final code more readable.

Edited December 31, 2024 by emcodem

[pixelsearch]

Hello,
Instead of creating the Global variable $out in the main body of the script and passing 2 parameters to MakeWstrPtr() , why not simply return the structure variable name when the function ends, something like this :

Global $g_tStruct = WstrPtr("test")

For $i = 5 To 1 Step -1
    CallEcho()
Next

Func WstrPtr($str)
    Local $tStruct = DllStructCreate("wchar [" & StringLen($str) + 1 & "]") ; +1 for null terminator
    DllStructSetData($tStruct, 1, $str)
    Return $tStruct
EndFunc

Func CallEcho()
    Local Static $pStruct = DllStructGetPtr($g_tStruct)
    DllCall($hDll, "WSTR", "Echo", "ptr", $pStruct)
EndFunc

Concerning the pointer, juste use a Static variable inside CallEcho() so its value won't be destroyed when the function ends. I tested the memory content and it was ok (e.g.  "t e s t" was constantly there, during the 5 calls to CallEcho)

Edited December 31, 2024 by pixelsearch
typo

[emcodem]

Thank you for your time and Brains.

The problem was that i was just poking around but not really understand what exactly is going on even if you told me already that i need to return the struct.
I don't want to use statics and globals because this will be used in a UDF that allows to access the exportet methods from my C dll.

This whole Echo code was just created to explain and debug this very issue.

From this perspective it might be good to just dump the idea of the MakeWstrPtr funciton completely and write out the for DllStructCreate,SetData and GetPtr everywhere. This would make the final code very ugly to read, there would be hundred lines like this, it would be hard to track the variables for me.

    $struct = DllStructCreate("wchar [" & StringLen($s_str) + 1 & "]") ; +1 for null terminator
    DllStructSetData($struct, 1, $s_str)

But i think i understand now every Aspect of this and i may go with the following solution:

Func __MakeWstrPtr($s_str, ByRef $struct)
    $struct = DllStructCreate("wchar [" & StringLen($s_str) + 1 & "]") ; +1 for null terminator
    DllStructSetData($struct, 1, $s_str)
    return DllStructGetPtr($struct)
EndFunc

Func CallEcho($sToEcho)
    Local $struct_1
    Local $_ptr1 = __MakeWstrPtr($sToEcho,$struct_1)
    DllCall($hdll, "WSTR", "Echo", "ptr", $_ptr1) ;the C side prints You wrote: to scite console
EndFunc

We still need 2 Lines for using MakeWstrPtr function where i would prefer one line but i think at least this way we will end up with much better readable code. This way all the releases of structs and ptrs is done when CallEcho returns which is highly desired.

If i understood correctly, to avoid the Byref Struct parameter, we could potentially return an Array holding struct and ptr but performance is a big concern so we want to avoid spending time on allocating arrays or similar just to make the code more readable.

 

The final result using above solution can look like:

Func UpdateOne($ptr_mongocollection, $search, $update, $options)
    ;~https://www.mongodb.com/docs/manual/reference/method/db.collection.updateOne/
    ;~commonly used options: {"upsert":true}
    ;~returns json str like { "modifiedCount" : 1, "matchedCount" : 1, "upsertedCount" : 0 }
    Local $_s1,$_s2,$_s3;
    Local $_searchptr   = __MakeWstrPtr($search,$_s1)
    Local $_updateptr   = __MakeWstrPtr($update,$_s2)
    Local $_optptr      = __MakeWstrPtr($options,$_s3)
    Local $a_result     = DllCall($hdll, "WSTR", "UpdateOne", "ptr",$ptr_mongocollection, "ptr", $_searchptr, "ptr", $_updateptr, "ptr", $_optptr);
    ConsoleWrite("UpdateOne: " & $a_result[0] & " Error: " & _WinAPI_GetLastError() & @CRLF)
    return $a_result[0]
EndFunc

 

Edited December 31, 2024 by emcodem
Add final result

[pixelsearch]

Yes, ByRef seems the way to go as you're calling __MakeWstrPtr from another function, so your final solution looks correct, bravo !

Just a reminder : in case $s_str is a huge string, then ByRef should also be considered for the $s_str parameter, something like :

Func __MakeWstrPtr(ByRef $s_str, ByRef $struct)
or even
Func __MakeWstrPtr(Const ByRef $s_str, ByRef $struct)

Some explanations from this link

ByRef should be used when passing large amounts of data (such as the contents of a file) where copying all the data would impose a significant performance penalty. [maybe your case when $s_str is a huge string]

Another advantage is that passing a parameter ByRef when the function is intended to change the content of the parameter removes any requirement to Return the changed value as the original is directly affected. [definitely your case for the $struct parameter]

Now, if your $s_str parameter contains only short strings, then it shouldn't slow down the whole process if you don't use ByRef $s_str

Good luck and wishing you a happy new year

Edited December 31, 2024 by pixelsearch
typo

[emcodem]

Thank you again and very well spotted, of course some of my functions will take potentially huge strings as parameters. 

Const ByRef is the perfect solution, lovely!

Happy new year too!