Questions about c library pointers and structures

[MattHiggs]

Hello all.  So, as I have stated before, I have never worked with and desperately tried to avoid using the dll process functions to work with c libraries, but I am now trying to learn more about how to do this by creating an autoit udf.  On that note, consider the following function embedded within a C library DLL:

If you want, you can see the full documentation for the function and overall library here.  I have also attached the external header file for the library to this post.  So, There are a couple of questions I am hoping to answer.  The first is in regards to pointers, as they are by far where I am struggling most.  For example, if we look at the function pictured above, the first parameter in the function is a pointer to a wimlib_tchar data type,  The documentation explains:

Quote

To support Windows as well as UNIX-like systems, wimlib's API typically takes and returns strings of wimlib_tchar which have a platform-dependent type and encoding.

On Windows, each wimlib_tchar is a 2-byte wchar_t. The encoding is meant to be UTF-16LE. However, unpaired surrogates are permitted because neither Windows nor the NTFS filesystem forbids them in filenames.

Indeed, the external header file for the library contains this typedef declaration:

Quote

#ifdef _WIN32

typedef wchar_t wimlib_tchar;

#else

/** See @ref sec_encodings */

typedef char wimlib_tchar;

#endif

So far, this is the code that I have which calls the dll, and attempts to load a wim file using the above function:

expandcollapse popup

#include-once
#include "wimlibconstants"
#include <WinAPI.au3>

Global $gsWimlibDLL = "C:\tools\wimlib\libwim-15.dll"
Global $ghwimlib = 0 ; initialized by _init_WIMlib() and contains the handle to DllOpen($gsWimDLL)
Global $giWIMRef = 0 ; reference var. Value is >= 1 if $gsWimlibDLL is loaded.
Func _init_wimlib ($initflags = 0)
    $giWIMRef += 1
    If $giWIMRef > 1 Then
        Return SetExtended($ghwimlib, 0) ; dll already loaded
    EndIf

    If Not FileExists ( $gsWimlibDLL ) Then
        Return SetError ( 2, 0, -1 )
    EndIf

    $ghwimlib = DllOpen ( $gsWimlibDLL )
    If $ghwimlib = -1 Then
        $giWIMRef = 0
        Return SetError ( 1, 0, -1 )
    EndIf
    DllCall ( $ghwimlib, "int:cdecl", "wimlib_global_init", $initflags )
    If @error Then
        Return SetError(3, 0, -1)
    EndIf

    Return SetExtended ( _WinAPI_GetLastError (), $ghwimlib )

EndFunc


Func _quit_wimlib ()
    If $ghwimlib = 0 Or $ghwimlib = -1 Then
        Return SetError ( 1, 0, False )
    EndIf
    $giWIMRef -= 1
    If $giWIMRef = 0 Then
        DllCall ( $ghwimlib, "none:cdecl", "wimlib_global_cleanup" )
        DllClose ( $ghwimlib )
        $ghwimlib = 0
    EndIf

EndFunc

Func _open_wim ($wimfile, $openflags, $wimret)
    $openwim = DllCall ( $ghwimlib, "int:cdecl", "wimlib_open_wim", "WSTR", $wimfile, "int", $openflags, "more" )
EndFunc

My question is, does the dllcall instance in the _open_wim function look like it was declared correctly in regards to the first parameter?  You may have noticed that all of the dllcall functions use the 'cdecl' calling method, as the documentation does explicitly state:

Quote

If you need to access the DLL from non-C/C++ programming languages, note that the calling convention is "cdecl".

 I know the whole dllcall instance isn't correct, and that leads to probably the part that confuses me most of all.  The wimret parameter, which is a pointer to a pointer of a Wimstruct data type.  The only information that the documentation gives in regards to "wimstruct"

How do I even begin to implement this into the Dllcall?  One last question.  I am also trying to recreate the structures that this library uses in a separate file called "wimlibconstants.au3."  One of the structures that this library uses is called wimlib_wim_info, and, while I was able to easily figure out how to write the first few data fields, the data fields in the documentation eventually started coming with "default values" for lack of a better term:

Notice how the date fields lower in the list have an additional ": {integer}" appended after the data field?  Does this affect how the structure needs to be written in autoit?  If so, how?  Below is what I have written so far for this particular data structure:

Global Const $tagwimlib_wim_info = "byte guid[" & $WIMLIB_GUID_LEN & "];dword image_count;dword boot_index;dword wim_version;dword chunk_size;word part_number;word total_parts;int compression_type;UINT64 total_bytes;dword

I know I have asked a lot, and thank you in advance for any help you can offer.  I have spent a long time trying to figure all of this out, and I am starting to get discouraged that this might be something that is beyond my ability to learn.  Any help is appreciated.

wimlib.h

Edited May 11, 2020 by MattHiggs

[MattHiggs]

Ok.  So I am trying to better understand the DLL process functions.  Under what circumstances would you supply a dllcall parameter with a pointer to a struct using dllstructgetpointer rather than just provide it a direct variable.  Also, when would the data type be followed up with a *?  For example, I have already previously asked for some help concerning this function in a c library I am trying to create an autoit udf for:

as you can see, the last parameter is a bit tricky, as, if I am reading it right, it is a pointer to a pointer of a "wimstruct" data type, and the wimstruct data type is not a defined structure within the library, as the documentation simply describes it as "Opaque structure that represents a WIM, possibly backed by an on-disk file."  Documentation for library can be found here.  Thankfully, however, I found a very useful resource that has helped me immensely in my efforts to figure out how utilize the c library in autoit, which came in the form of an existing C# assembly which acts as a wrapper around the same C library, as C# is much easier for me to decipher than C.    After looking at the source code of this assembly, I found the function I was looking for:

/// External function

public static Wim OpenWim(string wimFile, OpenFlags openFlags)
        {
            Manager.EnsureLoaded();

            ErrorCode ret = Lib.OpenWim(wimFile, openFlags, out IntPtr wimPtr);
            WimLibException.CheckWimLibError(ret);

            return new Wim(wimPtr);
        }
 /// Lib.openwim
 internal ErrorCode OpenWim(string wimFile, OpenFlags openFlags, out IntPtr wimRet)
        {
            switch (UnicodeConvention)
            {
                case UnicodeConvention.Utf16:
                    return Utf16.OpenWim(wimFile, openFlags, out wimRet);
                case UnicodeConvention.Utf8:
                default:
                    return Utf8.OpenWim(wimFile, openFlags, out wimRet);
            }
        }
        
 /// Utf16.OpenWim
 
 internal class Utf16d
        {
            internal const UnmanagedType StrType = UnmanagedType.LPWStr;
            internal const CharSet StructCharSet = CharSet.Unicode;
            [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
            internal delegate ErrorCode wimlib_open_wim(
                [MarshalAs(StrType)] string wim_file,
                OpenFlags open_flags,
                out IntPtr wim_ret);
            internal wimlib_open_wim OpenWim;
         }

So, the third parameter is defined as an IntPtr data type in C#, which, after some research, is "INT_PTR" in autoit.  So, that being the case, I am trying to figure out how the dllcall function should look.  should I write it like this:

Func _open_wim ($wimfile, $openflags)
    $wimret = DllStructCreate ( "INT_PTR" )
    $openwim = DllCall ( $ghwimlib, "int:cdecl", "wimlib_open_wim", "WSTR", $wimfile, "int", $openflags, "INT_PTR*", DllStructGetPtr ( $wimret ) )
EndFunc

Or should the dllcall look more like this (without the asterisk):

Func _open_wim ($wimfile, $openflags)
    $wimret = DllStructCreate ( "INT_PTR" )
    $openwim = DllCall ( $ghwimlib, "int:cdecl", "wimlib_open_wim", "WSTR", $wimfile, "int", $openflags, "INT_PTR", DllStructGetPtr ( $wimret ) )
    
EndFunc

Because remember, the third parameter is a pointer to a pointer.  Any feedback would be awesome.

Edited May 14, 2020 by MattHiggs

[TheXman]

I'm not sure if I have the patience to get you up to speed on structs, dll calls, and pointers as it relates to how to implement them in AutoIt but I'm willing to try.  Some people will get it and some won't.  I will take the first step by showing you a working example that you can use to ask questions and to build upon.

In the example below, it makes a call to the wimlib_open_wim function.  Note that as it is written it will fail with an error code of 47, unless you happen to have a test.wim file in the script's directory.  Error code 47 is WIMLIB_ERR_OPEN, which means it cannot open the specified file.  But the point is that it is a working example that you can play with.  You will also notice that if the DllCall is successful, it display the array that is returned from the function call.  That is there solely for debugging purposes.  It's a good technique to use when debugging to see if you are getting good data back from the call, what the data is, and how it is formatted.  Of course you can comment it out or delete it if/when it isn't need.

For the record, I don't deal with WIM files and do not really care to.  However, I am quite familiar with DLL calls, structs, and pointers.  So if I can manage to help you without having to get into creating WIM files, I will try to do so.  At least I will up to a point.  The more effort you show in truly wanting to learn, and how well you are able to absorb the information, will determine how willing I will be to continue helping.

 

Let the fun begin! 

 

expandcollapse popup

#include <Constants.au3>
#include <Debug.au3>

Global $gpWimStruct = 0

;Open WIM file
$gpWimStruct = wimlib_open_wim("test.wim")
If @error Then Exit MsgBox($MB_ICONERROR, "ERROR", "Function failed with @error = " & @error & " @extended = " & @extended)
ConsoleWrite("WIMStruct pointer = " & String($gpWimStruct) & @CRLF)

;Release WinStruct structure resources
wimlib_free($gpWimStruct)


; #FUNCTION# ====================================================================================================================
; Name ..........: wimlib_open_wim
; Description ...: Open a WIM file and create a WIMStruct for it.
; Syntax ........: wimlib_open_wim($sWimFilePath, $iOpenFlags = 0)
; Parameters ....: $WimFilePath         A string containing the path to the WIM file to open.
;                  $iOptionFlags        [optional] An integer containing the option flags. Default = 0
; Return values .: Success:             Pointer to a WimStruct structure.
;                  Failure:             0 and sets @error flag to non-zero.
;                                       @extended is set to the function's return code.
;                  @error:              1 - wimlib_open_wim returned an error, @extended = return code
; Author ........: TheXman
; Related .......: https://wimlib.net/apidoc/group__G__creating__and__opening__wims.html#ga5c28905bda14c8969ac504f1526d0ae9
; ===============================================================================================================================
Func wimlib_open_wim($sWimFilePath, $iOpenFlags = 0)

    Local $aReturn

    ;Open WIM
    $aReturn = DllCall("libwim-15.dll", "int:cdecl", "wimlib_open_wim", _
                       "wstr",  $sWimFilePath, _
                       "int",   $iOpenFlags, _
                       "ptr*",  Null)
    If @error Then
        ;Handle DllCall error
        Exit MsgBox($MB_ICONERROR, "ERROR", "wimlib_open_wim DllCall failed with @error " & @error)
    ElseIf $aReturn[0] <> 0 Then
        ;Handle bad return code from function
        Return SetError(1, $aReturn[0], 0)
    EndIf

    _DebugArrayDisplay($aReturn, "wimlib_open_wim $aReturn")

    ;All is good, return pointer
    Return $aReturn[3] ;Pointer to WIMStruct structure

EndFunc

; #FUNCTION# ====================================================================================================================
; Name ..........: wimlib_free
; Description ...: Release a reference to a WIMStruct.
; Syntax ........: wim_open_wim_example($pWimStruct)
; Parameters ....: $pWimStruct          Pointer to the WIMStruct to release.
; Return values .: None
; Author ........: TheXman
; Related .......: https://wimlib.net/apidoc/group__G__creating__and__opening__wims.html#ga5c28905bda14c8969ac504f1526d0ae9
; ===============================================================================================================================
Func wimlib_free($pWimStruct)

    ;Free winstruct resources
    DllCall("libwim-15.dll", "none", "wimlib_free", _
            "ptr",  $pWimStruct)
    If @error Then Exit MsgBox($MB_ICONERROR, "ERROR", "wimlib_free DllCall failed with @error " & @error)

    ;All is good
    Return

EndFunc

 

Why did you create a new topic with the same questions when you could have just added another post to the old one?

 

Edited May 15, 2020 by TheXman

[Jos]

@MattHiggs, 

You are around here long enough to know one shouldn't open multiple topics on the same item, so Please don't.

Jos

[MattHiggs]

@TheXman Thanks for the input.  Your code worked great.  I will try not to take too much of your time, but there aren't a lot of resources that cover this topic.  For now, I only have one question.  I am trying to import a documented structure contained within this library into autoit.  Using some other resources, I have been able to get by, but as I went through the data fields, I started noticing that some of the entries contained additional integers appended to the very end of the data field.  Consider this struct:

struct wimlib_wim_info {

	/** The globally unique identifier for this WIM.  (Note: all parts of a
	 * split WIM normally have identical GUIDs.)  */
	uint8_t guid[WIMLIB_GUID_LEN];

	/** The number of images in this WIM file.  */
	uint32_t image_count;

	/** The 1-based index of the bootable image in this WIM file, or 0 if no
	 * image is bootable.  */
	uint32_t boot_index;

	/** The version of the WIM file format used in this WIM file.  */
	uint32_t wim_version;

	/** The default compression chunk size of resources in this WIM file.
	 */
	uint32_t chunk_size;

	/** For split WIMs, the 1-based index of this part within the split WIM;
	 * otherwise 1.  */
	uint16_t part_number;

	/** For split WIMs, the total number of parts in the split WIM;
	 * otherwise 1.  */
	uint16_t total_parts;

	/** The default compression type of resources in this WIM file, as one
	 * of the ::wimlib_compression_type constants.  */
	int32_t compression_type;

	/** The size of this WIM file in bytes, excluding the XML data and
	 * integrity table.  */
	uint64_t total_bytes;

	/** 1 iff this WIM file has an integrity table.  */
	uint32_t has_integrity_table : 1;

	/** 1 iff this info struct is for a ::WIMStruct that has a backing file.
	 */
	uint32_t opened_from_file : 1;

	/** 1 iff this WIM file is considered readonly for any reason (e.g. the
	 * "readonly" header flag is set, or this is part of a split WIM, or
	 * filesystem permissions deny writing)  */
	uint32_t is_readonly : 1;

	/** 1 iff the "reparse point fix" flag is set in this WIM's header  */
	uint32_t has_rpfix : 1;

	/** 1 iff the "readonly" flag is set in this WIM's header  */
	uint32_t is_marked_readonly : 1;

	/** 1 iff the "spanned" flag is set in this WIM's header  */
	uint32_t spanned : 1;

	/** 1 iff the "write in progress" flag is set in this WIM's header  */
	uint32_t write_in_progress : 1;

	/** 1 iff the "metadata only" flag is set in this WIM's header  */
	uint32_t metadata_only : 1;

	/** 1 iff the "resource only" flag is set in this WIM's header  */
	uint32_t resource_only : 1;

	/** 1 iff this WIM file is pipable (see ::WIMLIB_WRITE_FLAG_PIPABLE).  */
	uint32_t pipable : 1;
	uint32_t reserved_flags : 22;
	uint32_t reserved[9];
};

As you can see, some of the data fields have a " : 1" appended to the end of the data field.  How would I import this structure into autoit?  Would I just continue creating the structure and ignore the extra appended data for these fields?  Displayed below is what I have so far:

Global Const $tagwimlib_wim_info = "byte guid[" & $WIMLIB_GUID_LEN & "];dword image_count;dword boot_index;dword wim_version;dword chunk_size;word part_number;word total_parts;int compression_type;UINT64 total_bytes;dword"

thanks

Edited May 16, 2020 by Melba23
Huge quote removed

[jchd]

2 minutes ago, MattHiggs said:

uint32_t pipable : 1;

These are bitfields. This C syntax is just giving a name to the 10th bit of the current word (there is a total of 32 bits declared: 10*1 + 22). There is not direct equivalent syntax in AutoIt.

Declare this word as unsigned and use bit functions separately to access bits of interest.

[MattHiggs]

On 5/16/2020 at 4:32 AM, jchd said:

These are bitfields. This C syntax is just giving a name to the 10th bit of the current word (there is a total of 32 bits declared: 10*1 + 22). There is not direct equivalent syntax in AutoIt.

Declare this word as unsigned and use bit functions separately to access bits of interest.

Are there any udfs/code examples that you or anyone is aware of which demonstrates the "Use bit functions separately to access bits of interest?"  Totally understood the first part about creating the struct normally, but not so much that last part.

[markyrocks]

idk.  Took me like 15 minutes to get to this point on visual studio.  You'd probably be better off just building a dll that just instantiates the wimstruct and just pulls whatever info out of it that you want.  I beat that bush around for awhile and then realized how easy .....

#include <Windows.h>
#include "header.h"  //had to import and rename header file
#include<iostream>



int main() {
    WIMStruct* wim;
    wimlib_tchar file =(wimlib_tchar)"file";
    wimlib_open_wim(&file, NULL, &wim);
}

 

Quote

Ok.  So I am trying to better understand the DLL process functions.  Under what circumstances would you supply a dllcall parameter with a pointer to a struct using dllstructgetpointer rather than just provide it a direct variable.  Also, when would the data type be followed up with a *?

if it calls for a dllstruct pointer ususally you'd pass a dllstruct pointer to it if you need to acces the struct after the fact for some kinda value within.  besides that if you don't need to access it you can just pass a 0.  you can make a regular buffer in the windows udf if the function but i've done that and passed it as a pointer to a struct and then just used that pointer to create a struct and then read what i needed.  You can use the * as a different way to pass a pointer like if the type calls for a void* you could use that or just use a generic like ptr really just depends.  (i think) this kinda stuff is really touch and go.  You just got to play with it until youre ready to pull your hair out.

Edited May 19, 2020 by markyrocks

[MattHiggs]

7 minutes ago, markyrocks said:

idk.  Took me like 15 minutes to get to this point on visual studio.  You'd probably be better off just building a dll that just instantiates the wimstruct and just pulls whatever info out of it that you want.  I beat that bush around for awhile and then realized how easy .....

#include <Windows.h>
#include "header.h"  //had to import and rename header file
#include<iostream>



int main() {
    WIMStruct* wim;
    wimlib_tchar file =(wimlib_tchar)"file";
    wimlib_open_wim(&file, NULL, &wim);
}

 

And it is at that point that I lose interest.  The whole point of creating a udf for wimlib is to improve my understanding of autoit.  Thanks for the info though.  I appreciate it.  

[jchd]

;
; In C:
; struct {
;   uint32_t a : 1
;   uint32_t b : 1
;   uint32_t c : 2
;   uint32_t d : 4
;   uint32_t e : 24
; } t;
;
; Warning: C doesn't specify the bit order which is implementation-dependant
;
; here assume the following layout:
; eeeeeeee eeeeeeee eeeeeeee ddddccba
;

Local $t = DllStructCreate("dword")
DllStructSetData($t, 1, 0x0123045D)
Local $a = BitAND(DllStructGetData($t, 1), 0x01)
Local $b = BitAND(BitShift(DllStructGetData($t, 1), 1), 0x01)
Local $c = BitAND(BitShift(DllStructGetData($t, 1), 2), 0x03)
Local $d = BitAND(BitShift(DllStructGetData($t, 1), 4), 0x0F)
Local $e = BitAND(BitShift(DllStructGetData($t, 1), 8), 0x00FFFFFF)
ConsoleWrite(Hex($a) & @TAB & Hex($b) & @TAB & Hex($c) & @TAB & Hex($d) & @TAB & Hex($e) & @LF)

 

[markyrocks]

9 minutes ago, MattHiggs said:

And it is at that point that I lose interest.  The whole point of creating a udf for wimlib is to improve my understanding of autoit.  Thanks for the info though.  I appreciate it.  

I'm not knocking autoit but when you start running up against these walls you start to realize the limitations of the language.  Autoit is great for many things, I appreciate the effort they put in to at least make these types of ventures possible.  i've been down this road and its not fun beating your head against your keyboard.  I mean go for it.  I updated my last post.  The only other advice i can give you is that you can just figure out how big the struct needs to be.  Allocate a buffer or create a struct the same size or couple bits to spare and you can just traverse it via memory reads.  I've found that making structs that are just of type bytes and then just reading specific parts of it via creating a new struct of a specific type at that pointer location is a useful.  But again you could always just readprocessmemory().

[MattHiggs]

1 minute ago, markyrocks said:

I'm not knocking autoit but when you start running up against these walls you start to realize the limitations of the language.  Autoit is great for many things, I appreciate the effort they put in to at least make these types of ventures possible.  i've been down this road and its not fun beating your head against your keyboard.  I mean go for it.  I updated my last post.  The only other advice i can give you is that you can just figure out how big the struct needs to be.  Allocate a buffer or create a struct the same size or couple bits to spare and you can just traverse it via memory reads.  I've found that making structs that are just of type bytes and then just reading specific parts of it via creating a new struct of a specific type at that pointer location is a useful.  But again you could always just readprocessmemory().

Its not that I don't understand that autoit has its limitations,  I just don't know nearly as much about writing assemblies in C or C++ when compared to autoit.  I want to solidify what I do know (autoit) before moving on to learn another language.  But C/C++ is definitely on the list.

[markyrocks]

8 minutes ago, MattHiggs said:

Its not that I don't understand that autoit has its limitations,  I just don't know nearly as much about writing assemblies in C or C++ when compared to autoit.  I want to solidify what I do know (autoit) before moving on to learn another language.  But C/C++ is definitely on the list.

well check out my sig.  I wrote the toolhelp32  couple  examples of using structs in there the whole thing relies on them.

[TheXman]

On 5/18/2020 at 6:10 PM, MattHiggs said:

Are there any udfs/code examples that you or anyone is aware of which demonstrates the "Use bit functions separately to access bits of interest?"

First and foremost, if you want to become proficient with working with DLL APIs in AutoIt, then there are several things that you need to have a good understanding of.  Among those things are how data is stored in memory, the base16 (Hex) numbering system, and the base2 (binary) numbering system.  Without a good understanding of at least those 3 things, trying to figure out the rest will be extremely difficult.

Since you have some sort of affinity for wanting to work with WIM, I will stick with WIM examples.  The flags field of the WIM_INFO structure is stored as a 32-bit unsigned integer.  That means that it is 4 bytes, 8 hex characters, or 32 bits (1's or 0's).  All of those represent the same storage area.  The difference is how it is logically represented.  At the lowest levels, everything in memory is binary, either a 1 or a 0.  So back to the flags fields.  It is made up of 32 binary bits that can either be a 1 or a 0, either it is on or off, true or false.  Currently, only 10, single-bit flags, of the 32 bits are being used for wim_info flags.  The other 22 are reserved (not in use yet).  It is called the flags field because in those 4 bytes of memory it can hold up to 32 different flags that can either be set (1) or unset(0), basically 32 toggle switches.  You can think of it in memory like this (assuming all bits are unset:

0000 0000 0000 0000 0000 0000 0000 0000 (base2)

Each flag bit, when set, represents a specific value.  Assuming the we are looking at a big-endian value, the bit values going from right to left are: 1, 2, 4, 8, 16, 32, 64, 128,...2147483648.  So if I want to represent the first 4 flags as being set, it would look like:

0000 0000 0000 0000 0000 0000 0000 1111 (base2/binary)  or  0x0000000F (base16/hex)  or  15 (base10/decimal)

With all of that said, below is a starter WIM UDF (wimlib.au3) that I have written and an example file to show the Wim_Open API that I showed you earlier, a new _wimlib_get_wim_info API function,  a new function called _wimlib_wiminfo_is_flag_set that lets you see if a given flag is set (true) or unset (false), and a function _wimlib_wiminfo_get_flag_name to lookup the flags name by its bit position.  Pay special attention to the flag enums and constants that I created in the UDF file.  And then related it back to what I said above about the the flags' bit values and bit positions.  In the example file, I added examples of how to get the wim_info information using the API, how the flags field can be retrieved from the WIM_INFO structure, how flags can be set and how flag states can be interrogated.  Hopefully, by figuring out how the structs, functions, enums, and constants work, and by playing around with the examples, it will become clearer in time.

I hope this helped 

P.S.

You may want to get familiar with AutoIt's bitwise functions (BitAND, BitOR, BitXOR, etc.) if you plan to play with bits.

 

 

Edited May 20, 2020 by TheXman
Removed my files due to lack of response and OP appears to have given up

[TheXman]

I updated the files attached to the previous post.  I added a new API (_wimlib_get_wim_info), a new function to check the state of a given WIM_INFO bit flag (_wimlib_wiminfo_is_flag_set), and a function that will let you lookup the bit flag's name by it's bit position (_wimlib_wiminfo_get_flag_name).  I also added examples of the new stuff in the example file.

The starter UDF file and the example file should get you back on track and motivated to continue learning more about structs, pointers, and dllcalls.

[jchd]

@MattHiggs did you try my code sample above?