How can my UDF find dependency DLLs

[emcodem]

Dears,

in order to generate a general purpose UDF for my mongodb driver, i need to come up with a solution for resolving the dependency dll files. 

The goal is to provide a zip download with a single folder e.g. MongoUDF.

As my dlls are portable and not installed on the system, the user unzips them along with my au3. Contents of the folder looks like this:

.\MongoDB.au3

.\dependencies\MongoCBridge.dll

.\dependencies\mongoc_driver_1.29.1\mongoc-1.0.dll

.\dependencies\mongoc_driver_1.29.1\bson-1.0.dll

Now the Problem is that the UDF code has to find these files but when the user #includes my MongoDB.au3, it looses the ability to know in which folder it was, which also means it does not know where the dependencies subfolder is. 

My current plan is to solve this with a Init function that the user must call before he can use any of my functions.

Func _Mongo_Init($sInstallDir = @ScriptDir & "\include\MongoDB_UDF\")
    Local $sBridgeDllPath = $sInstallDir & "\dependencies\MongoCBridge.dll"
    Local $sMongoCDllPath = $sInstallDir & "\dependencies\mongoc_driver_1.29.1"
    If Not(FileExists ( $sBridgeDllPath )) Then
        ConsoleWriteError("Not found: " & $sBridgeDllPath)
        Exit(42)
    EndIf
    _WinAPI_SetDllDirectory ($sMongoCDllPath)
    Global CONST $__hMongo_1_29_1 = DllOpen($sBridgeDllPath)
    _WinAPI_SetDllDirectory()
    If (@error) Then
        ConsoleWriteError("MongocBridgeDll")
    EndIf
EndFunc

Is this the best way to do it? Or maybe are there other well known UDF's that need external DLL's which are not in the PATH, how do they solve this?

Edited Thursday at 09:38 PM by emcodem

[MattyD]

hey mate, a couple of thoughts - take or leave them!

Having an "path" parameter in a startup function is a great idea, but make it the path to the main DLL. If you're expecting a folder layout at that point I guess you could hard code a relative path to the dependencies - but I'd personally just throw the dlls together in one folder as there's only 3 of them!

We shouldn't need to know that "include" path for this function - that should only be relevant when including the udf with the directive.

Also there's also no need to verify the dll physically exists - just attempt to open it and return the error from DllOpen if it fails.

[emcodem]

13 hours ago, MattyD said:

Having an "path" parameter in a startup function is a great idea, but make it the path to the main DLL. If you're expecting a folder layout at that point I guess you could hard code a relative path to the dependencies - but I'd personally just throw the dlls together in one folder as there's only 3 of them!

We shouldn't need to know that "include" path for this function - that should only be relevant when including the udf with the directive.

Also there's also no need to verify the dll physically exists - just attempt to open it and return the error from DllOpen if it fails.

Hey @MattyD thanks for investigating my stuff. 
I am very interested in what makes you think it is now different from what you describe, in the code above i just set the default directory to @scriptdir/include/MongoDB_UDF, if the user extracted it to that location he can call _Mongo_Init without parameters but any other case he needs to provide the installdir of the UDF.

About verifying, thats also very interesting, so i guess there is no "standard" for this (the UDF guide from here does not mention about external DLLs), so you would expect my UDF not to force program exit even if there was an obvious installation error - and prefer that the installation validation is part of the users script actually?

[RTFC]

My advice: don't use a zip; instead, use a proper installer that 1. creates your subdir structure and all its content wherever the user wants, 2.  the content includes an .ini (text) file that your main programme reads in at startup; this contains a variable that will hold the path to the dll(s); and 3. your installer writes the actual path of the dll's to the local .ini file upon (local) installation (wherever the user wanted it to be). Personally I use Inno Setup Compiler, which is free, and setting it up is trivially easy.

Edited Friday at 03:52 PM by RTFC
clarification

[emcodem]

Hey RTFC!

Thanks for your attention! Are there noteable example UDFs that work with installers?

[ahmet]

What about Subrogation? You can embed your dll file in your include script and load it from memory?

[MattyD]

7 hours ago, emcodem said:

I am very interested in what makes you think it is now different from what you describe

Its more of a conceptual thing. - The person using the UDF is the one creating the program, and should be the one responsible for the projects layout IMO.

That init function shouldn't care about how I've installed the UDF, it just needs to know where the DLL is.  Personally I wouldn't even have a default path specified -  just try to do a DllOpen("MongoCBridge.dll") by default.

Other autoit libraries with external dlls that come to mind - AutoItObject and MySQL

Edit: totally missed the second part!

7 hours ago, emcodem said:

so you would expect my UDF not to force program exit even if there was an obvious installation error

Yep. the udf should report an error - set @error or return false etc. The code calling the init func should decide how to proceed. There maybe other handles to close before exiting or whatever...

Edited Friday at 11:12 PM by MattyD

[emcodem]

2 hours ago, ahmet said:

What about Subrogation? You can embed your dll file in your include script and load it from memory?

I believe such a technique is very much preferred, even if the udf has 10mb this way. It solves a lot of problems like ensure dll versions and udf version matches and it should avoid possible antivirus issues. Also it makes everything fully portable, the user does not even need to care about anything when compiling his scripts to exe. 

I just fear my bridge dll has to be a single self contained static dll without dependencies but i was not yet able to include the mongoc dlls linked into my bridge dll. I am currently  waiting for reply in their forums about static linking issue and hoping to get a self contained build finally. 

@MattyD ok thanks then will check the examples out and use seterror. 

 

Edited Saturday at 12:04 AM by emcodem
Typo

[RTFC]

21 hours ago, emcodem said:

noteable example UDFs that work with installer

yap, this one for example.

[UEZ]

I wrote this function to list the imports of a DLL.

expandcollapse popup

;Coded by UEZ build 2024-06-09
#AutoIt3Wrapper_UseX64=n

#include <Array.au3>
#include <Debug.au3>
#include <String.au3>
#include <WinAPIFiles.au3>
#include <WinAPIProc.au3>
#include <WinAPIRes.au3>
#include <WinAPISys.au3>

Const $IMAGE_DIRECTORY_ENTRY_IMPORT = 1

Global $sFile = FileOpenDialog("Select a DLL file", "", "DLL (*.dll)", $FD_FILEMUSTEXIST)
If @error Then Exit

Global $a = _WinAPI_GetBinaryType2($sFile)
Global $b = _WinAPI_GetBinaryType2(_WinAPI_GetProcessFileName())
If $a <> $b Or $a = "Error" Or $b = "Error" Then Exit MsgBox($MB_ICONERROR, "Error", "Script and DLL have not same binary type!", 10)

Global $i, $aResult = _WinAPI_ListDLLImports($sFile)
_DebugArrayDisplay($aResult, StringRegExpReplace($sFile, ".+\\(.+)", "$1") & " imports")

Func _WinAPI_ListDLLImports($sFile, $iLevel = 1, $bRec = False, $bDisplayLocalFilesOnly = True, $iMaxRecLevel = 10)
    If $iLevel > $iMaxRecLevel Then Return
    Local Const $hModule = _WinAPI_LoadLibraryEx($sFile, BitOR($DONT_RESOLVE_DLL_REFERENCES, $LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR))
    If Not $hModule Then Return SetError(1, 0, 0)
    Local Const $tSize = DllStructCreate("ulong bytes")
    ;https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-imagedirectoryentrytodata
    Local Const $aReturn = DllCall("Dbghelp.dll", "ptr", "ImageDirectoryEntryToData", "ptr", $hModule, "boolean", True, "ushort", $IMAGE_DIRECTORY_ENTRY_IMPORT, "struct*", $tSize)
    If Not IsArray($aReturn) Or @error Then
        _WinAPI_FreeLibrary($hModule)
        Return SetError(2, 0, 0)
    EndIf
    If Not $aReturn[0] Then
        _WinAPI_FreeLibrary($hModule)
        Return SetError(3, 0, 0)
    EndIf
    Local $tIMAGE_IMPORT_DESCRIPTOR, $sDLLName, $sPath = StringRegExpReplace($sFile, "(.+\\).+", "$1"), $sFN, $i = 0, $sDLL = StringRegExpReplace($sFile, ".+\\(.+)", "$1")
    Static $sDLLFiles = ""
    ConsoleWrite(_StringRepeat(@TAB, $iLevel - 1) & $sDLL & @CRLF)
    If $iLevel > 1 Then $sDLLFiles &= $sDLL & "|"
    While 1
        $tIMAGE_IMPORT_DESCRIPTOR = DllStructCreate("dword dummy;dword TimeDateStamp;dword ForwarderChain;dword Name;dword FirstThunk;", $aReturn[0] + $i)
        If Not $tIMAGE_IMPORT_DESCRIPTOR.FirstThunk Then
            _WinAPI_FreeLibrary($hModule)
            If $iLevel = 1 Then
                Local $aResult = StringSplit(StringTrimRight($sDLLFiles, 1), "|", 2)
                $aResult = _ArrayUnique($aResult, 0, 0, 0, $ARRAYUNIQUE_NOCOUNT)
                _ArraySort($aResult)
                Return $aResult
            EndIf
            Return
        Else
            $sDLLName = _WinAPI_WideCharToMultiByte(DllStructCreate("char s[256];", $hModule + $tIMAGE_IMPORT_DESCRIPTOR.name).s, 65001)
            $sFN = $sPath & $sDLLName
            If $bRec Then
                If FileExists($sFN) Then
                    $iLevel += 1
                    _WinAPI_ListDLLImports($sFN, $iLevel)
                    $iLevel -= 1
                Else
                    If Not $bDisplayLocalFilesOnly Then $sDLLFiles &= $sDLLName & "|"
                    ConsoleWrite($iLevel & ":" & _StringRepeat(@TAB, $iLevel) &  $sDLLName & @CRLF)
                EndIf
            Else
                ConsoleWrite($iLevel & ":" & _StringRepeat(@TAB, $iLevel) &  $sDLLName & @CRLF)
                If Not $bDisplayLocalFilesOnly Then
                    $sDLLFiles &= $sDLLName & "|"
                Else
                    If FileExists($sFN) Then $sDLLFiles &= $sDLLName & "|"
                EndIf
            EndIf
        Endif
        $i += DllStructGetSize($tIMAGE_IMPORT_DESCRIPTOR)
    WEnd
EndFunc

; #FUNCTION# ====================================================================================================================
; Author.........: UEZ
; Modified.......:
; ===============================================================================================================================
Func _WinAPI_GetBinaryType2($sFile)
    Local $hFile = _WinAPI_CreateFile($sFile, 2, 2, 2)
    If Not $hFile Or @error Then Return SetError(1, 0, 0)
    Local $hMapping = _WinAPI_CreateFileMapping($hFile, 0, Null, $PAGE_READONLY, Null)
    If Not $hMapping Then
        _WinAPI_CloseHandle($hFile)
        Return SetError(2, 0, 0)
    EndIf
    Local Const $pAddress = _WinAPI_MapViewOfFile($hMapping, 0, 0, $FILE_MAP_READ)
    If Not $pAddress Or @error Then __ReturnGBT2($hMapping, $hFile, 3)
    Local Const $aHeader = DllCall("Dbghelp.dll", "ptr", "ImageNtHeader", "ptr", $pAddress)
    If @error Or IsArray($aHeader) = 0 Then Return __ReturnGBT2($hMapping, $hFile, 4)
    Local Const $tIMAGE_NT_HEADERS = DllStructCreate("dword Signature;ptr FileHeader;ptr OptionalHeader;", $aHeader[0])
    If @error Or Not IsDllStruct($tIMAGE_NT_HEADERS) Then Return __ReturnGBT2($hMapping, $hFile, 5)
    Local Const $tIMAGE_FILE_HEADER = DllStructCreate("word Machine;word NumberOfSections;dword TimeDateStamp;dword PointerToSymbolTable;dword NumberOfSymbols;word SizeOfOptionalHeader;word Characteristics;", _
                                                      DllStructGetPtr($tIMAGE_NT_HEADERS) + 4)
    If @error Or Not IsDllStruct($tIMAGE_FILE_HEADER) Then Return __ReturnGBT2($hMapping, $hFile, 6)
    __ReturnGBT2($hMapping, $hFile, 0)
    Switch $tIMAGE_FILE_HEADER.Machine
        Case 0x014c
            Return "x86"
        Case 0x0200
            Return "Intel Itanium"
        Case 0x8664
            Return "x64"
        Case Else
            Return "Error"
    EndSwitch
EndFunc   ;==>_WinAPI_GetBinaryType2

Func __ReturnGBT2(Byref $hMapping, ByRef $hFile, $iError)
    _WinAPI_CloseHandle($hMapping)
    _WinAPI_CloseHandle($hFile)
    If $iError Then Return SetError($iError, 0, 0)
EndFunc   ;==>__ReturnGBT2

 

Edited Saturday at 06:02 PM by UEZ

[emcodem]

4 hours ago, UEZ said:

I wrote this function to list the imports of a DLL.

 

Thanks buddy, but not sure how this can be of help in my case. If this is only about finding out which dependencies there are then it might not really be related to my problem because it is about my own dll so i know the dependencies because i actively added all dependencies manually to the project. Also i know the dependencies of the dependencies because i usually analyze the stuff that i add to my code using dumpbin and similar tools. I need to do this because i must make sure that my final release is fully portable and also i want to know which OS versions i can support (looking at you, bcrypt.dll ^^).

 

6 hours ago, RTFC said:

yap, this one for example.

Wow, that looks like a lot of work you did there! Thanks a lot, even when i think that an installer is not really a fit for my Mongo Driver project i'll check out how you do it because i was not really happy with the NSIS stuff in the past, i mean it works but it does not provide much convenience functions from my perspective.

Edited Saturday at 10:37 PM by emcodem

[UEZ]

Sorry, then I misunderstood your problem.

As far as I understand your problem, RTFC's suggestion would be the best. There are a lot of portable programs with a setup routine.

 

 

[argumentum]

...so you want to install AutoIt3.exe with a script.au3 and run it that way ?
If that is so, any installer or even compressor the creates executables can decompress in a temp folder, run your installer.au3 and that would be all you need.
AutoIt is better than NSIS if you're more familiar with AutoIt. My 2 cents. Just an idea

[emcodem]

40 minutes ago, argumentum said:

...so you want to install AutoIt3.exe with a script.au3 and run it that way ?

No i don't have any special or dedicated usage/installation scenario how in my mind, in best case my UDF can just be used like any Core Module and ideally, when users compile their scripts (which use my UDF) to exe, they don't have to think about my external dependency dlls but it just magically works.

An installer can be done but it would from my perspective not be helpful for compiled deployment of users scripts.

My current impression is that "embedding" my binaries into the au3 (e.g. as base64) will be the only option to "make it magically work". 
From there i either need to use some method like "Subrogation" (which might only work for a single dll but not a tree of dependencies) OR i dynamically dump the binaries to disk at runtime and include them from a temp location.

If i remember correctly, we had massive issues with "BinaryCall" (isnt that very similar to subrogation?) which comes with some JSON au3, e.g. it was preventing windows from shutdown when it was used while shutdown was initiated (https://ffastrans.com/frm/forum/viewtopic.php?t=1184&hilit=shutdown).

So i probably stick to the "Dump to disk" approach if nothing speaks against it?

 

 

Edited yesterday at 12:58 AM by emcodem