DllCall to BZIP2.dll method fails when I try to use function

[Hammerfist]

Hi guys.

I try to choose an algorithm to compress some string data in-memory, without file medium to store in sqlite as blob. I plan to store about 100k blobs each 80k symbols (uncompressed). First opinion was LZMA udf by Ward, but udf file doesn't exist.

Then i tried to use BZIP2 dll in DllCall, but it simply doesn't work and throws @error = 1.

 

Here is an example:

Spoiler

expandcollapse popup

$hBZDll = DllOpen(@ScriptDir & "\bzip2.dll") ; obtained at http://gnuwin32.sourceforge.net/packages/bzip2.htm

$data = "binbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbinbin"

$Ret = _DB_CompressData($data)
MsgBox(0, $Ret[2], Binary($Ret[0]))

Func _DB_CompressData($data)

    Local $Size_O, $Size_C, $Size_R, $a_pBuffer, $ret[3]
    $Size_O = BinaryLen($data)
    $Size_R = Ceiling(($Size_O*1.01)+600) ; reserve some bytes accordingly to BZIP2 documentation
    
    $a_pBuffer = DllStructCreate('source byte[' & $Size_O & '];sourceLen uint;dest byte[' & $Size_R & '];destLen uint;blockSize100k int;verbosity int;workFactor int;')
    DllStructSetData($a_pBuffer, "source", $data)
    DllStructSetData($a_pBuffer, "sourceLen", $Size_O)
    DllStructSetData($a_pBuffer, "destLen", $Size_R)
    DllStructSetData($a_pBuffer, "blockSize100k", 9) ; this and below are default values in BZIP2 docs
    DllStructSetData($a_pBuffer, "verbosity", 0)
    DllStructSetData($a_pBuffer, "workFactor", 30)
    
    $dllres = DllCall($hBZDll, "int", "BZ2_bzBuffToBuffCompress", _
    "dest", DllStructGetPtr($a_pBuffer, "dest"), _
    "destLen", DllStructGetPtr($a_pBuffer, "destLen"), _ ; destLen is uint * type, so we need a pointer
    "source", DllStructGetPtr($a_pBuffer, "source"), _
    "sourceLen", DllStructGetData($a_pBuffer, "sourceLen"), _
    "blockSize100k", DllStructGetData($a_pBuffer, "blockSize100k"), _
    "verbosity", DllStructGetData($a_pBuffer, "verbosity"), _
    "workFactor", DllStructGetData($a_pBuffer, "workFactor"))
    
    MsgBox(0, "Error", "@error = " & @error & @CRLF & "@extended = " & @extended)
    
    $Size_C = DllStructGetData($a_pBuffer, "destLen")
    
    $Ret[0] = DllStructGetData($a_pBuffer, "dest")
    $Ret[1] = $Size_O
    $Ret[2] = $Size_C

    Return $ret

EndFunc   ;==>_CompressData


#cs
BZ2_bzBuffToBuffCompress function prototype on C++

int BZ2_bzBuffToBuffCompress( char*         dest,
                              unsigned int* destLen,
                              char*         source,
                              unsigned int  sourceLen,
                              int           blockSize100k,
                              int           verbosity,
                              int           workFactor );
#ce

 

 

What should I do with this error? Where am I wrong?

Thanks.

Edited December 26, 2016 by Hammerfist

[Hammerfist]

And trancexx , I know about your Native API Compression =) Didn't use it due to low compression ratio - it's 34%. Btw $COMPRESSION_FORMAT_XPRESS_HUFF shows awesome 11% of compression, but I could not make it decompress the data.

_WinAPI_DecompressBuffer($a_pBuffer[0], $Size_O, $a_pBuffer[1], $Size_C, $COMPRESSION_FORMAT_XPRESS_HUFF) shows @error = 10 and @extended = c00000e8.

_WinAPI_CompressBuffer and _WinAPI_DecompressBuffer using the engine $COMPRESSION_FORMAT_XPRESS work very well, but if we add "_HUFF" it throws error.

[Hammerfist]

In my case the best choice is LZMA UDF by Ward, but there is an empty attachment in his post.

[JohnOne]

second time you mentioned that udf without a link.

[Hammerfist]

I meant this one.

UPD: got the copy in original thread, but it seems outdated.

Edited December 26, 2016 by Hammerfist

[jchd]

If the use is for compression of blob in SQLite storage and if you don't have concurency timing requirement, I'd develop an SQLite extension to compress/decompress data on the fly, keeping the applicative layer clean, simple, blindly ignorant of what's being done under its feet and avoiding manipulating data several times within AutoIt (forcibly slower than within C code).

[Hammerfist]

Can I write proper subroutines for SQLite using AutoIt? I think no.

And my app is kinda archive. It keeps a database in compressed state and extracts one blob of data on demand, one, two or ten times per day. So i can sacrifice some processor time at the import stage to reach maximum compression.

Applicative layer remains clean enough. I'll simply change the "$data = $uncompressed_blob" to "$data = _DB_UnpackBlob($compressed_blob)".

[jchd]

Yes you can (create SQLite functions written in AutoIt) but it isn't worth the burden, especially in you "archive" use case.

When the expected benefit is large enough it's much better to write directly an extension function in C and load it at connection time, or auto-load it for all subsequent connections in the same run, or statically include it in a homebrew SQLite.dll build.

If ever you're interessed to add AutoIt functions as extensions to SQLite you can use this (posted on the French forum): https://autoitscript.fr/forum/viewtopic.php?f=21&t=13556&p=94618

[Hammerfist]

Of course I am! =) Thank you, I'll investigate that.