Alternative data compression

[trancexx]

One of the main issues AutoIt coder could have is lack of built-in data compression mechanism. Doing it "manually" is not an option because of the slowness of the interpretter.

So, it's either using third party dlls (whatever) or some Windows API that exposes the funcionality.

In this post I will compress data using well known GDI+. That's graphical library, so it's not straightforward approach. First I'm making BITMAP out of data and then convert that image into a PNG.

PNG format includes form of compression, so at the end what's get is compressed data. Compression level is not very high compared to other methods, but considering everything - who cares .

Of course that compressing string "ABC" will not make much sense and will result in, IDK 150 bytes of output because PNG includes metadata that take space, but for data in sizes of more than few KB you will see the difference.

Besides you will see how your data looks if it's taken to be a picture.

Script will create PNG out of itself called Test_Image.png and then read it and print to console:

GDIP_Compress.au3

[guinness]

That's pretty clever trancexx.

[Melba23]

trancexx,

About 50% compression when I tested - clever.

M23

[UEZ]

That's a brilliant idea to use PNG compression algorithm for any kind of binary data.

Cool.

Br,

UEZ

Edited February 24, 2013 by UEZ

[UEZ]

Can you explain me the idea of the rotation of the image before I try analysing and understand it? I know it works only with value 6 (A 180-degree rotation followed by a horizontal flip (No rotation and a vertical flip))...

Further I can see in many progams that there is an option to set the compression rate also for PNG....

Thanks,

UEZ

Edited February 24, 2013 by UEZ

[JohnOne]

Do you think that if monkey programmers wrote standard code, compressed it to png, converted to jpg, reconverted to png, then extracted code, for long enough they would come up with a masterpiece?

[trancexx]

Can you explain me the idea of the rotation of the image before I try analysing and understand it? I know it works only with value 6 (A 180-degree rotation followed by a horizontal flip (No rotation and a vertical flip))...

Further I can see in many progams that there is an option to set the compression rate also for PNG....

Thanks,

UEZ

Yes, of course I can explain flipping.

If you look at BMP data line by line you will see that first line of data is drawn at the bottom of he pic, and last line is drawn at the top. For example try compressing this data:

$sData = "0x" & _
"000000" & "000000" & "000000" & "000000" & _
"000000" & "000000" & "000000" & "000000" & _
"000000" & "000000" & "000000" & "000000" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF"

... I've organized it so that every line is exactly a line inside BMP and every chunk is one pixel represented by RGB value. Considering visually RGB out of (0x000000) is black color and 0xFFFFFF is white one could expect that the image will be black up and white down. Something like (every word is one pixel):

BLACK BLACK BLACK BLACK
BLACK BLACK BLACK BLACK
BLACK BLACK BLACK BLACK
WHITE  WHITE  WHITE  WHITE 
WHITE  WHITE  WHITE  WHITE 
WHITE  WHITE  WHITE  WHITE

But if you look at the picture you will see it's other way around - what's up that's down and vice versa.

First line is black-ish and that's the padding I was commenting on in the code. It holds the original data size information too. Ignore it here.

GdipCreateBitmapFromScan0 creates top to down image and if I want correct data I must flip it to be down to top otherwise it's all mixed up.

I didn't try setting PNG compression level, I know about JPG, but if you know how's that done post it.

[UEZ]

Ah yes, I can remember it again that the bitmap data is set from bottom to top - now it makes sense.

Regarding compression settings - I don't now yet how to set any compression value for PNG but e.g. in IrfanView you can set it (0-9). My assumption was that it must be possible somehow. Maybe it uses any 3rd party dll for that.

Br,

UEZ

[KaFu]

PNG compression does not seem to be possible by GDI+.

http://stackoverflow.com/questions/4418454/c-seeking-png-compression-algorithm-library

[JohnOne]

It seems pretty cool that you could have the code to your compiled script stored in its own icon, or background image or similar.

[Andreik]

PNG compression does not seem to be possible by GDI+.

http://stackoverflow.com/questions/4418454/c-seeking-png-compression-algorithm-library

According to MSDN:

GDI+ has built-in encoders and decoders that support the following file types:

• BMP
• GIF
• JPEG
• PNG
• TIFF

and this is easy to show it in trancexx example. I wrote on console all supported encoders using GdipGetImageEncoders and I get this:

*.BMP;*.DIB;*.RLE

*.JPG;*.JPEG;*.JPE;*.JFIF

*.GIF

*.TIF;*.TIFF

*.PNG

[UEZ]

@Andreik: of course PNG is supported but it has only a default setting for PNG compression whereas you can set compression strength for JPG in GDI+ but not for PNG format.

Br,

UEZ

[nullschritt]

Would it be possible to process the data directly from variables instead of saving then loading a file? This would be much faster for multiple compression operations in a row.

[trancexx]

Would it be possible to process the data directly from variables instead of saving then loading a file? This would be much faster for multiple compression operations in a row.

Yes it would. Honestly, I didn't see much sense in that, that's why it's not there.

But I guess it could be useful if data is exchanged between entities, over TCP connection for example. In that case if you want to compress and decompress data this way, new function could be written that would load "Image" from the variable holding compressed data. That function could be something like this:

; _GDIPlus_Startup() must be called before usage
Func GdipImageLoadFromVariable($vImage)
    Local Const $sIID_IStream = "{0000000C-0000-0000-C000-000000000046}"
    Local Const $tagIStreamReadWrite = "Read hresult(struct*;dword;dword*); Write hresult(struct*;dword;dword*);"
    ; Create stream object
    Local $oStream = ObjCreateInterface(CreateStreamOnHGlobal(), $sIID_IStream, $tagIStreamReadWrite)
    ; Check for errors
    If @error Then Return SetError(1, 0, 0)
    ; Allocate space for binary
    Local $tBinary = DllStructCreate("byte[" & BinaryLen($vImage) & "]")
    DllStructSetData($tBinary, 1, $vImage)
    ; Write to stream
    Local $iWritten
    $oStream.Write($tBinary, DllStructGetSize($tBinary), $iWritten)
    ; Load Image from stream
    Local $hImage = GdipLoadImageFromStream($oStream())
    If @error Then Return SetError(2, 0, 0)
    Return $hImage
EndFunc

Func GdipLoadImageFromStream($pStream)
    Local $aCall = DllCall($ghGDIPDll, "dword", "GdipLoadImageFromStream", _
            "ptr", $pStream, _
            "ptr*", 0)
    If @error Or $aCall[0] Then Return SetError(1, 0, 0)
    Return $aCall[2]
EndFunc

Edited April 4, 2013 by trancexx

[nullschritt]

@trancexx yes, the idea was for transport over sockets, thanks for the example!

After testing the addition of this compression method, the data was actually bloated, and the send rate slower, as compared to using just LZMA alone. I was able to achieve faster data processing, AND faster data transfer by using LZMA + QuickLZ at different points, but eventually they began to overlap and corrupt data, I was looking for an alternative secondary compression algorithm, but this just seems a bit too bulky for the size of data I'm transfer, but perhaps it could be more useful on operation or transfers involving larger amounts of data?

Either way, nice code!

Edit: Also would there be a way to make this have more width and less length, I think this would be a neat way to print out data, bit it would be very difficult to reconstruct an image that is so tall and thin.

Edited April 5, 2013 by nullschritt

[mesale0077]

create png okey

but

create gif

SaveAsPic($sData, $sPic,"gif")

ConsoleWrite(BinaryToString(ReadFromPic($sPic)) & @CRLF) dont work

why?

Edited March 29, 2014 by mesale0077

[trancexx]

Because GIF format allows up to 8bpp and that is a problem because suitable initial BMP should be made to match the scheme.

To understand the issue fully you should really learn more about these formats. Don't because only freaks do that. Freaks and me.

[TommyDDR]

Thank you for sharing this script, can you tell me if the compression GDI limits or if it is me who is doing it wrong.
Here is a reduced easier to show you my problem script:

#include <ScreenCapture.au3>
#include "_Compress.au3"
#include "Bitmap_String.au3"

While(True)
    Local $hBmb = _ScreenCapture_Capture("")
    Local $img = _GDIPlus_BitmapCreateFromHBITMAP($hBmb)
    Local $binString = _Bitmap2BinaryString($img, 100)
    Local $imgCompress = Compress($binString)
    ConsoleWrite(BinaryLen($binString) & " -> " & BinaryLen($imgCompress) & @LF)
    _WinAPI_DeleteObject($hBmb)
    _GDIPlus_ImageDispose($img)
    Sleep(10)
WEnd

In running, we can see that the compression returns an empty string from time to time.

 

I see this problem only appears with certain images, here's a script and a clear picture that reproduces the bug:

#include "_Compress.au3"

Local $binString = FileRead(@ScriptDir & "\a.png.txt")
Local $imgCompress = Compress($binString)
ConsoleWrite(BinaryLen($binString) & " -> " & BinaryLen($imgCompress) & @LF)

In "GdipImageSaveToVariable", GdipSaveImageToStream return 7 -> "Win32Error"

 

Would you have any idea why?

 

Thank you.

_Compress.au3

Bitmap_String.au3

a.png.txt

Edited July 2, 2014 by TommyDDR

[Adele]

Good job but I can't run it. Please update for AutoIt 3.3.12.0.

Edited January 30, 2015 by Adele

[mesale0077]

$ghGDIPDll  to $__g_hGDIPDll change please