SQLite demonstration of native recognition of BLOB object in Listview

[GreenCan]

New release. 19 June 2013 (1.0.0.4)
 

There is a relation with this topic SQLite ListView and BLOB demo but I decided to start a completely new Topic because the approach is considerably different from the two previous examples which were only kick-offs to this demo.

This example shows how binary objects can be recognized natively in a database BLOB field without having to link to other fields that may contain information of the data object.

In the demo I used 2 approaches for native recognition

1. For multi-type binary objects, the file name is added in the header of the BLOB

Multi-type object can be images or any other kind of file.  Because of the object header data, there is no need to identify the object in the binary code

2. Objects without header data, this works only for images, an algorithm will identify the type of image.

The demo shows what happens whit objects which are not identifiable, see example 5.

             Credits to:

                1. trancexx: GIFAnimation.au3

                               '?do=embed' frameborder='0' data-embedContent>>

                2. smashly: _ImageResize()

                               Resizes and converts different graphicformats

                              

                3. rover: Customize Draw of Listview rows

                          Optimizations of WS_NOTIFY

                               I also thank rover for giving a second method to resolve the image space issue.

                               I implemented the one proposed by KaFu, because very simple to implement

                4. KaFu: Solved the Listview issue with image space in Columns one.

                              

                5. jchd: For some hints and background info on SQLite

                              

                6. Yashied: WinAPIEx.au3

                               '?do=embed' frameborder='0' data-embedContent>>

new release. Version 1.0.0.4
What's new:
- added fully generic Add, Edit, Add/Copy, Delete and Find buttons. With fully generic I mean, you don't have to bother about the table content, GUI field inputs will populate accoring to the table definition.
- Added Field validation, also according to how the columns were defined in the table. (see GUI dynamic input validation for more information)

tested a thousand times... on W7 and WXP 32/64

For a working example you have to download 2 files (see links in between the horizontal lines:

1. SQLite GreenCan_demo BLOB in Listview 1.0.0.4.zip
2. GreenCan_demo2.zip (if you already did, don't mind downloading it again)

Note:
For the Edit GUI, you will notice that sometimes one field is not editable, marked as (*PK) in the description. The field is a 'Primary Key autoincrement'. When appending the row, the PK will automatically increment, therefore it is not allowed to edit the field.
Other fields mared (*) cannot be empty, you can only save the row if these fields contain data.
In the case (*PK) is editable, you have to put unique data for the Primary key. If you do a copy/add without changing the field content, you will get a not unique Error.
Special case: I don't allow empty primary key (NULL), while SQLite does, but it's pretty useless anyhow because you can only have 1 NULL in a Primary key.

I have also included a very small non BLOB database, for example 0, to show that the generic edit/add works also here.

---

SQLite GreenCan_demo BLOB in Listview 1.0.0.4.zip

and don't forget to download this zip file to complete the required files for the demo

http://users.telenet.be/GreenCan/AutoIt/GreenCan_demo2.zip

---

 

SQLite GreenCan_demo BLOB in Listview 1.0.0.0.au3

I let you explore the demo and please give me feedback.

GreenCan

Edited June 23, 2013 by GreenCan

[jchd]

Very nice. Thanks for sharing.

[guinness]

Nice example. But (you knew it was coming!) the images don't all display in the listview and instead I get black images. I'm using Windows 7 x64 & AutoIt v3.3.8.1 x64. If you want a screenshot for confirmation let me know.

[GreenCan]

Nice example. But (you knew it was coming!) the images don't all display in the listview and instead I get black images. I'm using Windows 7 x64 & AutoIt v3.3.8.1 x64. If you want a screenshot for confirmation let me know.

Yes guinness,

please send me a screenshot because i don't know for which type you get black images. Can you verify if you have 3 bitmaps in your resources directory? I won' be

at home before Monday, so I wont be able to verify this earlier

Thanks

GreenCan

[guinness]

Using AutoIt x64 v3.3.8.1. I just ran SQLite GreenCan_demo BLOB in Listview 1.0.0.0.au3 in the ZIP file.

Edited May 17, 2013 by guinness

[jchd]

Black is beatiful

[GreenCan]

Using AutoIt x64 v3.3.8.1. I just ran SQLite GreenCan_demo BLOB in Listview 1.0.0.0.au3 in the ZIP file.

Guinness,

I tried it in almost every possible way: On XP, on Windows 7 32bit Virtual, physical, on Windows 7-x64, virtual, physical, on 6 different PCs, Compiled 32bit, compiled 64bit, non-compiled, not one single issue with not converted (Black) bitmaps.

I even ran it on a Windows server 2003 Entreprise x64E., there a very small cosmetic problem, while clicking on an icon and displaying the image, the icon gets blank, only when the row is in focus (obviously more a Listview issue than an image problem)

The only thing that I didn't do is to install the x64 version of Autoit and run it from Scite.

Obviously, while looking at the screenshot, it looks like the function _ImageResize() is the the cause of the issue on your system, and specifically _WinAPI_CreateCompatibleBitmap or _GDIPlus_EncodersGetCLSID if I got it right. 

It looks like the function cannot convert any of the graphic formats to BMP, except if it is already a BMP format (the database execute icon is a BMP file locate in the resources folder, the butterfly is a BMP format, I guess you will also see the Eye icon and the 'X' icon in the last example.

I don't see anything wrong with the function except that it could be cleaned up a bit.  I have not read anywhere that GDIPlus would have issues on x64.

Can I ask you to run the _ImageResize example on it's own (change the file names) and tell me if it converts the image correctly?  You may also want to compile my SQLite BLOB example and try it from the prompt.

expandcollapse popup

#include <GDIPlus.au3>

_ImageResize(@ScriptDir & "\__xxxxx.png", @ScriptDir & "\__xxxxx.bmp", 40, 40)

Func _ImageResize($sInImage, $sOutImage, $iW, $iH)
    Local $hWnd, $hDC, $hBMP, $hImage1, $hImage2, $hGraphic, $CLSID, $i = 0

    ;OutFile path, to use later on.
    Local $sOP = StringLeft($sOutImage, StringInStr($sOutImage, "\", 0, -1))

    ;OutFile name, to use later on.
    Local $sOF = StringMid($sOutImage, StringInStr($sOutImage, "\", 0, -1) + 1)

    ;OutFile extension , to use for the encoder later on.
    Local $Ext = StringUpper(StringMid($sOutImage, StringInStr($sOutImage, ".", 0, -1) + 1))

    ; Win api to create blank bitmap at the width and height to put your resized image on.
    $hWnd = _WinAPI_GetDesktopWindow()
    $hDC = _WinAPI_GetDC($hWnd)
    $hBMP = _WinAPI_CreateCompatibleBitmap($hDC, $iW, $iH)
    _WinAPI_ReleaseDC($hWnd, $hDC)

    ;Start GDIPlus
    _GDIPlus_Startup()

    ;Get the handle of blank bitmap you created above as an image
    $hImage1 = _GDIPlus_BitmapCreateFromHBITMAP ($hBMP)

    ;Load the image you want to resize.
    $hImage2 = _GDIPlus_ImageLoadFromFile($sInImage)

    ;Get the graphic context of the blank bitmap
    $hGraphic = _GDIPlus_ImageGetGraphicsContext ($hImage1)

    ;Draw the loaded image onto the blank bitmap at the size you want
    _GDIPLus_GraphicsDrawImageRect($hGraphic, $hImage2, 0, 0, $iW, $iW)

    ;Get the encoder of to save the resized image in the format you want.
    $CLSID = _GDIPlus_EncodersGetCLSID($Ext)

    ;Generate a number for out file that doesn't already exist, so you don't overwrite an existing image.
    Do
        $i += 1
    Until (Not FileExists($sOP & $i & "_" & $sOF))

    ;Prefix the number to the begining of the output filename
    $sOutImage = $sOP & $i & "_" & $sOF

    ;Save the new resized image.
    _GDIPlus_ImageSaveToFileEx($hImage1, $sOutImage, $CLSID)

    ;Clean up and shutdown GDIPlus.
    _GDIPlus_ImageDispose($hImage1)
    _GDIPlus_ImageDispose($hImage2)
    _GDIPlus_GraphicsDispose ($hGraphic)
    _WinAPI_DeleteObject($hBMP)
    _GDIPlus_Shutdown()
EndFunc


_GDIPlus_ShutDown()

Thanks

[GreenCan]

new release. Version 1.0.0.4
What's new:
- added fully generic Add, Edit, Add/Copy, Delete and Find buttons. With fully generic I mean, you don't have to bother about the table content, GUI field inputs will populate according to the table definition.
- Added Field validation, also according to how the columns were defined in the table. (see GUI dynamic input validation for more information)

See post 1 for download and more details

[guinness]

I just saw your message now. Do you still need me to test? I was using the 64-bit version of AutoIt, so your concerns might be correct.

[GreenCan]

I just saw your message now. Do you still need me to test? I was using the 64-bit version of AutoIt, so your concerns might be correct.

If you have an empty moment in your life, yes

Thanks

[guinness]

OK, so I download the ZIP file of the latest version and trancexx's GIF animation UDF and then ran the example. This is what I got...

 

! SQLite.au3 Error
--> Function: _SQLite_Query
--> Query: SELECT count(*) FROM Customer
--> Error: malformed database schema (IPK_Customer) - no such table: main.Customer

! SQLite.au3 Error
--> Function: _SQLite_Open
--> Error: Library used incorrectly

! SQLite.au3 Error
--> Function: _SQLite_Open
--> Error: Library used incorrectly

_SQLite_LibVersion=3.6.7
! SQLite.au3 Error
--> Function: _SQLite_Open
--> Error: Library used incorrectly

! SQLite.au3 Error
--> Function: _SQLite_Open
--> Error: Library used incorrectly

! SQLite.au3 Error
--> Function: _SQLite_Open
--> Error: Library used incorrectly

What version of SQLite do you have?

[jchd]

I post this in a rush, forgive helicopter view.

The schema is a little bizarre. Once CustomerId is declared "INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT" there is absolutely no reason to create a separate unique index: CustomerId is now an alias for rowid and PK are unique by nature.

CREATE UNIQUE INDEX [iPK_Customer] ON [Customer] ([CustomerId]); -- you can remove this: it's useless and slows things down.

Not having much time now but what's the purpose of separate int and int64 columns? SQLite int[eger]s are 1 to 64 bits internally and AutoIt ints should work in all cases, unless I'm missing something.

[GreenCan]

The schema is a little bizarre. Once CustomerId is declared "INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT" there is absolutely no reason to create a separate unique index: CustomerId is now an alias for rowid and PK are unique by nature.

CREATE UNIQUE INDEX [iPK_Customer] ON [Customer] ([CustomerId]); -- you can remove this: it's useless and slows things down.

Good point jchd.

You're right but I am lucky...

I just snooped the table from Chinook_Sqlite_AutoIncrementPKs.sqlite ( http://chinookdatabase.codeplex.com/releases/view/55681 )

I didn't care looking at the schema at all.  But you did, so my 5 for you. 

 

Not having much time now but what's the purpose of separate int and int64 columns? SQLite int[eger]s are 1 to 64 bits internally and AutoIt ints should work in all cases, unless I'm missing something.

I am not separating but joining all the INT like types to input mask Integer.  But i agree that INT64 and probably other datatypes shouldn't be there. I picked the INT64 Datatype wrongly from SQLite Expert.

I will check this further

Thanks

[GreenCan]

OK, so I download the ZIP file of the latest version and trancexx's GIF animation UDF and then ran the example. This is what I got...

What version of SQLite do you have?

sqlite3_x64.dll (3.7.16.1)

! SQLite.au3 Error

--> Function: _SQLite_Query

--> Query: SELECT count(*) FROM Customer

--> Error: malformed database schema (IPK_Customer) - no such table: main.Customer

I downloaded my zip file and checked the integrity of GreenCan_demo1.db, nothing wrong...all works fine here.

I see that you are using 3.6.7?

Are you doing the test on the same system as the one in  post 5?  If that's the case, it would be very strange indeed.

Edited June 21, 2013 by GreenCan

[jchd]

I see. About types, Expert offers a number of custom types which are mapped to the few SQLite knows about. Most of this types are for user convenience and apply display suger, e.g. SMALLMONEY which limits the amount entered, uses an automatic 2 decimals and adds the user's locale currency (on display only). You still can store whatever type you want in these columns, following SQLite rules; only display thru Expert is affected.

Personally I'm not much in favor of these "parasitic" types but I understand that Bogdan choose to implement them for user's convenience. The drawback is that you don't get the same results when e.g. you simply _ArrayDisplay the result of a select in your (AutoIt or whatever else) application. This can be sometimes more confusing than helping.

[guinness]

I will test it on the weekend as now I'm quite busy. Sorry.

[GreenCan]

Personally I'm not much in favor of these "parasitic" types but I understand that Bogdan choose to implement them for user's convenience. The drawback is that you don't get the same results when e.g. you simply _ArrayDisplay the result of a select in your (AutoIt or whatever else) application. This can be sometimes more confusing than helping.

Yes, I agree 200%

Honestly, I also find SQLite Datatypes pretty confusing.

SQLite accepts any rubbish data type without complaining and doesn't even care if data types are respected in the same column

The example below shows it

expandcollapse popup

#include <SQLite.au3>
#include <SQLite.dll.au3>

Local $hQuery, $aRow, $aNames, $sSQL
_SQLite_Startup()

_SQLite_Open(":memory:")
;~ _SQLite_Open(@ScriptDir & "\GreenCan_demo1.db", $SQLITE_OPEN_READWRITE)

_SQLite_Exec(-1, "CREATE TABLE Test_table (A_AutoInc INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, B_Integer INTEGER, C_Int64 INT64, D_Anything ANYTHING, E_Text TEXT);")
If @error Then Exit MsgBox(48,"Error",_SQLite_ErrMsg())

_SQLite_Exec(-1, "INSERT INTO Test_table(B_Integer, C_Int64, D_Anything, E_Text) VALUES ('string', 113, 114, 'row 1');")

_SQLite_Exec(-1, "INSERT INTO Test_table(B_Integer, C_Int64, D_Anything, E_Text) VALUES (222, 223.6, 'text', 5.50);")

_SQLite_Exec(-1, "INSERT INTO Test_table(B_Integer, C_Int64, D_Anything, E_Text) VALUES (252.7677, 'text', 254, 'test');")

_SQLite_Query(-1, "SELECT ROWID,* FROM Test_table ORDER BY A_AutoInc;", $hQuery)

_SQLite_FetchNames($hQuery, $aNames)
ConsoleWrite(StringFormat(" %-10s  %-10s  %-10s  %-10s  %-10s  %-10s ", $aNames[0], $aNames[1], $aNames[2], $aNames[3], $aNames[4], $aNames[5]) & @CRLF)
While _SQLite_FetchData($hQuery, $aRow, False, False) = $SQLITE_OK ; Read Out the next Row
    ConsoleWrite(StringFormat(" %-10s  %-10s  %-10s  %-10s  %-10s  %-10s ", $aRow[0], $aRow[1], $aRow[2], $aRow[3], $aRow[4], $aRow[5]) & @CRLF)
WEnd
ConsoleWrite(@CR)

$sSQL = "PRAGMA table_info(Test_table)"
_SQLite_Query(-1, $sSQL, $hQuery)
_SQLite_FetchNames($hQuery, $aNames)
ConsoleWrite(StringFormat(" %-10s  %-10s  %-10s  %-10s  %-10s  %-10s ", $aNames[0], $aNames[1], $aNames[2], $aNames[3], $aNames[4], $aNames[5]) & @CRLF)
While _SQLite_FetchData($hQuery, $aRow) = $SQLITE_OK
    ConsoleWrite(StringFormat(" %-10s  %-10s  %-10s  %-10s  %-10s  %-10s ", $aRow[0], $aRow[1], $aRow[2], $aRow[3], $aRow[4], $aRow[5]) & @CRLF)
WEnd

_SQLite_QueryFinalize($hQuery)
;~ _SQLite_Exec(-1, "DROP TABLE Test_table;")
_SQLite_Close()
_SQLite_Shutdown()

Pragma result

 

 cid         name                type                notnull     dflt_value  pk         

 0           A_AutoInc          INTEGER      1                       1          

 1           B_Integer          INTEGER      0                       0          

 2           C_Int64             INT64             0                       0          

 3           D_Anything      ANYTHING     0                       0          

 4           E_Text              TEXT                0                       0

And Data query result in SQLite table

A_AutoInc   B_Integer   C_Int64     D_Anything   E_Text     

 1                     string         113            114                 row 1      

 2                    222              223.6        text                 5.5        

 3                    252.7677   text            254                test

The approach, (as I understand) called 'general dynamic type system' leaves the responsibility to the developer to verify column data integrity. Personally, I think this is poor DB management but I also understand that SQLite being public domain doesn't need to go as far as Oracle, MSSQL, etc.

And to make it completely ridiculous...especially if you start mixing query engines:

Data query result in SQLite Expert doesn't look the same

 

A_AutoInc    B_Integer    C_Int64    D_Anything    E_Text

  1                   0                    113           114                  row 1

 2                  222                 224           text                   5.5

 3                  253                 0               254                  test

In my script I use Pragma to get the datatype. Simplified the datatype is then reduced to Text, Integer, Real input masks (and BLOB), all other uncovered datatypes are reduced to Text input masks.

That is why I have put all these exotic Datatypes in my script.  Remove Int64 would reduce the datatype to a input Text mask and result in an averse effect.

GreenCan

Edited June 22, 2013 by GreenCan

[guinness]

I now have sqlite3_x64.dll v3.7.17.0 (http://www.autoitscript.com/autoit3/files/beta/autoit/archive/sqlite/SQLite3) and I'm still getting the same error. My system is exactly the same.

[jchd]

Diverging opinion: I regard dynamic typing as one of the most useful features (= design decision) of SQLite.

It lets you enforce strict typing if so you whish but leaves options open when you find it more useful to store polymorphic data.

Remember that you can always add any kind of check constraint where you want type (or [some] data consistency) enforced.

A simple (completely made up) example:

CREATE TABLE [Checks] (
  [Name] CHAR NOT NULL COLLATE NOCASE DEFAULT '- unknown -', 
  [Date] [CHAR(10)] NOT NULL DEFAULT (date('now')), 
  [Sex] [CHAR(1)] DEFAULT 'M', 
  [Children] INT NOT NULL DEFAULT 2, 
  [ZipCode] [CHAR(5)] NOT NULL, 
  CONSTRAINT [ckName] CHECK(typeof(Name) = 'text' and length(Name) > 1 and upper(substr(Name, 1, 1)) between 'A' and 'Z'), 
  CONSTRAINT [ckDateRange] CHECK(Date like '____-__-__' and Date between '1800-01-01' and '2013-06-22'), 
  CONSTRAINT [ckSexValid] CHECK(Sex in ('M', 'F')), 
  CONSTRAINT [ckChildren] CHECK(typeof(Children) = 'integer' and Children between 0 and 25), 
  CONSTRAINT [ckZipCode] CHECK(typeof(ZipCode) = 'text' and length(ZipCode) = 5 and cast(Zipcode as int) between 01000 and 99999));

Edited June 22, 2013 by jchd

[GreenCan]

 

Diverging opinion: I regard dynamic typing as one of the most useful features (= design decision) of SQLite.

It lets you enforce strict typing if so you whish but leaves options open when you find it more useful to store polymorphic data.

Remember that you can always add any kind of check constraint where you want type (or [some] data consistency) enforced.

I get your point.  It's clear with your example.