Kobo Cover Fixer

[TheDcoder]

Let's not forget about https://wormhole.app for easy file sharing needs, it supports files up to 10 GB.

[TheSaint]

17 hours ago, jchd said:

Could you send me the original Kobo database if it isn't too big?

I'm not sure that is wise, as it has some personal content, the extent of which I have not investigated.

I am currently thinking of approaching someone about the number folders. Maybe the author of calibre or the author of the DeDRM plugin. They both clearly have experience with the Kobo database etc.

Here's a link I found at the MobileRead Forum. The zip file contains a database file etc. It appears to be provided by part of the Kobo development team, back in 2010. Not sure if it differs much to the current database. I will have a play with it myself. Might be a newer one available somewhere.

http://download.kobobooks.com/desktop/tools/kobo-dir-fresh-1-4.zip

Edited May 29, 2023 by TheSaint

[TheSaint]

Further tweaking of the Results window today, along with some new features, all working toward the fix and create side of things.

One thing I have noticed, which runs counter to my earlier size fix experiments, is that all images are proving to be in full color, and not the gray scale versions I had encountered. That might be due to a change by Kobo, moving toward the eventual color E-Ink devices, or the gray scale images may have just been an exception, perhaps related to the fact I was working with only side-loaded ebooks at that point.

DOWNLOAD

Kobo Cover Fixer.au3  (70 downloads)  SEE THE FIRST POST in this topic for the latest version.
27.11 kB

Edited September 11, 2023 by TheSaint

[TheDcoder]

2 hours ago, TheSaint said:

I'm not sure that is wise, as it has some personal content, the extent of which I have not investigated.

Don't worry, I'm sure @jchd will keep your erotica library a secret

Or alternatively, you could delete all but a few entries in the database and send that instead.

[TheSaint]

5 hours ago, TheDcoder said:

Don't worry, I'm sure @jchd will keep your erotica library a secret

You wish ... I'm not sex obsessed like some young man I know.  

5 hours ago, TheDcoder said:

Or alternatively, you could delete all but a few entries in the database and send that instead.

I am still pondering such, but I remain unconvinced that it is necessary, for all the investigative work needed. As I said earlier, there is only one column in  one table, out of the many tables and columns that has a number anything like what the sub-folder names are, and I have cross-referenced that with many ebooks, both the first of the two numbers and then the last, and I got no matches. So I am guessing some formula or method is being used, which wouldn't need to exist in the database. I've done some research today, which has led me to believe that these number folders are used to make browsing quicker and less of a memory (battery drain), than if all ebook cover images were in the same folder, as seemed to be the prior situation with Kobo devices.

I've inspected that database file from 2010, hoping that while the link was posted then, they would have kept it updated, because It still works today. But on opening with the SQLite viewer program, I have seen that while many of the same columns exist in the content table, probably three quarters are missing, and there are far fewer tables. That said, only the content table has appeared to be relevant anyway, and all the columns the sqlite code queries are present I think, though perhaps not the exclusions ones ... but then they probably aren't needed. There is data present for a few free ebooks from Project Gutenberg, along with a folder called images with lots of different size ebook covers. Alas, no sub-folders, so clearly this database predates the use of sub-folders.

P.S. There is also the fact, that only I have the image sub-folders and images to compare against. So a checking and editing effort I cannot see the point of.

Edited May 29, 2023 by TheSaint

[jchd]

Kobos databases seem to evolve quickly and depend on which model/firmware you run. Hard to get a clear picture of all this mess.

[TheSaint]

@jchd - Very true. For now I have reached out on GitHub on the DeDRM discussion group, to see if anyone well versed in Kobo things might reply.

---

Last night and today so far, I have done some more work on my script, made some more changes and additions to the Results window, including having a hidden column on the far right of the ListView, for the ImageID, the value of which is required to read values back from the Results.ini file more easily. In the Author and Title inputs line, I have also added an ISBN input at far left, and reduced the other two inputs to accommodate. I've also added a SETTINGS button, giving access to the Program Settings window, all of which has been coded and is working. I will likely add other elements to it as I experiment with image creation.

The Alternate Export Drive is only relevant, if like me, you have copied all of the content of your Kobo device to a folder on your PC, and are working with that. The aim with this option, is to update both locations with the same images etc (device and PC folder).

You may have also noted, that I have the Cover Images - Source element fully coded now, and some good size images are showing for some of the entries that need a fix. The images being displayed, were sourced, ironically, from those ebook entries in calibre, and calibre extracts them from the ebook (EPUB) file. So clearly good images do already exist, which for some reason the Kobo device software is not using ... or not perfectly. I've named these source image files, with the Title and then ISBN value, after manually copying each over to the Source folder.

As you can see in the ebook cover image being displayed, it is badly sized (formatted).

I don't know why, because the image file calibre extracted from the ebook file, is perfectly okay, as seen when I select that source image instead.

That particular ebook was a giveaway freebie from Tor Books and was side-loaded. Most of the other side-loaded ebooks don't have this issue, but a few do. I'm yet to see that issue with ebooks that weren't side-loaded, but downloaded via Wifi instead.

If you look carefully at the Results window, at the image number columns , you can see they have a different type of value for the ImageID, for a side-loaded ebook, which comes immediately after the two number sub-folders. I have located all my side-loaded ebooks in a folder on the root of my device drive, named 'Imports', inside an author named sub-folder. When the Kobo device starts up, it reads that folder and imports any new ebooks into its database etc. I'm not connected via Wifi when that happens.

DOWNLOAD

Kobo Cover Fixer.au3  (39 downloads)  SEE THE FIRST POST in this topic for the latest version.
39 kB

Edited September 11, 2023 by TheSaint

[Jos]

I always thought that DeDRM is a plugin for Calibre and will use the Adobe Digital Editions information (read license info) to open the EPUB at import time and remove the DRM protection making the file shareable.

Would it be a better approach to use the Calibre metadata.db sqlite database to do any of the fixes?
I normally use Calibre's option to modify the metadata and add any cover when missing before putting it on the ereader. 

Edited May 30, 2023 by Jos

[TheSaint]

3 minutes ago, Jos said:

I always thought that DeDRM is a plugin for Calibre and will use the Adobe Digital Editions information (read license info) to open the EPUB at import time and remove the DRM protection making the file shareable.

True, but they have a discussion forum which I have used before, so I thought I would give it a go.

5 minutes ago, Jos said:

Would it be a better approach to use the Calibre sqlite datatabase to do any of the fixes?

Now there's an idea. That might be helpful, and I hadn't thought of that ... thanks.  

Hopefully calibre creates the image number sub-folders, and doesn't just rely on the Kobo software to do it for the ebooks it side-loads.

The calibre forum was going to be my next port of call for help.

[Jos]

We have 2 Kobo devices and use an Docker version of Calibre-Web to serve the book to either Kobo. There is a one time setup to do.
I first update the Meta data at import time when anything is missing and then place the book on either shelf.
Then simply press sync on the Kobo to get the books. This way I haven't seen any issues with thumbs or covers.

[TheSaint]

@Jos - Thanks for that, I will give it some consideration, though at a glance it seems beyond my current skill level, never done anything server wise.

[TheSaint]

Another productive day working on my program.

I've changed a few things, including some button renaming and bugfixes for misnamed or changed inputs and columns on the Results window.

I've added a bunch of things, and the Results window might well be in its final state (look) now.

1. COPY (clipboard) button, to copy the ebook title and ISBN value to the clipboard, to make renaming source files easier.

2. REFRESH button, to reload the Cover Images from that source folder.

3. Some 'Placebo' buttons added - CREATE, REMOVE, REPLACE. These are not yet supported by code.

4. LOG button and file, to keep and show a record of processes that have occurred.

5. MISSING IMAGES button and file, to display a list of ebooks where there are no images. This includes extra code to create that list.

6. SETTINGS button has a different icon, that looks better to me.

7. 'Cover Images - Source' now shows a file count.

NOTE - I have come across an issue, which can be seen in the selected entry in the above screenshot, which I need to figure out something for. For some reason the Title is split into two parts, with the second part existing in the 'Author' record and thus column. That entry is a Various authors entry, but you can see two others in the screenshot, that have a single author (which should be J. Robert Kennedy). There are several other such entries. Things can never be simple, can they ... though this could be as simple as an extra pipe character, as you can see the ISBN field entry has also been impacted. EDIT - No it wasn't a pipe, but rather a case of not using a pipe when I should have, as a ' - ' was being used as separator, and thus caused the issue. The entries no longer appear as having missing images now. I normally use pipes, and don't know why I didn't in this instance ... had a dumb moment I guess.  

Behind the scenes, along with creating and populating the 'Missing.txt' file for the MISSING IMAGES button, the program is now populating the 'Record.ini' file, which just contains the list of ebooks that have at least one image in the device folders. Entries that are saved to the 'Results.ini' file are now written with IniWriteSection, instead of individually, like for the 'Record.ini' file. While I have used that function many times before over the years, I believe this is the first time I have built a section via code ... less 'writes' likely mean a speed improvement, and it appeared that way to me, though there was also the new addition of populating the 'Record.ini' file.

I've yet to add code to the 'Missing Images' buttons or the FIX COVER IMAGE SIZE and BACKUP IMAGES button ... those and the 'Placebo' buttons. Aside from them, everything else is working.

I should be working on the FIX COVER IMAGE SIZE and ADD button processes next. They should be simple enough to do, as all the required information is available and recorded. Then I will likely move onto the 'Placebo' button processes, if information about how the image number sub-folders isn't forthcoming.

The BACKUP IMAGES button will also likely be coded for before the FIX button gets used (tested). At the moment, I am imagining a 'BACKUPS' PC folder with author name sub-folders, where existing images on the Kobo device are copied to. These could then be used for a simple restore if they ever go missing from the Kobo device.

I've also gone through all the entries and then copied all the needed source images from calibre ebook folders, and renamed them using the COPY button ... the title split issue ones aside, which I've skipped for now.

EDIT
All good now.

DOWNLOAD

Kobo Cover Fixer.au3  (79 downloads)  SEE THE FIRST POST in this topic for the latest version.
44.02 kB

Edited September 11, 2023 by TheSaint

[TheSaint]

Lots of work on my program again today, though half of that at least, would have been me investigating GDIPlus commands and usage, and I managed to finally scrape a few of them together and get a good result.

I worked on the BACKUP IMAGES code first, completed that and did a full backup of device images to a PC folder, a sub-folder of 'Ebook Covers', named 'Backups', inside single author named sub-folders.

Then I worked on code that checks for the USB connected Kobo device, which resulted in me adding a couple more elements to the Results window, below the 'Cover Images - Source' list, which you can see in the above screenshot was the FIND KOBO DEVICE button and Input field.

Then I started work on the code for the ADD button, culminating in some research at the AutoIt Forum for GDIPlus commands that would let me alter the DPI if needed and the JPG quality. I'm not using the latest AutoIt (couple of point versions behind), so I don't know if something like the old _GDIPlus_BitmapSetResolution command exists again. No idea why it was removed or what it might have been replaced with? Anyway, I needed to make a simple 'GDIP.au3' Include file just for that function, which I had discovered in my searches.

I ran out of time to complete the ADD button code today, but it currently creates a 'Test,jpg' file in the script directory, for the first of the three image files (largest one). The DPI (300 x 300) and JPG Quality (60%) is being set. At this point, until I discover otherwise, I am working on the notion that file size, rather than image dimensions, is the most important consideration, along with probably DPI resolution. From what I have seen with device folders that have all three image files, the exact dimensions while roughly similar (large, medium, small) are all over the place. And as some of my source files are quite large, storage space used up on the Kobo device then becomes quite important.

You see something like the following when you click the ADD button.

DOWNLOAD

Kobo Cover Fixer.au3  (54 downloads)  SEE THE FIRST POST in this topic for the latest version.
62.4 kB

GDIP.au3

If anyone can improve upon the following GDIPlus code and share, that would be great and much appreciated.

; BIG THANKS for bits of the following code to UEZ, funkey & Synapsee
; https://www.autoitscript.com/forum/topic/120163-set-pixels-per-inch-using-gdi/?do=findComment&comment=834906
; https://www.autoitscript.com/forum/topic/164388-changing-the-dpi-of-a-bmp/?do=findComment&comment=1198966
; https://www.autoitscript.com/forum/topic/183492-problem-when-manipulating-images-with-different-dpi-using-gdiplus/?do=findComment&comment=1317885
Local $CLSID, $hBitmap, $hBMP, $pData, $result, $savfle, $tData, $tGUID, $tParams
$savfle = @ScriptDir & "\Test.jpg"
_GDIPlus_Startup()
$hBitmap = _GDIPlus_BitmapCreateFromFile($covimg)
$hBMP = _GDIPlus_BitmapCreateHBITMAPFromBitmap($hBitmap)
_GDIPlus_BitmapDispose($hBitmap)
$hBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hBMP)
_WinAPI_DeleteObject($hBMP)
$result = _GDIPlus_BitmapSetResolution($hBitmap, 300, 300)
$CLSID = _GDIPlus_EncodersGetCLSID("JPG")
$tGUID = _WinAPI_GUIDFromString($CLSID)
$tParams = _GDIPlus_ParamInit(1)
$tData = DllStructCreate("int Quality")
DllStructSetData($tData, "Quality", 60)
$pData = DllStructGetPtr($tData)
_GDIPlus_ParamAdd($tParams, $GDIP_EPGQUALITY, 1, $GDIP_EPTLONG, $pData)
_GDIPlus_ImageSaveToFileEx($hBitmap, $savfle, $CLSID, $tParams)
_GDIPlus_BitmapDispose($hBitmap)
_GDIPlus_Shutdown()

 

Edited September 11, 2023 by TheSaint

[TheSaint]

Spent more time today, playing around with GDIPlus, working out the code to do image resizing for when it is required etc.

I also improved some of my code and turned some into functions, including for the GDIPlus code used by the ADD button.

When you select an ebook entry in the ListView, it now also selects the source image in the List field if the image and entry exist. This would only be for those images that I need to add or fix.

I've added some percentage stuff to the Settings window, which is used by the GDIplus code.

GDIPlus code is working well, though I am only exporting a 'Test.parsed' (JPG) file right now, with the final code currently disabled while I am still testing and adding missing elements. I still have to cater for the Alternate Drive element, checking if it is required and where to send a second copy of the resulting image file, plus updating the blank entries in the ListView and in the associated files.

Once that is all done, I can then test with my Kobo device.

Then I will work on the code for fixing wrongly formatted images, where the actual cover is just a small portion of the image, usually placed on a bigger black image. This will be not much more than doing a simple file replacement, using the GDIplus code I have already created.

Then I will move onto doing the placebo code and images ... more GDIplus stuff, involving identity text on the resulting images that will be copied to the empty sub-folders.

So we are slowly getting there ... the end is almost in sight.

I don't yet know whether I will need to (update) write anything to the KoboReader.sqlite file. It depends I guess on whether my Kobo device will auto add the new images to that database file or not.

DOWNLOAD

Kobo Cover Fixer.au3  (47 downloads)  SEE THE FIRST POST in this topic for the latest version.
75.76 kB

Edited September 11, 2023 by TheSaint

[TheSaint]

Today I finished the ADD button code, by adding the Alternate Export Drive code, and implemented  the writing of processes and results to the Log file. I then cloned all that for the FIX button, modifying it to suit. So both ADD and FIX are now finished, while temporarily set to Testing mode with a few bits of their code commented out.

Here's some screenshots.

   

 

      

I will work on the placebo images code next. Once that is done, then I will actually test all the processes with my Kobo device.

DOWNLOAD

Kobo Cover Fixer.au3  (47 downloads)  SEE THE FIRST POST in this topic for the latest version.
88.53 kB

Edited September 11, 2023 by TheSaint

[TheSaint]

Started work on the Placebo code today, and completed the first phase. I also updated or changed a bunch of things and added some new elements to the Results window.

1. Added a SUBS button and input, between Title and Author.

2. Two of the three Placebo buttons were renamed. CREATE became BUILD, REPLACE became RELOCATE and swapped positions with the REMOVE button.

3. Kobo device path input was shortened, so that the LOG button, which was reduced in size, could also sit beneath the 'Cover Images' list, allowing for enough space to add a 'View Placebo.txt' file button where the LOG button was.

4. The BACKUP IMAGES button was reduced in width and made multi-line, so that there was enough space to add a RELOAD button on its right.

5. When an entry on the 'Ebooks Check' list has an image fixed, the row entry line gets colored LIME (see screenshot above).

6. When an entry on the 'Ebooks Check' list has an image added, the row entry line gets colored RED. The appropriate column(s) now also get the blank entry(s) populated.

The first phase for Placebo Images, using the BUILD button, is creating number folders within a 'Placebo' folder within the 'Ebook Covers' folder. In my situation, that meant 74 number folders, named as '1' to '74'. That matches the number of empty image folders on my Kobo device. The second phase (not yet coded) will be relocating the created placebo images to those 74 folders on my device.

During the BUILD (first phase), placebo image files are created in those 74 folders. In my scenario, that means 40 images per folder, to match the number of ebook entries on the list that don't have images. These images are black bold text on a white background. All up that meant 2,960 images.

   

The notion at this point, because I don't know what sub-folders belong to the ebooks with no images, is to discover that via browsing my Kobo device, when the placebo images are on it. With the result of that and matching up with the records created, I should then be able to enter the correct sub-folder names using the SUBS button and input, for the selected list entry with no images. That sub-folder entry is written to the 'Results.ini' file, which the CREATE button (not yet coded) can use to add the correct created images to the sub-folders ... that's the theory anyway.

NOTE - It has occurred to me, that I might be able to make up sub-folders or use existing empty ones, and just edit the 'KoboReader.sqlite' file to suit. I will certainly attempt that, if the current placebo method fails. It will only fail I guess, if none of the empty image folders match up with ebook entries missing images. There may also be a requirement of the missing image entries in the 'KoboReader.sqlite' file to be populated. I guess it all depends on the Kobo device capability and how strict things are.

DOWNLOAD

Kobo Cover Fixer.au3  (45 downloads)  SEE THE FIRST POST in this topic for the latest version.
94.02 kB

Edited September 11, 2023 by TheSaint

[TheDcoder]

Uhh... I don't think that's a placebo, it's a placeholder

[TheSaint]

15 hours ago, TheDcoder said:

Uhh... I don't think that's a placebo, it's a placeholder

Wrong, it is not a placeholder at all, it is a file pretending to be the correct one, so a placebo file.  

A placeholder, is something you keep in reserve for something.

Edited June 5, 2023 by TheSaint

[TheDcoder]

14 minutes ago, TheSaint said:

it is a file pretending to be the correct one, so a placebo file.  

Okay, so it's an imposter

[TheSaint]

I have mostly finished my program now, some possible tweaks aside as I test it further, and no writing to the copy or device 'KoboReader.sqlite' files yet.

As you might notice, I have changed and added some more elements to the above Results window.

1. LOG button was relocated yet again, and resized to suit. This was because I decided I needed some folder open buttons.

2. FIX COVER IMAGE SIZE was renamed to FIX COVERS, and reduced in size, to allow for the other additions etc.

3. Added a button to open the device images folder, and another button to do the same with any Alternate Drive selected. These two small buttons are to the left of the Cover Image, at the bottom of the window.

That RED colored line (row), was for the first column image being created and added ... file name with subs also inserted into that column.

I've completed the code for the RELOCATE, REMOVE and CREATE buttons, including support for the Alternate Drive. This means all controls are now coded for.

The CREATE button can only work if you have manually assigned sub-folders for the selected entry.

I realized just after I went to bed last night, that I forgot to disable .jpg which I was using for visual testing with the GDIPlus code that created the placebo images. So that means yesterday's script upload has .jpg instead of .parsed enabled for placebo file creation. I fixed that today, and I also needed to rename all my created placebo .jpg files to .parsed, so I wrote a tiny bit of code to do that, first thing.

I've tested the RELOCATE and REMOVE buttons code. Though I have done no testing with my Kobo device yet, just with the cloned folders setup. Full testing of everything with my Kobo device comes next, but maybe not today, as I have other things to concern myself with now, and I want the time etc to do it all properly.

The following script has everything enabled, even though some of it as I said, remains untested at this point. That means the Test.jpg or Test.parsed files are not being used anymore.

DOWNLOAD

Kobo Cover Fixer.au3  (44 downloads)  SEE THE FIRST POST in this topic for the latest version.
105.56 kB

NOTE - I've only just realized now, after writing that comment about the RED line, that I should have assigned a different color when the CREATE button is used ... probably FUCHSIA ... just to differentiate what happened with what. RED = ADDED, LIME = FIXED.

P.S. I also keep forgetting to remove the checkboxes from the ListView in column one. I did have some vague notion of maybe using them when I started.

Edited September 11, 2023 by TheSaint