Kobo Cover Fixer

TheSaint · May 25

Okay, after almost a year long break, I decided to have another look, and discussed that here in Chat.

And in the post after that one, I mentioned the following.

Quote

My latest notion, is to search for Python scripts that deal with sending or updating an ebook on a Kobo device ... probably something calibre related.

I'm not good with Python, dislike it even, but maybe I might find something and understand it enough ... one can hope.

My thought is to search for those '.parsed' endings of the image file names, or the text just preceding that file extension.

That has borne some fruit, though whether I can understand the Python code enough to use it, is another matter.

I found the following at 'C:\Users\TheSaint\AppData\Roaming\calibre\plugins\Kobo Utilities.zip'. In a Python file named 'action.py'.

if remove_fullsize_covers and not ending == ' - N3_FULL.parsed':

So I knew I was maybe getting somewhere, so I scrolled on to the following.

if image_id:
   cover_image_file = self.device.images_path(path, image_id)
   debug_print("_open_cover_image_directory - cover_image_file='%s'" % (cover_image_file))
   cover_dir = os.path.dirname(os.path.abspath(cover_image_file))
   debug_print("_open_cover_image_directory - cover_dir='%s'" % (cover_dir))
   if os.path.exists(cover_dir):
       open_local_file(cover_dir)
total_books += 1

Which seemed relevant but didn't mean much to me, so I scrolled further and found something better.

result = next(cursor)
                        debug_print("_test_covers - contentId='%s', imageId='%s'" % (contentID, result['ImageId']))
                        hash1 = qhash(result['ImageId'])
#                        debug_print("_test_covers - hash1='%s'" % (hash1))
                        xff   = 0xff
                        dir1  = hash1 & xff
                        dir1  &= 0xff
#                        debug_print("_test_covers - dir1='%s', xff='%s'" % (dir1, xff))
                        xff00 = 0xff00
                        dir2  = (hash1 & xff00) >> 8
#                        debug_print("_test_covers - hash1='%s', dir1='%s', dir2='%s'" % (hash1, dir1, dir2))
                        cover_dir = os.path.join(path, ".kobo-images", "%s" % dir1, "%s" % dir2)
                        debug_print("_test_covers - cover_dir='%s'" % (cover_dir))
#                        self.device.delete_images(result['ImageId'], path)
                        removed_covers +=1
                    except StopIteration:
                        debug_print("_test_covers - no match for contentId='%s'" % (contentID,))
                        not_on_device_books += 1
                    total_books += 1

That seems very relevant to me, having noted the 'dir1' and 'dir2' bits as well as 'cover_dir = os.path.join(path, ".kobo-images", "%s" % dir1, "%s" % dir2)'

Alas though, my knowledge of Python is poor, and my understanding quite minimal. But at least I seem to be in the right ballpark.

What I no doubt need now, is help from someone with a good working knowledge of Python.

I am attaching the full 'action.py' file, in the hope that someone reading all this at AutoIt, can help me.

In a nutshell and at a guess, I am after some code in AutoIt, that can get me those 'dir1' and 'dir2' values. Presumably some kind of hashing of the 'ImageId' which I imagine is taken from the metadata of a Kobo epub file. That much seems clear to me, most of the rest doesn't.

NOTE - ImageId does exist in the KoboReader.sqlite device file, one for each ebook in a column called ImageId. So perhaps it is being read from there.

Even if what is going on can just be fully explained to me, I can maybe work out the rest myself.

I'm not sure why the whole thing needs to be so convoluted, and in comparison, the Kindle approach is much more friendly.

action.py

Edited May 25 by TheSaint

TheSaint · May 25

Here's some examples of ImageId ... SIDE-LOADED and SENT BY WIFI values. Those values in RED are the ImageId, and are clearly very different when side-loaded.

EBOOK = Shadows Of Self by Brandon Sanderson
SIDE-LOADED = file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub
IMAGE DEVICE PATH = 50\121\file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub - N3_FULL.parsed
DIR 1 = 50
DIR 2 = 121

EBOOK = The Big Bad Wolf by James Patterson
SENT BY WIFI = ff0a942a-2f28-4aa2-ba97-318fce090264
IMAGE DEVICE PATH = 20\244\ff0a942a-2f28-4aa2-ba97-318fce090264 - N3_FULL.parsed
DIR 1 = 20
DIR 2 = 244

EBOOK = Wish You Well by David Baldacci
SENT BY WIFI = f941f47c-b9aa-48a9-96d8-6b6bcb85362c
IMAGE DEVICE PATH = 3\184\f941f47c-b9aa-48a9-96d8-6b6bcb85362c - N3_FULL.parsed
DIR 1 = 3
DIR 2 = 184

TheSaint · May 26

In regard to that Python script portion I posted above (two posts back).

Even it could be adjusted to run stand-alone, passing the ImageId to it, then it would be usable for me, and even more so if I was able to compile it into an EXE, something I have done before with a Python script.

It would need to export the dir1 and dir2 values though, even if just to a text file. They are the important bit.

TheDcoder · May 27

On 5/26/2024 at 12:26 AM, TheSaint said:

I am after some code in AutoIt, that can get me those 'dir1' and 'dir2' values.

Those values are derived from the hash calculated by the qhash function which takes the Image ID as the input, I asked ChatGPT to convert it into AutoIt:

Func qhash($inputstr)
    Local $instr = ""
    If IsString($inputstr) Then
        $instr = $inputstr
    Else
        Return -1
    EndIf

    Local $h = 0x00000000
    For $i = 1 To StringLen($instr)
        $h = BitShift($h, 4) + Asc(StringMid($instr, $i, 1))
        $h = BitXor($h, BitShift(BitAND($h, 0xf0000000), -23))
        $h = BitAND($h, 0x0fffffff)
    Next

    Return $h
EndFunc

Func BitShift($value, $shift)
    If $shift >= 0 Then
        Return BitShiftLeft($value, $shift)
    Else
        Return BitShiftRight($value, Abs($shift))
    EndIf
EndFunc

Func BitShiftLeft($value, $shift)
    Return $value * 2 ^ $shift
EndFunc

Func BitShiftRight($value, $shift)
    Return Int($value / 2 ^ $shift)
EndFunc

I think you can figure out the rest of the logic to obtain the exact dir1 and dir2 values, it's just BitAND and BitOR operations.

I think ChatGPT would be of great benefit to you here, you can use DDG's ChatGPT interface for free

TheSaint · May 27

3 minutes ago, TheDcoder said:

Those values are derived from the hash calculated by the qhash function which takes the Image ID as the input, I asked ChatGPT to convert it into AutoIt:

3 minutes ago, TheDcoder said:

I think ChatGPT would be of great benefit to you here, you can use DDG's ChatGPT interface for free

BIG Thanks bud, I will now check it out.

It did actually occur to me that ChatGPT might be able to convert for me, but last time I tried to use it on this PC I could not get ChatGPT working.

TheSaint · May 27

19 minutes ago, TheDcoder said:

I think you can figure out the rest of the logic to obtain the exact dir1 and dir2 values, it's just BitAND and BitOR operations.

To be perfectly honest bud, I have nearly always avoided using BitAND etc, so I am not well versed in it.

And your code gives me an error when I throw an ImageId at it.

$result = qhash("f941f47c-b9aa-48a9-96d8-6b6bcb85362c")

: ==> Badly formatted "Func" statement.:
Func BitShift($value, $shift)

P.S. Did you try your code with the three ImageId examples (in red) that I provided in a post above?

Anyway I am trying to make sense of what your code is supposed to be doing.

Edited May 27 by TheSaint

TheDcoder · May 27

5 minutes ago, TheSaint said:

P.S. Did you try your code with the three ImageId examples (in red) that I provided in a post above?

No way bud, just asked ChatGPT's latest and greatest model to do the conversion for you.

5 minutes ago, TheSaint said:

Anyway I am trying to make sense of what your code is supposed to be doing.

Do that bud, take help from GPT, one of the rare cases where it is useful

Also maybe watch a simple explainer video on how bit operations work.

TheSaint · May 27

2 hours ago, TheDcoder said:

I think ChatGPT would be of great benefit to you here, you can use DDG's ChatGPT interface for free

Actually I found that worked for me, and I have been using ChatGPT for the first time.

It has revealed some things to me, but not others.

I gave it the following Python code.

hash1 = qhash(result['ff0a942a-2f28-4aa2-ba97-318fce090264'])
xff = 0xff
dir1 = hash1 & xff
dir1 &= 0xff
xff00 = 0xff00
dir2  = (hash1 & xff00) >> 8

And it converted it to AutoIt for me.

#include <Crypt.au3>

Local $result = ; Your dictionary containing the 'ff0a942a-2f28-4aa2-ba97-318fce090264' value

Local $hash1 = _Crypt_HashData($result['ff0a942a-2f28-4aa2-ba97-318fce090264'], $CALG_SHA1) ; Hash the value using SHA1 algorithm

Local $xff = 0xff
Local $dir1 = $hash1 & $xff
$dir1 &= 0xff

Local $xff00 = 0xff00
Local $dir2 = BitShift($hash1 & $xff00, 8)

ConsoleWrite("dir1: " & $dir1 & @CRLF)
ConsoleWrite("dir2: " & $dir2 & @CRLF)

But the dictionary aspect has me stumped. I've tried using a number in case it is positional in the SQL database or something.

I may also have not provided enough code, as there is a preceding line. And as you may have noticed, I have been excluding the debug_print lines, as not needed.

result = next(cursor)
hash1 = qhash(result['ff0a942a-2f28-4aa2-ba97-318fce090264'])
xff = 0xff
dir1 = hash1 & xff
dir1 &= 0xff
xff00 = 0xff00
dir2  = (hash1 & xff00) >> 8

So I added that extra first line (see above), and that resulted in the following, which did not make things any clearer for me.

#include <Crypt.au3>

Local $result = ; Your result obtained from the cursor

Func qhash($value)
    Return _Crypt_HashData($value, $CALG_SHA1)
EndFunc

Local $hash1 = qhash($result['ff0a942a-2f28-4aa2-ba97-318fce090264'])

Local $xff = 0xff
Local $dir1 = BitAND($hash1, $xff)
$dir1 = BitAND($dir1, 0xff)

Local $xff00 = 0xff00
Local $dir2 = BitShift(BitAND($hash1, $xff00), 8)

ConsoleWrite("dir1: " & $dir1 & @CRLF)
ConsoleWrite("dir2: " & $dir2 & @CRLF)

I also asked a direct question, and got the following response.

Quote

The Kobo software organizes ebook cover images into numbered subfolders within the ".kobo" or "Kobo" directory on the device's internal storage. The numbering of these subfolders is typically determined by the Kobo software itself and follows a specific naming convention or algorithm.

While the exact algorithm used by Kobo to determine the numbering of these subfolders is not publicly documented, it is likely designed to efficiently manage and store a large number of ebook cover images while maintaining a structured organization. The numbering of the subfolders may be based on factors such as the number of ebooks on the device, the order in which ebooks were added or accessed, or other internal criteria used by the Kobo software.

If you are interested in understanding the specific logic behind how Kobo software decides on the numbered subfolders for storing ebook cover images, you may need to reach out to Kobo customer support or refer to official documentation provided by Kobo. They may be able to provide more insights into the internal workings of the software and how it manages ebook cover images within the device's storage structure.

So maybe I will ask Kobo support after all, though I'd be surprised if they gave me what I needed to know.

Edited May 27 by TheSaint

TheSaint · May 27

So I asked it to provide a formula based on results. Here is the reply.

Quote

Based on the information provided, if the value "ff0a942a-2f28-4aa2-ba97-318fce090264" resulted in subfolders "20" and "244" when processed through the formula, we can reverse engineer the formula to determine how these subfolder numbers were derived.

Given the subfolders "20" and "244":

"20" corresponds to the result of dir1 in the formula.

"244" corresponds to the result of dir2 in the formula.

Let's analyze the results:

For "20" (dir1):

The value of xff is 0xff (255 in decimal).

The bitwise AND operation between the hashed value and xff followed by another bitwise AND operation with 0xff resulted in "20".

For "244" (dir2):

The value of xff00 is 0xff00 (65280 in decimal).

The bitwise AND operation between the hashed value and xff00, followed by a right shift by 8 bits, resulted in "244".

Based on this analysis, the formula to determine the subfolder numbers "20" and "244" from the value "ff0a942a-2f28-4aa2-ba97-318fce090264" in AutoIt could be as follows:

#include <Crypt.au3>

Func qhash($value)
    Return _Crypt_HashData($value, $CALG_SHA1)
EndFunc

Local $hash1 = qhash("ff0a942a-2f28-4aa2-ba97-318fce090264")

Local $xff = 0xff
Local $dir1 = BitAND($hash1, $xff)
$dir1 = BitAND($dir1, 0xff)

Local $xff00 = 0xff00
Local $dir2 = BitShift(BitAND($hash1, $xff00), 8)

ConsoleWrite("dir1: " & $dir1 & @CRLF)
ConsoleWrite("dir2: " & $dir2 & @CRLF)

Quote

This formula should produce the subfolder numbers "20" and "244" when applied to the value "ff0a942a-2f28-4aa2-ba97-318fce090264" in AutoIt.

Well I must be missing something because I get '0' for both folders.

TheDcoder · May 27

What the f**k bud, don't let it generate code for you, just let it explain stuff that you don't understand.

At this point the code is completely screwed up, start over. Only make changes yourself and only if you understand them.

It's a big mistake to replace the custom qhash function with _Crypt_HashData, the AI just hallucinated it.

Edited May 27 by TheDcoder

TheSaint · May 27

ChatGPT had this to say when I queried that. It also said no need to swear at a machine.

Quote

The _Crypt_HashData function in AutoIt is suitable for generating hash values, similar to the qhash function in Python that you provided. Both functions are used for hashing data using cryptographic hash algorithms.

If you are experiencing discrepancies in the hash values generated by _Crypt_HashData in AutoIt compared to the qhash function in Python, it's important to ensure that the hash algorithm and encoding are consistent between the two implementations. In this case, both should be using the SHA-1 algorithm.

If you are still encountering issues with the hash values not matching between AutoIt and Python, you may want to double-check the encoding of the input data, the handling of the hashed output, and any additional processing that may be affecting the results.

You can also verify the hash values generated by both _Crypt_HashData in AutoIt and the qhash function in Python separately to ensure they produce the same hash output for the same input data. This can help identify any potential discrepancies in the hashing process between the two environments.

Just kidding about the swear bit. :muttley:

TheDcoder · May 27

35 minutes ago, TheSaint said:

ChatGPT had this to say when I queried that.

Basically every fact it said is incorrect, it's just hallucinating the details.

That's why it is dangerous, it is very convincing even when it lies, so you at-least need to know enough to know when it is lying.

According to the python code you gave me the hash function is custom-made, it is not one of the standard algorithms, certainly not something which _Crypt_HashData can handle.

There's a reason why I gave you the AutoIt version of that function

TheSaint · May 27

2 minutes ago, TheDcoder said:

There's a reason why I gave you the AutoIt version of that function

Well I did also try to use your version in place of it, but to no avail.

In later dealings it seems convinced that the hashes are stored in a dictionary arrangement, based on the use of 'result'. That could well be.

TheDcoder · May 27

14 minutes ago, TheSaint said:

In later dealings it seems convinced that the hashes are stored in a dictionary arrangement, based on the use of 'result'. That could well be.

That's pure rubbish bud, the output of the qhash function is just a plain number.

TheSaint · May 27

9 minutes ago, TheDcoder said:

That's pure rubbish bud, the output of the qhash function is just a plain number.

And how is that not the case too with a dictionary lookup?

Clearly the following two lines with 'result' are important, and there seems to me to be some kind of lookup at play.

result = next(cursor)
hash1 = qhash(result['ImageId'])

Anyway, I've had enough for one night.

Edited May 27 by TheSaint

TheDcoder · May 27

1 hour ago, TheSaint said:

and there seems to me to be some kind of lookup at play.

Bud that's just getting the image id from whatever "result" is, it is not related to the hash function. Do you understand the concept of Maps in AutoIt? This is the exact same thing.

You can swap it out with any arbitrary string and it will work, no "dictionary" required.

TheSaint · May 27

12 hours ago, TheDcoder said:

Do you understand the concept of Maps in AutoIt?

Never really looked much into them.

Anyway ....

I understand a few things a bit more now.

Your reference to qhash being a custom function, had me check the action.py script, and I see now where you got it from (near the start of the script) and then converted.

I understand some aspects of your conversion, but not all, so I am taking it on faith you got it right.

The other two things I wanted to clear up, were 'return' and 'cursor'.

Browsing through the script, searching on 'cursor', I saw some SQL executions, and it suddenly occurred to me that 'cursor' might be an SQL function, and looking that up in my SQL text book, and reading some of what I found, it seems I was right. It would appear that 'cursor' appears to be a row in a database table.

Looking at 'return' next, that then appears to be a variable storing all elements of the row.

And that the query of result['ImageId']), returns the ImageId element of that row.

So that means processing the ImageId as in qhash("ff0a942a-2f28-4aa2-ba97-318fce090264") would be right.

So using the following, I got a return of '55'.

$result = qhash("ff0a942a-2f28-4aa2-ba97-318fce090264")
MsgBox(262144, "Result", $result)

So I then used that with what ChatGPT had converted for me of the other relevant Python code.

Local $xff = 0xff
Local $dir1 = $result & $xff
$dir1 &= 0xff

Local $xff00 = 0xff00
Local $dir2 = BitShift($result & $xff00, 8)

ConsoleWrite("dir1: " & $dir1 & @CRLF)
ConsoleWrite("dir2: " & $dir2 & @CRLF)

Alas, that gave me wrong values.

dir1: 55255255
dir2: 21739

They should be '20' and '244'.

So I am about to try some variations with ChatGPT, especially as yesterday it presented a few things, and I have just tried the only one I kept a record of.

$dir1 = BitAND(Hex(StringToBinary($result)), 0xFF)
$dir2 = BitShift(BitAND(Hex(StringToBinary($result)), 0xFF00), 8)

Which gave me the following, still wrong, but at least more in the ballpark.

dir1: 207
dir2: 13

By the way, the BitShift portion (function calls) of your converted code are automatically being skipped in favor of AutoIt's regular BitShift function.

I did force it to use yours instead, by renaming your function to ShiftBit where relevant. But that resulted in huge number returns for all three values.

So that is where I am right now, and hopefully ChatGPT will enable me to succeed this time.

Anyway the following is my full code right now.

Global $dir1, $dir2, $result, $xff, $xff00

$result = qhash("ff0a942a-2f28-4aa2-ba97-318fce090264")
;MsgBox(262144, "Result", $result)
$xff = 0xff
$dir1 = $result & $xff
$dir1 &= 0xff

$xff00 = 0xff00
$dir2 = BitShift($result & $xff00, 8)

;$dir1 = BitAND(Hex(StringToBinary($result)), 0xFF)
;$dir2 = BitShift(BitAND(Hex(StringToBinary($result)), 0xFF00), 8)

ConsoleWrite("dir1: " & $dir1 & @CRLF)
ConsoleWrite("dir2: " & $dir2 & @CRLF)

Exit

Func qhash($inputstr)
    Local $i, $instr = ""
    If IsString($inputstr) Then
        $instr = $inputstr
    Else
        Return -1
    EndIf

    Local $h = 0x00000000
    For $i = 1 To StringLen($instr)
        $h = BitShift($h, 4) + Asc(StringMid($instr, $i, 1))
        $h = BitXor($h, BitShift(BitAND($h, 0xf0000000), -23))
        $h = BitAND($h, 0x0fffffff)
    Next

    Return $h
EndFunc

Something else I noticed, was another choice (Unicode) in the original Python qhash function. Any reason why that has been omitted?

Edited May 28 by TheSaint

TheSaint · May 28

Gawd, ChatGPT let me down.

It started out well enough, then it kept pretending it had modified its presented code, when it was returning identical code to the last reply.

So I then tried to lead it to the variations if gave last night, and it just got silly and repeated my own code, and at one point hard coded (assigned) the values I wanted returned.

Then just as I was going to massage it in teh direction I wanted, it gave teh following message.

Quote

You’ve reached the maximum number of messages for one day. Please continue this chat tomorrow.

Clearly while it is a new day for me, it must be including all the replies from last night as well, which were a lot more.

You'd think they would have some reasonable leeway with different time zones. I had more than a 10 hour break, in which I slept.

P.S. And gawd, what an idiot. I just discovered that I mistakenly grabbed the ShiftBit lines from my code and used them in my first example with ChatGPT today. What a horrible mistake, and I thought I took all care not to do that ... bloody brain, can't trust it.

Edited May 28 by TheSaint

TheSaint · May 28

Okay, I managed to check that a Python script would work and produce the results I want.

Testing.py

def qhash (inputstr):
    instr = ""
    if isinstance (inputstr, str):
        instr = inputstr
    elif isinstance (inputstr, unicode):
        instr = inputstr.encode ("utf8")
    else:
        return -1

    h = 0x00000000
    for i in range (0, len (instr)):
        h = (h << 4) + ord(instr[i])
        h ^= (h & 0xf0000000) >> 23
        h &= 0x0fffffff

    return h

hash1 = qhash("ff0a942a-2f28-4aa2-ba97-318fce090264")
print("_test_covers - hash1='%s'" % (hash1))
xff   = 0xff
dir1  = hash1 & xff
dir1  &= 0xff
print("_test_covers - dir1='%s', xff='%s'" % (dir1, xff))
xff00 = 0xff00
dir2  = (hash1 & xff00) >> 8
print("_test_covers - hash1='%s', dir1='%s', dir2='%s'" % (hash1, dir1, dir2))

That returns the following when run via a BAT file.

_test_covers - hash1='12448788'
_test_covers - dir1='20', xff='255'
_test_covers - hash1='12448788', dir1='20', dir2='244'
Press any key to continue . . .

Testing.bat

@echo off

python Testing.py

pause
cls
exit

Then I added a print to file statement at the end of the script.

with open("output.ini", "a") as f:
  print("[Sub Folders]", file=f)
  print(f"dir1={dir1}", file=f)
  print(f"dir2={dir2}", file=f)

And got an easily accessible result in an INI file.

[Sub Folders]
dir1=20
dir2=244

Now I just have to see if I can compile that script into an EXE, so that a Python install is not required.

Edited May 28 by TheSaint

TheSaint · May 28

I managed to compile the Python script to an EXE.

And my memory said I only needed that Testing.exe file, but clearly not, so I now have heaps of required files, all adding up to 12.2 Mb. So much for a nice small script.

Anyway ....

That done, the next thing was passing the ImageId value on the command-line.
That took a bit of research, but I got there, and was reminded just how horrible it is to program with Python.

So I have added and also replaced with the following in my Python script.

import sys

ImageId = sys.argv[1]

hash1 = qhash(ImageId)

And now before I go any further, I need to check if this has all been worth the trouble.

In other words, I need to find an ebook on my Kobo device that doesn't have any cover images assigned, grab its ImageId value via the SQL database file, and then run that through my Python script, to get the two numbered folder names. Then create those folders on my Kobo device, and then add the three correctly named cover image files to that path. Then fire up my device and see if the covers are now displaying where they should be.

If that works, then we will be cooking with gas, and all the hassle would have finally been worth it ... and I will feel back in control of my device.

And of course that means I can finally finish my Kobo Cover Fixer program, and call it a full success.

Meanwhile, I am sure that TheDcoder will come up with the AutoIt code solution, and I can ditch my Python script.
But if not, then I can certainly make do ... and always have something to tease TheDcoder about. :muttley:

Kobo Cover Fixer

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

jchd

TheDcoder

Jos

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members