A simple help request

[TheSaint]

I'm not quite running latest AutoIt (AutoIt v3.3.15.0+), but before I bother to upgrade to it, I'd like to know if it will solve my issue.

The issue is related to the following code and its return value.

Local $inputstr = "file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub"
Local $hash = qhash($inputstr)
ConsoleWrite("Hash: " & $hash & @CRLF)

Func qhash($inputstr)
    If Not IsString($inputstr) Then Return -1

    ; Convert input string to UTF-8 encoded bytes using StringToBinary
    Local $utf8Bytes = StringToBinary($inputstr, 4) ; 4 = UTF-8 encoding
    Local $length = BinaryLen($utf8Bytes)

    Local $h = 0x00000000
    Local $i, $byte, $byteVal, $temp

    ; Process each byte in the UTF-8 byte sequence
    For $i = 1 To $length
        $byte = BinaryMid($utf8Bytes, $i, 1)  ; Get each byte
        $byteVal = Dec(Hex($byte))             ; Convert byte to integer value

        ; Apply hash logic
        $h = BitShift($h, -4) + $byteVal
        $temp = BitAND($h, 0xF0000000)
        $h = BitXOR($h, BitShift($temp, 23))
        $h = BitAND($h, 0x0FFFFFFF)
    Next

    Return $h
EndFunc

According to ChatGPT, I need to upgrade to latest AutoIt, for the following reason.

Quote

The StringToBinary($string, 4) call — where the second parameter specifies UTF-8 encoding — was introduced in AutoIt 3.3.15.0 (released after your version).

In AutoIt 3.3.14.2, StringToBinary() ignores the second parameter, so it converts the string using the default internal UCS-2 encoding — not UTF-8, which is what Python uses.

That means all the logic using BinaryMid, Hex, and Dec is working on the wrong byte values — and that's why you're still getting the wrong value.

I've converted some Python code to AutoIt, but this appears to be the stumbling block.

So if someone using latest AutoIt, could run my script and report the return value, that would be great.

ChatGPT tried to come up with a suitable work-around but that failed. It tried to manually write a UTF-8 encoder in AutoIt, but said it is likely complex and error-prone.

P.S. It would be even better, if someone could provided a reliable workaround, that meant I did not have to upgrade.  

Edited Saturday at 11:40 AM by TheSaint

[ioa747]

When I run it, the console says:
Hash: 30822386

[TheSaint]

13 minutes ago, ioa747 said:

When I run it, the console says:
Hash: 30822386

Yep, I get the same with my version of AutoIt, so it looks like ChatGPT once again is talking out of its posterior.

THANKS.

[ioa747]

Below is a workaround that manually encodes the string to UTF-8 and then processes it similarly to your original function.

Here's a modified version of your qhash function that manually converts the string to UTF-8:

expandcollapse popup

Local $inputstr = "file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub"
Local $hash = qhash($inputstr)
ConsoleWrite("Hash: " & $hash & @CRLF)

Func qhash($inputstr)
    If Not IsString($inputstr) Then Return -1

    ; Convert input string to UTF-8 encoded bytes
    Local $utf8Bytes = StringToUTF8($inputstr)
    Local $length = StringLen($utf8Bytes)

    Local $h = 0x00000000
    Local $i, $byte, $byteVal, $temp

    ; Process each byte in the UTF-8 byte sequence
    For $i = 1 To $length
        $byte = Asc(StringMid($utf8Bytes, $i, 1)) ; Get each byte as an integer
        $byteVal = $byte                     ; Use the byte value directly

        ; Apply hash logic
        $h = BitShift($h, -4) + $byteVal
        $temp = BitAND($h, 0xF0000000)
        $h = BitXOR($h, BitShift($temp, 23))
        $h = BitAND($h, 0x0FFFFFFF)
    Next

    Return $h
EndFunc

Func StringToUTF8($str)
    Local $utf8 = ""
    For $i = 1 To StringLen($str)
        Local $char = AscW(StringMid($str, $i, 1)) ; Use AscW for Unicode characters
        If $char < 128 Then
            $utf8 &= Chr($char)
        ElseIf $char < 2048 Then
            $utf8 &= Chr(192 + BitShift($char, 6)) & Chr(128 + BitAND($char, 63))
        ElseIf $char < 65536 Then
            $utf8 &= Chr(224 + BitShift($char, 12)) & Chr(128 + BitAND(BitShift($char, 6), 63)) & Chr(128 + BitAND($char, 63))
        ElseIf $char < 1114112 Then
            $utf8 &= Chr(240 + BitShift($char, 18)) & Chr(128 + BitAND(BitShift($char, 12), 63)) & Chr(128 + BitAND(BitShift($char, 6), 63)) & Chr(128 + BitAND($char, 63))
        EndIf
    Next
    Return $utf8
EndFunc

 

[TheSaint]

Okay, thanks I will now try that.

But meanwhile ChatGPT has discovered that my Python code is for Python 2 not 3, and that is also part of the problem.

Here's its revised code for latest AutoIt.

; Test
Local $inputstr = "file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub"
ConsoleWrite("Hash: " & qhash($inputstr) & @CRLF)

Func qhash($inputstr)
    If Not IsString($inputstr) Then Return -1

    Local $utf8Bytes = StringToBinary($inputstr, 4) ; 4 = UTF-8
    Local $length = BinaryLen($utf8Bytes)

    Local $h = 0x00000000
    Local $i, $byte, $val, $temp

    For $i = 1 To $length
        $byte = BinaryMid($utf8Bytes, $i, 1)
        $val = Dec(Hex($byte))

        $h = BitShift($h, -4) + $val
        $h = BitXOR($h, BitShift(BitAND($h, 0xF0000000), 23))
        $h = BitAND($h, 0x0FFFFFFF)
    Next

    Return $h
EndFunc

Your code returned the same wrong value of 30822386

[ioa747]

Here's with revised code
Hash: 30822386

[Nine]

Since your input string contains only single byte characters in UTF-8, I don't think the problem is the conversion.

As you can see :

ConsoleWrite(@AutoItVersion & @CRLF)
Local $inputstr = "file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub"
ConsoleWrite(Binary($inputstr) & @CRLF)
consoleWrite(StringToBinary($inputstr, 4) & @CRLF)

Provide both the same binary :

3.3.16.1
0x66696C655F5F5F5F6D6E745F6F6E626F6172645F496D706F7274735F4272616E646F6E5F53616E646572736F6E5F536861646F77735F4F665F53656C665F2D5F4272616E646F6E5F53616E646572736F6E5F65707562
0x66696C655F5F5F5F6D6E745F6F6E626F6172645F496D706F7274735F4272616E646F6E5F53616E646572736F6E5F536861646F77735F4F665F53656C665F2D5F4272616E646F6E5F53616E646572736F6E5F65707562

 

[TheSaint]

Yeah, I'm not sure where the issue lies.

Here is the original Python 2 (seemingly) code.

def qhash (inputstr):
    instr = ""
    if isinstance (inputstr, str):
        instr = inputstr
    elif isinstance (inputstr, unicode):
        instr = inputstr.encode ("utf8")
    else:
        return -1

    h = 0x00000000
    for i in range (0, len (instr)):
        h = (h << 4) + ord(instr[i])
        h ^= (h & 0xf0000000) >> 23
        h &= 0x0fffffff

    return h

As you can see in the range loop, there are aspects that AutoIt doesn't really have an equivalent for out of the box.

[TheSaint]

Okay, I don't believe it, ChatGPT actually solved it. First time ever for me.

expandcollapse popup

Local $inputstr = "file____mnt_onboard_Imports_Brandon_Sanderson_Shadows_Of_Self_-_Brandon_Sanderson_epub"
ConsoleWrite("Final Hash: " & qhash($inputstr) & @CRLF)

Func StringToUTF8($str)
    Local $utf8 = ""
    For $i = 1 To StringLen($str)
        Local $c = AscW(StringMid($str, $i, 1))
        If $c <= 0x7F Then
            $utf8 &= Chr($c)
        ElseIf $c <= 0x7FF Then
            $utf8 &= Chr(0xC0 + BitShift($c, -6))
            $utf8 &= Chr(0x80 + BitAND($c, 0x3F))
        Else
            $utf8 &= Chr(0xE0 + BitShift($c, -12))
            $utf8 &= Chr(0x80 + BitAND(BitShift($c, -6), 0x3F))
            $utf8 &= Chr(0x80 + BitAND($c, 0x3F))
        EndIf
    Next
    Return $utf8
EndFunc

Func UnsignedRightShift($value, $bits)
    If $bits >= 32 Then Return 0
    If $value < 0 Then $value += 0x100000000
    Return Int($value / (2 ^ $bits))
EndFunc

Func qhash($inputstr)
    If Not IsString($inputstr) Then Return -1

    Local $utf8 = StringToUTF8($inputstr)
    Local $h = 0x00000000
    Local $temp, $val

    For $i = 1 To StringLen($utf8)
        $val = Asc(StringMid($utf8, $i, 1))

        $h = Mod($h * 16 + $val, 0x100000000) ; unsigned 32-bit left shift
        $temp = BitAND($h, 0xF0000000)
        $h = BitXOR($h, UnsignedRightShift($temp, 23))
        $h = BitAND($h, 0x0FFFFFFF)
    Next

    Return $h
EndFunc

There was also an issue with the following Python line.

h ^= (h & 0xf0000000) >> 23

What ChatGPT had to say.

Quote

YES! That’s what I love to hear!

After all the deep dives, false leads, and careful debugging — we finally nailed it. That rogue signed right-shift was the real villain the whole time. 😅

So to recap:

✅ What Made It Work

• Manual UTF-8 encoding (for full compatibility)

• 32-bit-safe unsigned math via Mod()

• Proper unsigned right-shift function (>> 23)

• Careful bit masking throughout

🧠 And Now You’ve Got:

• A fully functional, AutoIt-compatible qhash() that matches Python exactly

• No dependency on newer AutoIt versions

• A nice forensic trail of how to handle bitwise hashing properly in a language with signed integer quirks

 

[TheDcoder]

Hey bud, I am getting a deja vu by looking at this code, didn't we already discuss it earlier?

Anyway, good to see ChatGPT help you solve it.

[argumentum]

Func qhash($inputstr, $iSeed = 0x00000000)
    If Not IsString($inputstr) Then Return -1
    Local $utf8 = StringToUTF8($inputstr)
    Local $temp, $val
    For $i = 1 To StringLen($utf8)
        $val = Asc(StringMid($utf8, $i, 1))
        $iSeed = Mod($iSeed * 16 + $val, 0x100000000) ; unsigned 32-bit left shift
        $temp = BitAND($iSeed, 0xF0000000)
        $iSeed = BitXOR($iSeed, UnsignedRightShift($temp, 23))
        $iSeed = BitAND($iSeed, 0x0FFFFFFF)
    Next
    Return $iSeed
EndFunc

...runned the code via "Wetware-GPT" and claims that it should be this way. I don't know what's the big difference.

[TheDcoder]

16 minutes ago, argumentum said:

Wetware-GPT

What in tarnation is that?

[TheSaint]

4 hours ago, TheDcoder said:

Hey bud, I am getting a deja vu by looking at this code, didn't we already discuss it earlier?

Anyway, good to see ChatGPT help you solve it.

You are right.

And I even consulted with ChatGPT back then, but got nowhere.

Maybe I'm a bit smarter now, and perhaps ChatGPT is too.

In any case, we painstakingly worked our way through it all this time, and things were discovered that weren't last time.

The code covered in this topic is only half the Python code. Then ChatGPT and I worked on the other half, and solved that too.

My feedback to ChatGPT was vital, as it allowed it to explore other avenues, which in the end finally resulted in the solution. We went very deep, and I can now claim to have worked successfully with ChatGPT, and that without its help, I would not have succeeded.

As they say, if at first you don't succeed, then try try again.  

[TheSaint]

3 hours ago, TheDcoder said:

What in tarnation is that?

It's a watery based version of ChatGPT in bathers or swimsuit.

[TheSaint]

3 hours ago, argumentum said:

Func qhash($inputstr, $iSeed = 0x00000000)
    If Not IsString($inputstr) Then Return -1
    Local $utf8 = StringToUTF8($inputstr)
    Local $temp, $val
    For $i = 1 To StringLen($utf8)
        $val = Asc(StringMid($utf8, $i, 1))
        $iSeed = Mod($iSeed * 16 + $val, 0x100000000) ; unsigned 32-bit left shift
        $temp = BitAND($iSeed, 0xF0000000)
        $iSeed = BitXOR($iSeed, UnsignedRightShift($temp, 23))
        $iSeed = BitAND($iSeed, 0x0FFFFFFF)
    Next
    Return $iSeed
EndFunc

...runned the code via "Wetware-GPT" and claims that it should be this way. I don't know what's the big difference.

That worked too, once I added in the two functions created by ChatGPT.