Convert UTF-8 in a windows valid path - [SOLVED]

[bi5bo]

I have some paths in an sql database utf-8 encoded.
I need to convert in a windows correct path. Some dir and file names have japanese or arabic characters and/or symbols.

How can i do? i've tried a lot of solutions but no success

By example i need to convert this (utf-8): ââââ (In my db field chars are different)

 in this: ★★★

in php utf8_encode() and utf8_decode() makes the job, but in autoit?

Thanks in advance
 

[Jos]

Moved to the appropriate forum, as the Developer General Discussion forum very clearly states:

Quote

General development and scripting discussions. If it's super geeky and you don't know where to put it - it's probably here.

---

Do not create AutoIt-related topics here, use the AutoIt General Help and Support or AutoIt Technical Discussion forums.

Moderation Team

[jchd]

Func _CodepageToString($sCP, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = 65001
    Local $tText = DllStructCreate("byte[" & StringLen($sCP) & "]")
    DllStructSetData($tText, 1, $sCP)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sCP), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sCP), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_CodepageToString

Func _StringToCodepage($sStr, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = 65001
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodepage, "dword", 0, "wstr", $sStr, "int", StringLen($sStr), _
            "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    Local $tCP = DllStructCreate("char[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodepage, "dword", 0, "wstr", $sStr, "int", StringLen($sStr), _
            "struct*", $tCP, "int", $aResult[0], "ptr", 0, "ptr", 0)
    Return DllStructGetData($tCP, 1)
EndFunc   ;==>_StringToCodepage

Here you are.

[bi5bo]

Hi  Jchd, Thank you for your quick reply.
Unfortunately, even this function won't return the correct value.

ConsoleWrite (_CodepageToString(_StringToCodepage("★")))

The result is:  ?

I lost a couple of days reading your posts before ask here, and trying out functions like MultibyteToWideChar () and others.
The workaround could be to pass the correct string from PHP to autoit, cause the PHP function utf8_decode () function does the job correctly.
But I would like to do it in autoit.  So I could read the db directly from autoit and make same loops job with it.
Any idea?

[jchd]

The Scite console hasto be switched to UTF8 in order to display UTF8 properly. So use MsgBox to display Unicode easily.

Which DB engine do you use which doesn't have UTF16 output?

[bi5bo]

I have already activated Scite on utf8.

I am using mysql 5.6. that supports UTF16, but I've read everywhere it's better to have a UTF8mb4 character set to support all languages.
So I changed text fields collation on utf8mb4_general_ci.
Since I activated utf8_general_ci I have no longer any errors, inserting filenames.

[bi5bo]

Strange behaviour.
Using MsgBox your function returns the characters as expected.
But Scite was switched in utf8.

Now i must understand why not return as expected with mysql results.

[jchd]

All depends on how you retrieve the UTF8 data, I mean which UDF and functions you use to transfer data to/from the engine. Remember that unless precautions are taken, AutoIt strings are UCS2 (a subset of UTF16LE limited to the Unicode BMP).

MySQL's utf8 is a buggy and terribly bad implementation of Unicode UTF8, which MySQL now calls utf8mb4 (why make it simple?).

AutoIt doesn't handle UTF8 by itself (AutoIt strings are UCS2) but it's possible to retrieve UTF8 data and convert it to UCS2. UTF16-LE codepoints above the BMP (that is > 0xFFFF) can be stored but aren't dealt with perfectly since an UCS2 codepoint is a single 16-bit encoding unit.

I've no experience with MySQL, only SQLite and some other RDBMS, but the encoding used to transfer data to/from the engine can be distinct from the encoding used by the storage engine.

Note that there is a very good and stable ADO UDF available by @mLipok.

[bi5bo]

Finally solved!! 😀
Your function works great.

 

I just had to set the db in utf8 before doing the query.
It took me one week to understand it.

so:

$mysql= _MySQL_Real_Query($MysqlConn, "Set NAMES 'utf8mb4'");
$mysql_bool = _MySQL_Real_Query($MysqlConn, $query)
$res = _MySQL_Store_Result($MysqlConn)
$row = _MySQL_Fetch_Row_StringArray($res)

$path=_CodepageToString($row[2]);

Thanks again Jchd !