Yet another Encoding topic ANSI->UTF8

[careca]

Hello, trying to read from the console, and ive seen some posts where people mention changes in registry and in scite properties file, which i did, but none worked.

So now im stuck with this, as you can see, in the console the words are faithfully represented, but when autoit reads it, its lacking, in the help it says its a limitation that may be addressed in the future, but is there anything i can do, now to fix this?

 

Thanks.

 

Oh, the change in registry was this

[HKEY_CURRENT_USER\Console]
"CodePage"=dword:0000fde9
"FaceName"="Lucida Console"

Edited July 12, 2018 by careca

[jchd]

Try converting the captured text with this:

Func _OemToStr($sOEM, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & StringLen($sOEM) & "]")
    DllStructSetData($tText, 1, $sOEM)
    ConsoleWrite(_vardump($tText) & @LF)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_OemToStr

You may have to explicitely pass the source codepage as parameter.

[careca]

Hi, thank you for the reply, but it doesn't seem to be working somehow.

I tested this way, it's faster and easier:

file 1.txt with: Hello - 你好

#include <StringConstants.au3>
Global $iCodepage = 65001
$FO = FileOpen(@ScriptDir&'\1.txt')
$FR =FileRead($FO)
FileClose($FO)
MsgBox(64, '$FR', $FR)
;ConsoleWrite('$FR - '& $FR &' - '&@MSEC&@CRLF)
;$t1_unicode = Stringtobinary($FR,4)
;$t2_unicode = BinaryToString($t1_unicode, 4)
;ConsoleWrite('$t2_unicode - '& $t2_unicode &' - '&@MSEC&@CRLF)
;MsgBox(64, '$t2_unicode', $t2_unicode)
MsgBox(64, 'Read', _OemToStr($FR, $iCodepage))

Func _OemToStr($sOEM, $iCodepage)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & StringLen($sOEM) & "]")
    DllStructSetData($tText, 1, $sOEM)
    ;ConsoleWrite(_vardump($tText) & @LF)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_OemToStr

 

[jchd]

Sorry I left a debugging aid in between.

But what's the difference?

[careca]

No difference, "Hello - 你好" becomes "Hello - ??" same as not converting at all.

Either from console, or a txt file, when autoit handles them, they become jibberish.

 

[jchd]

This kind of conversion requires that you correctly determine the actual encoding (and codepage when applicable) of the source.

[careca]

Thank you for the help, i didn't notice this is a royal mess, thought i could just convert one or two types of encoding and it's done, but it's so much bigger than that.

As i looked for the code page of a language such as Polish...there's no way i can mess with the encoding for all the countries that use something else that's not unicode. The time and effort for this just doesn't compensate the results.

I only wish that one day we can get a uniformized standard for all countries, something that works everywhere and for everyone.

I actually learned a bit about this, but still a lot to learn and i thank you for the effort and help.

 

[jchd]

The unique standard you're after is just that: Unicode. With it you can unambiguously and portably represent any character humanity ever used (albeit it's still a work in progress).

The mess is 8-bit character codepages and 2-byte codepages for some regions like Asia.

To focus again on your issue: in a CMD console type CHCP <Enter>. The system will return the actual codepage in force, which you should be able to use as parameter for the ANSI -> Unicode conversion function.

DOS used codepage 852 for central/eastern languages. Windows uses 1252 or 1250. Just check what's used in your own setup. If I'm not mistaken then your console "ŠŚŤŽŹšśťžźŁĄŻłąşŞĽľżŔÁÂĂÄĹĆÇ..." will convert verbatim in AutoIt native Unicode strings.

[begginner]

I had similar problem as careca after executing dos and powershell commands by "Run"  and displaying results in Gui  using $STDOUT_CHILD . Instead of polish characters there was bunch of cryptic gibberish.  I used  _WinAPI_WideCharToMultiByte to do conversion  with any codepages but it didn't work.  I was so desperate to 

use StringReplace but problem was some of characters wasn't even appeared in output so was no way to replace them.  Then I tried this function that "jchd" posted

and it TOTALY WORKED !!! . THANK YOU SO MUCH for your help

[begginner]

BTW  I tried exactly same command as   careca did  it with _OemToStr function and it works no problem

This is consolewrite output in SciTe before _OemToStr function

 

 

and after

 

 

and this is Gui window

CHCP command output "Active code page: 852"

Hope it helps

 

Edited July 17, 2018 by begginner