[solved] convert string into ANSI from UTF-8/Unicode

[Myicq]

I have a need to convert a unicode string into the appropriate ANSI encoding.

Purpose is to insert the ANSI string into an old application, which with proper encoding selected, displays the same character.

This seems like a simple job.

I have the following:

#include <WinAPI.au3>
$string = "שָׁלוֹם" ; shalom in Hebrew
$out = _WinAPI_WideCharToMultiByte($string)
ConsoleWrite($out & @CRLF )

My expected outcome is (using my default ANSI, Western Europe codepage):

ùÈÑìåÉí

which when encoded in Win1255 ("Hebrew") correctly displays שָׁלוֹם.

Problem is that I can ONLY get question marks, or empty string, no matter when I put as parameter for the WideChartoMultiByte...

What am I missing here ?

How can I tell the WideCharToMultiByte function which encoding I would like ?

So far I am looking for

UTF8 ==> GBK (or GB2312) for Chinese/Simplified

UTF8==> ANSI (Hebres, Win1255)

UTF8==> KoreanJohab

I can't be the only person ever to do this ?

The conversion works perfectly using sofware such as "Charco" (freeware), but I would love to have this natively in my script.

Edited December 3, 2012 by Myicq

[JohnOne]

#include <WinAPI.au3>

$string = "שָׁלוֹם" ; shalom in Hebrew

$out = _WinAPI_WideCharToMultiByte($string, 65001)

ConsoleWrite($out & @CRLF )

maybe.

[Myicq]

$out = _WinAPI_WideCharToMultiByte($string, 65001)

ConsoleWrite($out & @CRLF )

Well that results in same string I started with

I need to have the multi byte string, so that שָׁלוֹם ==> bytes [F9 C8 D1 EC E5 C9 ED] which when encoded with Win1255 Hebrew gives me the "Shalom" again. But in an old application that does not understand UniCode.

[JohnOne]

Have you tried other flags?

[Myicq]

Yes, I have tried other flags, most return just a bunch of ?????.

[JohnOne]

I don't know much about this, but according to unicode support section of help file.

UTF 8 has to be read in using the appropriate flags, so perhaps your literal string

does not fit that criteria.

Also, according to MSDN widechartomulibyte is for utf 16.

[guinness]

What about this?

Func _ANSIToUnicode($sString)
    #cs
        Local Const $SF_ANSI = 1
        Local Const $SF_UTF16_LE = 2
        Local Const $SF_UTF16_BE = 3
        Local Const $SF_UTF8 = 4
    #ce
    Local Const $SF_ANSI = 1, $SF_UTF8 = 4
    Return BinaryToString(StringToBinary($sString, $SF_ANSI), $SF_UTF8)
EndFunc   ;==>_ANSIToUnicode

Func _UnicodeToANSI($sString)
    #cs
        Local Const $SF_ANSI = 1
        Local Const $SF_UTF16_LE = 2
        Local Const $SF_UTF16_BE = 3
        Local Const $SF_UTF8 = 4
    #ce
    Local Const $SF_ANSI = 1, $SF_UTF8 = 4
    Return BinaryToString(StringToBinary($sString, $SF_UTF8), $SF_ANSI)
EndFunc   ;==>_UnicodeToANSI

Edited November 28, 2012 by guinness

[AZJIO]

Myicq

Encoding UDF - http://autoit-script.ru/index.php/topic,510.0.html

[Myicq]

@AZJIO .. this looks like what I need. Just need to find out how to register, my Russian is kind of rusty...

Could you perhaps upload the script to some temp filehost like http://ge.tt ?

If not I will try to create an account.

(Wonder how many other good resources are hiding in the Russian / French / German / Brazilian.. sites. )

Cheers.

myicq

[guinness]

Out of curiosity did you try the examples I posted or did you just skip my post?

[Myicq]

Out of curiosity did you try the examples I posted or did you just skip my post?

Guiness,

I did try that, and sorry to say that it only returned the same string

I had:

$string = "שָׁלוֹם" ; shalom..
$out = _UnicodeToANSI($string)
; rule out that ConsoleWrite plays trick on me..
filewrite(@ScriptDir & "out.txt", $out)

Func _UnicodeToANSI($sString)
#cs
Local Const $SF_ANSI = 1
Local Const $SF_UTF16_LE = 2
Local Const $SF_UTF16_BE = 3
Local Const $SF_UTF8 = 4
#ce
Local Const $SF_ANSI = 1, $SF_UTF8 = 4
Return BinaryToString(StringToBinary($sString, $SF_UTF8), $SF_ANSI)
EndFunc ;==>_UnicodeToANSI

Results are:

Expected: F9 C8 D1 EC E5 C9 ED    ; 7 byte:  Shalom in Win-1255
Actual:  D7 A9 D6 B8 D7 81 D7 9C D7 95 D6 B9 D7 9D  ; 14 byte: Shalom in UTF-8

Same if I put the input file in UCS2-LE, output is just the same as I input.

The conversion done by the "Charco" program is correct, not sure how it's done (http://www.marblesoftware.com/Marble_Software/Charco.html)

If I convert to UCS2 and look at the mapping [http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1255.TXT] it is correct for each and every character / point. But I doubt it should be necessary to use a lookup table for this operation ?

Hope this gives some clues. Help still much appreciated !

[AZJIO]

Myicq

I have a bad Internet connection to download the file, but you can download from this link http://u.to/ZkxsAQ - IncludeEncoding.au3

[guinness]

Myicq,

Fair enough. I will look into then.

[Myicq]

@Wise Ones

I don't have the skills myself to dig into this - at least not if I have to know what's going on. So this is just a pointer to a possibile solution.

I found the .NET class "Encoding" which seems to contain a conversion API.

From an example:

public static void PrintCPBytes(string str, int codePage)
{
Encoding targetEncoding;
byte[] encodedChars;
// Get the encoding for the specified code page.
targetEncoding = Encoding.GetEncoding(codePage);
// Get the byte representation of the specified string.
encodedChars = targetEncoding.GetBytes(str);
// Print the bytes.
Console.WriteLine
("Byte representation of '{0}' in Code Page '{1}':", str, codePage);
for (int i = 0; i < encodedChars.Length; i++)
Console.WriteLine("Byte {0}: {1}", i, encodedChars[i]);
}

Found here

There is a lot more information here

So the question basically is: how to use Encoding.GetEncoding etc from within AutoIT ? The only thread I could find about this topic was closed because of GameAutomation.

Hope this helps. It would be nice to have this turn into a UDF for all Windows Codepages

[jchd]

You're all making this way more complex than it really is.

The UTF-16 LE (or rather UCS2-LE that AutoIt uses) to ANSI conversion will use the codepage specified in the _WinAPI_MultiByteToWideChar call.

So the conversion you need is:

#include <WinAPI.au3>
$string = "שָׁלוֹם" ; shalom in Hebrew
$out = _WinAPI_WideCharToMultiByte($string, 1255) ; explicit Hebrew codepage

Pass $out to your old application.

[Myicq]

You're all making this way more complex than it really is.

The UTF-16 LE (or rather UCS2-LE that AutoIt uses) to ANSI conversion will use the codepage specified in the _WinAPI_MultiByteToWideChar call.

So the conversion you need is:

#include <WinAPI.au3>
$string = "שָׁלוֹם" ; shalom in Hebrew
$out = _WinAPI_WideCharToMultiByte($string, 1255) ; explicit Hebrew codepage

$string = "Москва"  ; Russian known city
$out = _WinAPI_WideCharToMultiByte($string, 1251) ; explicit Cyrillic codepage
filewrite(@ScriptDir & "out2.txt", $out)

Pass $out to your old application.

jchd,

indeed it seems like everyone incl me was running around like headless chicken.

What confused me I think was the help file, which shows only a few values (and vague descriptions) of possible values for codepage. Indeed the MSDN documentation also does not mention that other parameters are allowed.

Where can I suggest examples and additions to the help in this matter ?

And for others, the list of Windows ANSI codepages are here:

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/

So thanks for your help, jchd.

[jchd]

Indeed the MSDN documentation also does not mention that other parameters are allowed.

Untrue. The page clearly mentions:

Parameters

CodePage [in]

Code page to use in performing the conversion. This parameter can be set to the value of any code page that is installed or available in the operating system. For a list of code pages, see Code Page Identifiers. Your application can also specify one of the values shown in the following table.

Edited December 3, 2012 by jchd