File Encoding problem

[mLipok]

I have created few ini files.

When I start to use Au3.3.14.0 then all file operations change behavior as it was described in SCRIPT BREAKING CHANGE.

Maybe there is something that I did not understand or forgot, I do not know.
But it is at the moment is not important to me.

It was bring me to a problem and I am looking for how to solve it.

this INI should contain such a 

07=inne pisma|I - Inne|Inne pismo|???;wniosek o rozstrzygnięcie 

but acutally this contain something like this:

  07=inne pisma|I - Inne|Inne pismo|???;wniosek o rozstrzygniĂ„â„Ëcie

 

 

So I want to ask:

What happend ?
What I miss ?
Is there any way or what I can do now to convert it back to correct format ?

 

 Thanks for any help.

 

[jchd]

This looks like UTF8. Note that when std INI functions don't translate native UTF16 AutoIt strings into UTF8 by themselves (maybe they should do that by now ???), so you have to convert by yourself before writing and after reading.

[BrewManNH]

INI files are not Unicode compatible, that's because they were created back before Unicode was the standard in Windows.

If you want to use an ini file with Unicode encoding, first create an .ini file by using the FileOpen() function with the mode parameter set to a Unicode parameter.
 

[jchd]

This works out of the box:

Local $sSection = "section_αβγδεζηθ"
Local $sKey = "key_ЖЗИЙЛЦЩ"

IniWrite("test.ini", _StringToUtf8($sSection), _StringToUtf8($sKey), _StringToUtf8("value_ℂℍℕℙℚℝℤ"))
Local $sValue = _Utf8ToString(IniRead("test.ini", _StringToUtf8($sSection), _StringToUtf8($sKey), "Failure"))
MsgBox(0, "", "Section = " & $sSection & @LF & "Key = " & $sKey & @LF & "Value = " & $sValue)

Func _StringToUtf8($sIn)
    Return(BinaryToString(StringToBinary($sIn, 4), 1))
EndFunc

Func _Utf8ToString($sIn)
    Return(BinaryToString(StringToBinary($sIn, 1), 4))
EndFunc

... as long as the INI file is not meant to be read/written by non UTF8-aware programs.

Edited July 28, 2015 by jchd

[BrewManNH]

Seems like a lot of code when all you need to do is this:

#include <fileconstants.au3>
Local $sSection = "section_αβγδεζηθ"
Local $sKey = "key_ЖЗИЙЛЦЩ"
FileClose(FileOpen("test.ini", $FO_UTF16_LE + $FO_OVERWRITE)) ; all the magic is done here
IniWrite("test.ini", $sSection, $sKey, "value_ℂℍℕℙℚℝℤ")
Local $sValue = IniRead("test.ini", $sSection, $sKey, "Failure")
MsgBox(0, "", "Section = " & $sSection & @LF & "Key = " & $sKey & @LF & "Value = " & $sValue)

to get it to write in the INI file as Unicode, once the file is created using UTF16 little endian, you never have to worry about it again. BTW, you have to use $FO_UTF16_LE (or $FO_UNICODE) to encode the INI file or it won't work. IniRead and IniWrite can be used normally because the file is encoded correctly already

Edited July 28, 2015 by BrewManNH

[jchd]

Ha, so then the INI functions are now UTF-aware as long as a BOM is there. This is relatively recent unless I missed something. I misread your previous post as if one needed to parse the UTF16-le file pedestrianly.

[BrewManNH]

It's something I came across while researching INI file encoding, it's mentioned in the help file now, although not in enough detail. The UTF16 LE part isn't mentioned in the help file. If you try it with any other Unicode format other than UTF16 LE the ini file becomes corrupted when you use the INI functions.

[mLipok]

Thanks for your answers

Here is whats going on:

_Example1()

Func _Example1()
    FileDelete('test.ini')
    IniWrite('test.ini','Test1','1','Piorko')
    MsgBox(0, '1', FileGetEncoding('test.ini'))

    IniWrite('test.ini','Test1','2','Piórko')
    MsgBox(0, '2', FileGetEncoding('test.ini'))
    MsgBox(0, '2', IniRead('test.ini','Test1','2',''))

    Local $sConntent = FileRead('test.ini')
    FileDelete('test.ini')
    FileWrite('test.ini',$sConntent)

    MsgBox(0, '3', FileGetEncoding('test.ini'))
    MsgBox(0, '3', IniRead('test.ini','Test1','2',''))


    MsgBox(0, '4', FileRead('test.ini'))

EndFunc

 

 

EDIT:

I notice that I use some text parsing on this INI file, and this cause the problem.
This snippet shows when this happend.

Edited July 30, 2015 by mLipok
typo in function name

[jchd]

Hi,

What BrewmanNH explained is that you just need to create the file as UTF16-LE, like this:

_Example1()

Func _Example1()
    FileDelete('test.ini')
    Local $h = FileOpen('test.ini', 2 + 1024)   ; <--- here
    FileWrite($h, @CRLF)                        ; you need to write something
    FileClose($h)
    MsgBox(0, '0', FileGetEncoding('test.ini')) ; 1024 = UTF16-LE
    IniWrite('test.ini','Test1','1','Piorko')
    MsgBox(0, '1', FileGetEncoding('test.ini'))

    IniWrite('test.ini','Test1','2','Piórko')
    IniWrite('test.ini','Test1','3','αβγδεζηθ')
    IniWrite('test.ini','Test1','4','ЖЗИЙЛЦЩ')
    MsgBox(0, '2', FileGetEncoding('test.ini'))
    MsgBox(0, '2', IniRead('test.ini','Test1','2',''))
    MsgBox(0, '3', FileRead('test.ini'))
EndFunc

If you ever delete the file, you must recreate it as UTF16-LE again. Then both *INI functions, FileRead* and FileWrite* don't need extra steps to correctly handle it.

[mLipok]

Thanks
In the meantime I myself have choosen $FO_ANSI 

Is it wrong , decision ?
I saw this @BrewManNH answer

It's something I came across while researching INI file encoding, it's mentioned in the help file now, although not in enough detail. The UTF16 LE part isn't mentioned in the help file. If you try it with any other Unicode format other than UTF16 LE the ini file becomes corrupted when you use the INI functions.

But could you show any example what wrong could happen when any body use $FO_ANSI instead $FO_UTF16_LE_NOBOM 

?

Then this thread will be regarded as complementary to describe this problem.

 

 

[jchd]

If you use ANSI, then you must use my previous example with functions to convert between AutoIt UTF16 strings and UTF8 when section names, keys and/or values use non ANSI characters. Also as you know, ANSI is not portable accross machines using distinct locales while UTF8 is. UTF8 is essentially undistiguishable from ANSI, so UTF8 values work.

BrewManNH insisted on Unicode: "If you try it with any other Unicode format other than UTF16 LE the ini file becomes corrupted when you use the INI functions." ANSI isn't an Unicode format.

To see how things get bad, run this modified version of the same example as above:

_Example1()

Func _Example1()
    FileDelete('test.ini')
;~  Local $h = FileOpen('test.ini', 2 + 1024)   ; <--- here
;~  FileWrite($h, @CRLF)                        ; you need to write something
;~  FileClose($h)
    MsgBox(0, '0', FileGetEncoding('test.ini')) ; 1024 = UTF16-LE
    IniWrite('test.ini','Test1','1','Piorko')
    MsgBox(0, '1', FileGetEncoding('test.ini'))

    IniWrite('test.ini','Test1','2','Piórko')
    IniWrite('test.ini','Test1','3','αβγδεζηθ')
    IniWrite('test.ini','Test1','4','ЖЗИЙЛЦЩ')
    MsgBox(0, '2', FileGetEncoding('test.ini'))
    MsgBox(0, '2', IniRead('test.ini','Test1','2',''))
    MsgBox(0, '3', FileRead('test.ini'))
EndFunc

Also try uncommenting the 3 lines and change 1024 into 256 (UTF8) or 2048 (UTF16_BE) to see what happens.

[mLipok]

Thanks @jchd and @BrewManNH

Now all is clear. In my case 512 was ok as I do not need to use all others char sets , but for compatibility and confidence that everything will always OK, it's better I'll use the suggested 1024.

btw.
to use this examples there is need to set in your SciTEGlobal.properties, for example: code.page=65001

 

 

[guinness]

I am surprised it worked even before v3.3.14.0, as I have know about ini files only reading unicode when encoded before hand, for something like 4 years. 

[mLipok]

even @guinness learned something new today

 

[jchd]

to use this examples there is need to set in your SciTEGlobal.properties, for example: code.page=65001

This applies only to the SciTe console and to output Unicode correctly one also nees a modified version of ConsoleWrite to feed it UTF8 strings:

; Unicode-aware ConsoleWrite
Func __ConsoleWrite($s)
    ConsoleWrite(BinaryToString(StringToBinary($s, 4), 1))
EndFunc   ;==>__ConsoleWrite

 

[mLipok]

 

For console is:

output.code.page=65001

and 

code.page=65001

is for main SciTE panel.

---

Actually , as I see, if it is not set to 65001 then I'm unable to copy this

IniWrite('test.ini','Test1','3','αβγδεζηθ')
    IniWrite('test.ini','Test1','4','ЖЗИЙЛЦЩ')

to script in SciTE.

So it is imposible to use it in SciTE.

 

If I'm wrong , please correct me.

 

 

[jchd]

To be able to copy this in the editor pane of SciTe you need to switch the source file encoding to UTF8: File > Encoding > UTF8