Characters that don't break IniWrite()?

[baconaise]

Hello!

I am using IniRead() and IniWrite() to interact with save.ini.

I need to use a rarely-typed character as a delimiter, but all the rarely-typed characters I try break upon saving (e.g. Θ turns into T).

Is there a list of characters which will not break when I IniWrite() them?

 

Edit:

The solution is to add

BitOR($FO_UNICODE, $FO_OVERWRITE)

to the IniWrite line, so it looks like this:

Local $Save_File = FileOpen(@MyDocumentsDir & "\Test\Save.ini",BitOR($FO_UNICODE, $FO_OVERWRITE))

 

Edited March 13, 2023 by baconaise
Adding Solution to first post for posterity

[ioa747]

43 minutes ago, baconaise said:

but all the rarely-typed characters I try break upon saving (e.g. Θ turns into T)

this shouldn't have happened. try to save as utf8

I prefer the |

 

[baconaise]

6 minutes ago, ioa747 said:

this shouldn't have happened. try to save as utf8

I'm missing something.

Documentation says, "If you want to use an ini file with Unicode encoding, first create an .ini file by using the FileOpen() function with the mode parameter set to a Unicode parameter."

So, I start with:

FileOpen(@MyDocumentsDir & "\Test\Save.ini",$FO_UTF8)

Then, I do my:

IniWrite(@MyDocumentsDir & "\Test\Save.ini", "Example", "Test1", $Temporary_Text)
FileClose(@MyDocumentsDir & "\Test\Save.ini")

But, it has the same issue as before. What am I missing?

[ioa747]

I do not know. I have to see the script to understand

[pixelsearch]

@baconaise I recently started using the following syntax for a .ini file

Local $sIniFile = @ScriptDir & "\" & StringTrimRight(@ScriptName, 4) & ".ini"
...
IniWrite($sIniFile, "Main", "Pattern", StringToBinary(GUICtrlRead($ebRegExp), 4)) ; 4 = $SB_UTF8 (string data is UTF8)
...
GUICtrlSetData($ebRegExp, BinaryToString(IniRead($sIniFile, "Main", "Pattern", ""), 4)) ; 4 = $SB_UTF8 (binary data is UTF8)

Hope it helps

Edited March 13, 2023 by pixelsearch

[mistersquirrle]

Testing this out I also had the output saving as a 'T', the issue was I don't think that $FO_UTF8 is the right flag. This code allowed the character to save:

#include <Constants.au3>

Global $sSpecialChar = 'Θ' ; Θ
Global $sPath = @ScriptDir & '\Save.ini'
ConsoleWrite('SpecialChar: ' & $sSpecialChar & @CRLF)
ConsoleWrite('Path: ' & $sPath & @CRLF)

Global $hFile = FileOpen($sPath, BitOR($FO_UNICODE, $FO_CREATEPATH, $FO_OVERWRITE))
If $hFile = -1 Then
    ConsoleWrite('FileOpen error' & @CRLF)
    Exit
EndIf
FileClose($hFile)

IniWrite($sPath, "Example", "Test1", $sSpecialChar)

Something to note is that using ConsoleWrite to display the character did not display it correctly in SciTE, and you also need to make sure that when the file is saved that the character is saved correctly in the source file.

And, depending on what you're using these characters for, keep in mind that there are some specific separator characters: https://www.compart.com/en/unicode/search?q=separat#characters So you could use the invisible separator which should never be typed.

[baconaise]

@mistersquirrle It was the $FO_OVERWRITE that fixed it. Thank you!

[mistersquirrle]

Glad that worked, just keep in mind that $FO_OVERWRITE will, you know, overwrite the file each time the script runs. So anything you add is removed when the script starts next time. I just mostly had $FO_OVERWRITE so that it would remove and add the file again to make sure it was the correct encoding while testing.

[pixelsearch]

56 minutes ago, mistersquirrle said:

Something to note is that using ConsoleWrite to display the character did not display it correctly in SciTE,

For the record, @jchd indicate in several posts a way to display correctly Unicode characters in Scite Console.
For instance, the character discussed in the precedent posts (Unicode 920, hex 0x398) can be displayed like this in Scite Console :

; ConsoleWrite(ChrW(0x398) & @crlf)
ConsoleWrite(BinaryToString(StringToBinary(ChrW(0x398), 4), 1) & @crlf)

Why it works is still mystery to me because :
* StringToBinary() uses a 2nd param. = 4 ("string data is UTF8")
* BinaryToString() uses a 2nd param. = 1 ("binary data is ANSI")

Also what confuses me is that we read in the help file, topic "Unicode Support" :

Current Limitations :
Console operations are converted to ANSI. 

So how a Unicode character can been displayed in Scite Console which accepts only ANSI characters (code 0-255) and this character got a code > 255 ?

I really wish jchd could teach me this in his own words

Also, I noted that Scite menu had an influence on the script as it appears in Scite :
When I pasted mistersquirrle's code in Scite, I had to go to File => Encoding => change "Code Page Property" to "UTF-8"

Only then the ChrW(0x398) appeared correctly in this line of code :

Global $sSpecialChar = 'Θ' ; ChrW(0x398) shows T in Console

 

Edited March 13, 2023 by pixelsearch

[jchd]

To display Unicode correctly in the Scite console, change the Scite console setting to Unicode (charset 65001) and check that the font you use contains enough of Unicode for your needs. DejaVu Sans Mono is perfect for that. Then use the double conversion as cited above.

7 hours ago, pixelsearch said:

Why it works is still mystery to me because :
* StringToBinary() uses a 2nd param. = 4 ("string data is UTF8")
* BinaryToString() uses a 2nd param. = 1 ("binary data is ANSI")

First conversion is from a UCS2 (subset of Unicode used by AutoIt) to binary UTF8, second conversion is from binary UTF8 to UTF8 characters.

[pixelsearch]

@jchd thanks for the explanation, I'll try asap what you suggested in your post.

If you have a moment, could you please give your opinion on the following question, in case it's related to what you already explained ?

Picture above : the script "test1.au3" is run and shows in Scite Console Chr(128) = €
Is it normal that Chr(128) is euro sign when I don't use a codepage 1252 ?

As you can see, I didn't change the encoding at all and it shows UTF-8
If it can help, when I type Alt+128 in NotePad, it displays a Ç in NotePad and not the euro sign (no matter the NotePad file is UTF-8 or not)

Thanks  

[jchd]

What works for me :

In file C:\Users\**username**\AppData\Local\AutoIt v3\SciTE\SciTEUser.properties

font.monospace=font:DejaVu Sans Mono,size:11
code.page=65001
output.code.page=65001
NewFileEncoding=UTF8
utf8.auto.check=4

1 hour ago, pixelsearch said:

Picture above : the script "test1.au3" is run and shows in Scite Console Chr(128) = €
Is it normal that Chr(128) is euro sign when I don't use a codepage 1252 ?

You're using UTF8 as source encoding. Chr(128) outputs the ASCII character of the SciTE output console. By default it's the default user setting, aka locale ASCII codepage. Also built-in ConsoleWrite "ASCIIfies" everything before display.

To fix that in an UTF8 source with above settings in force, and to be able to display a good share of Unicode BMP use this:

expandcollapse popup

CW("€")

; Mixed language strings
$s = "Большая проблема  大问题  बड़ी समस्या  مشكلة كبيرة Test Title 😭" & @LF
$s &= "Our familly " & ChrW(0xD83D) & ChrW(0xDC68) & ChrW(0xD83C) & ChrW(0xDFFB) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC69) & ChrW(0xD83C) & ChrW(0xDFFF) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC66) & ChrW(0xD83C) & ChrW(0xDFFD)

CW($s)



Func CW($s = "")
    If @Compiled Then
        _CUI_ConsoleWrite($s)
    Else
        _ConsoleWrite($s)
    EndIf
EndFunc   ;==>CW

Func _CUI_ConsoleWrite(ByRef $s)
    Local Static $hDll = DllOpen("kernel32.dll")
    Local Static $hCon = __CUI_ConsoleInit($hDll)
    DllCall($hDll, "bool", "WriteConsoleW", "handle", $hCon, "wstr", $s & @LF, "dword", StringLen($s) + 1, "dword*", 0, "ptr", 0)
    Return
EndFunc   ;==>_CUI_ConsoleWrite

; internal use only
Func __CUI_ConsoleInit(ByRef $hDll)
    DllCall($hDll, "bool", "AllocConsole")
;~  The following 2 lines don't work for compiled scripts due to a MS bug
;~      see last post in thread https://developercommunity.visualstudio.com/t/setconsoleoutputcpcp-utf8-results-in-code-page-850/413190
;~      where MS support acknowledges it's a (still unfixed) bug in Windows itself
;~      So you can only display current locale codepage using current console font in a console when your script is compiled
;~      When running your script from SciTE, the UTF8 setting of SciTE overrides this bug and Unicode can be displayed
;~  DllCall("Kernel32.dll", "bool", "SetConsoleCP", "uint", 65001)
;~  DllCall("Kernel32.dll", "bool", "SetConsoleOutputCP", "uint", 65001)
    Return DllCall($hDll, "handle", "GetStdHandle", "int", -11)[0]
EndFunc   ;==>__CUI_ConsoleInit

; Unicode-aware ConsoleWrite
Func _ConsoleWrite(ByRef $s)
    ConsoleWrite(BinaryToString(StringToBinary($s & @LF, 4), 1))
EndFunc   ;==>_ConsoleWrite

The string $s is a sequence of joined emojis with FitzPatrick mods, rendered as one glyph by the Unicode renderer.

Using CW() you don't have to add a @LF. This function displays verbatim Unicode in GUI or CUI scripts run from SciTE. OTOH compiled scripts can only display characters in the user console codepage due to a bug in Windows (see comments in code).