Forcing UTF8+BOM encoding for every file in SciTE?

[jchd]

I can't remember how to do that and where it should be.

Is it at all possible to do so that it survives installation of new versions of SciTE4AutoIt3?

TIA

[trancexx]

What do you mean for every file?

What if file doesn't have BOM?

Btw, SciTEUser.properties survives reinstallations, so does its settings.

Edited September 12, 2013 by trancexx

[wraithdu]

Use these properties in your SciTEUser.properties file, hopefully.

code.page
output.code.page

To support a DBCS language such as Japanese, a code page can be set here. This ensures that double byte characters are always treated as a unit so the caret is never located between the two bytes of a double byte character.
 

Code page Value

Default (single byte character set) 0

UTF-8 65001

Japanese Shift-JIS 932

Simplified Chinese GBK 936

Korean Wansung 949

Traditional Chinese Big5 950

Korean Johab 1361

Setting code.page to 65001 starts Unicode mode and the document is treated as a sequence of characters expressed as UTF-8. Display is performed by converting to the platform's normal Unicode encoding first so characters from any language will be displayed. Correct glyphs may only be displayed if fonts are chosen that contain the appropriate glyphs. The Tahoma font contains a wide range of glyphs so may be a good choice. 
This property can not set a single byte character set.
If output.code.page is set then it is used for the output pane which otherwise matches the edit pane.

http://www.scintilla.org/SciTEDoc.html

Edited September 12, 2013 by wraithdu

[Jos]

Quick check told me there is no setting for it but could be done with a lua script.

Woud be happy to add that into the standard installer and have a properties keyword set this to the correct value.

You could then add that into the sciteuser.properties to survive any upgrades.

Jos

[jchd]

Sorry for being lazy: I meant "every new file created by SciTE", not every file in the wild...

code.page=65001

did work up to some point to force the console into using UTF8, but it doesn't do anything in the last few releases of SciTE4AutoIt3 (W7x64 if that matters). That contradicts the SciTE doc.

I didn't know about output.code.page=65001 but it's NOP as well (for forcing the console into UTF8). Until recently as exposed above, I had the console display Unicode without this setting, just the previous one.

I know for sure that Dos "CHCP 65001" works, but not more beyond in terms of Windows consoles.

[trancexx]

^^ SciTE has one particular property called unicodeMode. It's enum value defined like this:

enum UniMode {
    uni8Bit = 0, uni16BE = 1, uni16LE = 2, uniUTF8 = 3,
    uniCookie = 4
};

When unicodeMode is set to uniUTF8 the BOM you want is generated.
Considering SciTE doesn't have user-settable property named unicode.mode the only way to force BOM genaration is by using menu option (or executing some script).

Edited September 12, 2013 by trancexx

[jchd]

I see, well I believe I see. This is in contrast with NotePad++ which has such a user setting.

I'm completely Lua- and SciTE-illiterate, so forgive me if I'm thick asking whether a silent script (or some other way) is possible at launch time?

I don't ask for my own needs (in fact I currently don't need Scite, nor AutoIt at all). I see or get PMed scripts in various languages encoded in their own ANSI codepage. Those have litterals and comments destroyed in my system and it's a pain to recode them after fixing. If everyone used UTF8 scripts, exchanges would be painless and unambiguous regardless of the region they live in.

[trancexx]

Adding something like that to SciTE directly is trivial. However, the SciTE-for-AutoIt policy so far was that any customizations on SciTE core level should be kept at minimum because of maintainence issues with upgrades.

What's current policy is beyond me, ask developers.

[Jos]

Sorry for being lazy: I meant "every new file created by SciTE", not every file in the wild...

 

That is also what I meant.

I will add the LUA script I found on this topic and make that dependent on a keyword in the properties files.

More to follow.

Jos

[jchd]

I didn't plan to push the idea as a firm request but if you find it can be done someday and it's useful for users at large, then it's fine.

[jchd]

@Jos,

BTW neither code.page=65001 or output.code.page=65001 actually work to display Unicode correctly in the SciTE console. I don't test systematically every version/beta so I can't say when it stopped working but it's something relatively recent, I'd say with the current beta series.

Is this something that can be a consequence of recent changes in SciTE4AutoIt3 or should I look somewhere else? Apart from regular Win7 updates, I don't change this system much.

What I find strange is that I can paste Unicode e.g. 한국어 텍스트의 예 טקסט עברית ירושלים to the output console and it shows fine. ConsoleWrite is certainly not the culprit since is doesn't work either when launched with F5 (3.3.8.1), albeit it used to work fine with previous Scite4AutoIt3.

@Others,

Do you have any success with these settings?

[czardas]

I generally use the function BinaryToString() if I want to use non-ascii characters in my script. This just seems a safer alternative than inputting raw characters. It does require more messing about to do this, but I doubt I'll change the habit. Even so it would be nice to be able to use the real characters in comments.

Edited September 13, 2013 by czardas

[Jos]

@Jos,

 

BTW neither code.page=65001 or output.code.page=65001 actually work to display Unicode correctly in the SciTE console. I don't test systematically every version/beta so I can't say when it stopped working but it's something relatively recent, I'd say with the current beta series.

 

Is this something that can be a consequence of recent changes in SciTE4AutoIt3 or should I look somewhere else? Apart from regular Win7 updates, I don't change this system much.

 

What I find strange is that I can paste Unicode e.g. 한국어 텍스트의 예 טקסט עברית ירושלים to the output console and it shows fine. ConsoleWrite is certainly not the culprit since is doesn't work either when launched with F5 (3.3.8.1), albeit it used to work fine with previous Scite4AutoIt3.

I am not going to be able to check this in the coming days, but could you PM me a simple replicator script in a zipped format so I can test next week when i am back home?

Thanks

Jos

[jchd]

Jos,

Yes of course, enjoy some good time (at least I hope it is), this isn't going to kill anyone in the meantime!

czardas,

BinaryToString? Really? That seems utterly impractical to me   or to be used once a year!

[czardas]

BinaryToString? Really? That seems utterly impractical to me   or to be used once a year!

 

It is impractical, but for someone who knows so little about the differences between the methods employed in various kinds of software (and the consequences which may occur), I feel I have finally gained some control over how my code will perform worldwide. I can also grab the binary for purposes beyond my personal AutoIt projects - post my code on the web etc...

Edited September 13, 2013 by czardas

[jchd]

But isn't precisely why thousands of experts painfully specified Unicode?

[czardas]

LOL, true but I don't see what else I can do in a kitchen with so many cooks. I am not an expert.

[jchd]

You don't need to. Just copy Unicode from charmap.exe in advanced display mode into an UTF-enabled editor and you're done: worldwide fame!

[czardas]

Hmm, maybe I'm wrong but doesn't charmap only cater for within the BMP. If so I'm afraid that's totally inadequate.

Edited September 14, 2013 by czardas

[jchd]

Yeah but you'd have to find commonly spread/available font(s) covering more. I now realize you're likely using musical symbols, so no charmap is impotent.

But then how do you handle encoding with surrogates?