#include fails badly if the included file has BOM

[anonymous]

I had occasion to do something like this:

...get system data ...
_report()
HotKeySet("{F1}", "_report")
OnAutoItExitRegister("_report")

#include "variables.au3"

Msgbox(0,"test","test")
... stuff ...

Func _report()

BlockInput(0)
Msgbox(0,"","... stuff to tell the user ...")

EndFunc

othercode.au3 was simply some variable assignments, nothing more, but some of those variables were non-English strings and therefore included unicode characters.

The _exit hooks repeatedly threw up an error. I eventually tracked it down to this issue:

The included file being Unicode was no problem. But it had been created in a text editor that saved unicode text with a BOM (byte order mark) and that threw AutoIt badly. It couldn't recognise the included file as code nor anything after it.

Resave the identical #included file as unicode without a BOM and it immediately worked.

In this day of unicode, AutoIt probably needs to be fixed so that it copes with .au3 files, including #included .au3 files, that have BOM, as this is quite common. Worst of all it doesn't give the user any idea what's up and the cause is subtle (many users may not even be aware of it).

[Stilez]

(submitted by: Stilez)

Typo: "_exit hooks" should read "_report hooks"

[Jon]

What format was it saved in exactly and which BOM (UFT8, UTF16 LE BE...)? All the AutoIt text file functions use the same code (File...() #include, etc.) and they all support BOMs.

[Jpm]

Replying to Jon:

What format was it saved in exactly and which BOM (UTF8, UTF16 LE BE...)? All the AutoIt text file functions use the same code (File...() #include, etc.) and they all support BOMs.

Jon you can use the attached file to see that something wrong if the #include is used

[Valik]

Jon, this is probably an issue where the BOM is inserted inline into the middle of source code because of how #include works. The BOM needs stripped before the rest of the file is inserted in the "output" file.

[jchd]

Interim workaround: change encoding of the main file to UTF8+BOM. This will allow includes in both ANSI and/or UTF8+BOM (they can then be freely mixed). However upper-ANSI characters in non-UTF8 includes will be badly interpreted.

It looks like the encoding of the main file determines the encoding of the whole source. It is no problem with [lower ANSI only) ANSI includes since they _are_ valid UTF8 format.