Autoit3Wrapper character encoding problem

[Allow2010]

how can i tell scite to open the files in utf8 mode? As you can see in the pictures scite opens them with broken characters until i switch to utf8

When i create a new file (reight clich->New-Au3 file) it is created as DOS Ascii file.

When i use Compile with options it is always converted to an utf-8 file

What is the easiest way to make sure all files have the same format (how to convert) and what format is recommended?

Edited August 27, 2015 by Allow2010

[Jos]

It does that automatically in case there is no BOM. It will check the file for Special charters and will open in UTF8 wo BOM when none are detected.
In case some special ansi characters are present in the file, like your test.au3, it will open in ANSI.

So when you take your posted example, both files are merged and then when opened it recognised the Special ansi character and will open as ANSI, which corrupts the UTF8 character from the main file.

Unsure how this need to be properly handled, but I am sure somebody can tell me.

Jos

[Jon]

I just checked and the scripts in that file open in ANSI with mangled characters in scite, but correctly open as uft8 in other editors. It doesn't matter if you convert encoding to utf8 add a space and re-save - it opens as mangled again. The file will contain a non-ansi utf8 sequence so I would have thought it would be opened ok by scite. I'm sure it used to when I last tested that sort of thing.

That's just scite though, doesn't explain anything that happens in the tool chain.

 

Edited August 27, 2015 by Jon

[Jos]

@Jon,Both files open fine when this settings are used ( not yet the default in the current production as I forgot to make that change in the last release)

utf8.auto.check=4

But that isn't really the issue here. We have 2 files of which one opens in UTF8 wo BOM and the other as ANSI. au3stripper simply concatenates the files without any conversion, so then when opened by SciTE it sees the special ANSI character and will obvuously corrupt the UTF character from the files file.

Jos

Edited August 27, 2015 by Jos

[Jon]

Concatenating the files is going to be problematic. Better to just open the all files with auto-detect encoding, merge them in memory, and then always write out the result as utf8 wo BOM. Does au3stripper have auto-detect code in already? I can't recall if it's AutoIt program or a custom one.

[Jos]

It is a custom program written in BCX ( a basic to C code generator) as I used that to convert the AutoIt3 script for gaining speed.
It doesn't have any UTF8 support so don't think it will be an simple change.

EDIT: Could always use the AutoIt3 script I already have for merging all code into one file first.

Jos 

Edited August 27, 2015 by Jos

[Jon]

BCX, blast from the past!

[Allow2010]

@Jon,Both files open fine when this settings are used ( not yet the default in the current production as I forgot to make that change in the last release)

utf8.auto.check=4

But that isn't really the issue here. We have 2 files of which one opens in UTF8 wo BOM and the other as ANSI. au3stripper simply concatenates the files without any conversion, so then when opened by SciTE it sees the special ANSI character and will obvuously corrupt the UTF character from the files file.

Jos

Sorry for this noob question, but how can i apply this setting to my SciTE ? It would at least help a bit in my everyday work...for now i can live without au3stripper...

Edited August 27, 2015 by Allow2010

[Jos]

Just put that line in your SciTeUser.properties : SciTE Menu: Options/Open User Options File

As to using au3stripper: just ensure you only use utf8 with or without bom and you should be fine.

Jos

[Allow2010]

Thank you.

I noticed something else, a bit offtopic:

i reinstalled my system and switched to win 10. I installed autoit, SciTE and the beta files as usual.

I now noticed that compiling my files creates temp files in the root of the drive where i compile.

They are named like this: ~CoXXXX and are 0 bytes. Never had this before on Win8.1.

Beta Bug or Win10 Problem?

Edited August 28, 2015 by Allow2010

[Jos]

Any idea which utility does that?

Jos

[mLipok]

I now noticed that compiling my files creates temp files in the root of the drive where i compile.

They are named like this: ~CoXXXX and are 0 bytes.

Funny I have the same problem ~CoXXXX.tmp with 0 size.
Windows 7 Pro 64Bit

EDIT:

and I confirm that this is related to pressing F7 in S4A3

I wonder: Is it related to last change in AutoItWrapper or Au3Stripper ?

 

Edited August 28, 2015 by mLipok

[mLipok]

What is the easiest way to make sure all files have the same format (how to convert) and what format is recommended?

expandcollapse popup

;~ #AutoIt3Wrapper_Version=B
#include <File.au3>
#include <array.au3>



_Convert_Au3_Files_Encoding()

Func _Convert_Au3_Files_Encoding()

    ; beta include
    Local $sBetaFolder = RegRead("HKEY_LOCAL_MACHINE\SOFTWARE" & ((@OSArch = 'X64') ? "\Wow6432Node" : "") & "\AutoIt v3\AutoIt", "InstallDir") & '\Beta\Include\'
    If Not @error Then _Encoding_Converter($sBetaFolder)

    ; relase inculde
    _Encoding_Converter(RegRead("HKEY_LOCAL_MACHINE\SOFTWARE" & ((@OSArch = 'X64') ? "\Wow6432Node" : "") & "\AutoIt v3\AutoIt", "InstallDir") & '\Include\')

    _Encoding_Converter(@ScriptDir)
    _Encoding_Converter('..\MyProject1\')
    _Encoding_Converter('..\MyProject2\')
    _Encoding_Converter('..\MyProject3\')
    _Encoding_Converter('..\MyProject4\')
;~  _Encoding_Converter('')
;~  _Encoding_Converter('')

;~  ; and your own UDF folder
;~  Local $sFolderOwnUDF = FileSelectFolder('Choose your UDF folder','')
;~  if not @error then  _Encoding_Converter($sFolderOwnUDF)

EndFunc   ;==>_Convert_Au3_Files_Encoding


Func _Encoding_Converter($sFolder, $iENCODING = $FO_UTF8)
    ConsoleWrite($sFolder & @CRLF)
    Local $aIncludeList = _FileListToArray($sFolder, '*.au3', $FLTA_FILES, True)
    if @error then Return
    Local $sFileContent
    Local $hFile
    Local $iFile_Old_Encoding = -1
    Local $iFile_New_Encoding = -1
    Local $iFile_Attrib = ''
    Local $bFile_AttribChanged = False
    For $iFile = 1 To $aIncludeList[0]
        $iFile_Old_Encoding = FileGetEncoding($aIncludeList[$iFile])
        $iFile_Attrib = FileGetAttrib($aIncludeList[$iFile])
        If StringInStr($iFile_Attrib, 'R') Then
            FileSetAttrib($aIncludeList[$iFile], '-R')
            $bFile_AttribChanged = True
        Else
            $bFile_AttribChanged = False
        EndIf
        If $iFile_Old_Encoding = $iENCODING Then
            ConsoleWrite('> ')
        Else
            ConsoleWrite('! ')
            $sFileContent = FileRead($aIncludeList[$iFile])
;~          ClipPut($sFileContent)
;~          $sFileContent = ClipGet()

            FileDelete($aIncludeList[$iFile])
            $hFile = FileOpen($aIncludeList[$iFile], $FO_OVERWRITE + $iENCODING)
            ConsoleWrite(FileWrite($hFile, $sFileContent) & ' ')
            FileClose($hFile)
        EndIf
        If $bFile_AttribChanged Then FileSetAttrib($aIncludeList[$iFile], '+R')
        $iFile_New_Encoding = FileGetEncoding($aIncludeList[$iFile])
        ConsoleWrite('$iFile_[Old/New]_Encoding=[ ' & $iFile_Old_Encoding & ' / ' & $iFile_New_Encoding & ' ] ' & $aIncludeList[$iFile] & @CRLF)
    Next
EndFunc   ;==>_Encoding_Converter

 

Edited August 28, 2015 by mLipok
script cleanup

[Allow2010]

thanks, will try that...

[Allow2010]

Any idea which utility does that?

Jos

it happends on F5 in SciTE...

[Jos]

it happends on F5 in SciTE...

Which ran AutoIt3Wrapper and that in turn AU3Check & AutoIt3.exe or also Tidy?

Jos

[Allow2010]

i tested with a script that has only one line

MsgBox(0,"","")

i just renamed tidy.exe and au3check.exe and press f5 and i get two temp files...

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "Z:\test.au3" /UserParams    
+>23:41:53 Starting AutoIt3Wrapper v.15.729.1555.6 SciTE v.3.6.0.0   Keyboard:00000407  OS:WIN_10/  CPU:X64 OS:X64    Environment(Language:0407)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\User\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\User\AppData\Local\AutoIt v3\SciTE
! *** AU3CHECK Error: *** Skipping AU3Check: C:\Program Files (x86)\AutoIt3\au3check.exe Not Found !

>Running:(3.3.14.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "Z:\test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
+>23:41:53 AutoIt3.exe ended.rc:0
+>23:41:53 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 1.297

 

Edited August 28, 2015 by Allow2010

[mLipok]

I do test:

1. rename Au3Check.exe to -Au3Check.exe
2. press CTRL+F5 in S4A3

here is output from console:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /Prod /AU3check /in "Z:\test_ObjEvent.au3"
+>08:02:39 Starting AutoIt3Wrapper v.15.729.1555.6 SciTE v.3.6.0.0   Keyboard:00000415  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0415)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\user\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user\AppData\Local\AutoIt v3\SciTE 
! *** AU3CHECK Error: *** Skipping AU3Check: C:\Program Files (x86)\AutoIt3\au3check.exe Not Found !

+>08:02:39 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.5599

How you see there was only AutoIt3Wrapper launched.
So I do a further test. I added In AutoIt3Wrapper.au3  some MsgBox for testing:

Func __TempFile($sDirectoryName = @TempDir, $sFilePrefix = "~", $sFileExtension = ".tmp", $iRandomLength = 7)
    ; his will create 2 files but this is to avoid same name when ran multiple times
    MsgBox(0, 'Test 1', 'Check you root directory for *.tmp files')
    Local $TempFile = _WinAPI_GetTempFileName($sDirectoryName, $sFilePrefix)
    If $sFileExtension <> ".tmp" Then
        FileCopy($TempFile, $TempFile & $sFileExtension, 1)
    Else
        $sFileExtension = ""
    EndIf
    MsgBox(0, 'Test 2 - Check Again', 'File: ' & $TempFile & $sFileExtension)
    Return $TempFile & $sFileExtension
EndFunc   ;==>__TempFile

it seams that the problem starts here:

Global $TempConsOut = __TempFile("", "~ConOut")

 

Edited August 29, 2015 by mLipok

[Jos]

Currently on the road, but will have some time in the next couple of days to have a look at this in detail.

Jos

[Allow2010]

thanks!