The problem with diacritics in Beta

[mLipok]

  On 5/6/2015 at 12:39 AM, jchd said:

Oops time: need to Zzzz a bit, it's 02:38 here and me answering on the fly is irresponsible!

​Right at me the same way.
I also need to rest.
We'll see in the morning what others come up with

[jchd]

mLipoK,

I don't have any issue with the file encoding test script you posted above (which I used verbatim except that I first opened a new file in SciTE and switched to UTF8 with BOM, then pasted your code there), neither with run (F5):
 

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\jc\Documents\AutoMAT\tmp\tst.au3" /UserParams    
+>08:39:51 Starting AutoIt3Wrapper v.15.503.1200.1 SciTE v.3.5.4.0   Keyboard:0000040C  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:040C)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\jc\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\jc\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.13.20)  from:C:\Program Files (x86)\AutoIt3\Beta  input:C:\Users\jc\Documents\AutoMAT\tmp\tst.au3
+>08:39:51 AU3Check ended.rc:0
>Running:(3.3.13.20):C:\Program Files (x86)\AutoIt3\Beta\autoit3.exe "C:\Users\jc\Documents\AutoMAT\tmp\tst.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
+>08:40:02 AutoIt3.exe ended.rc:0
+>08:40:02 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 11.49

nor with compile (F7):
 

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /NoStatus /prod /in "C:\Users\jc\Documents\AutoMAT\tmp\tst.au3"
+>08:41:09 Starting AutoIt3Wrapper v.15.503.1200.1 SciTE v.3.5.4.0   Keyboard:0000040C  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:040C)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\jc\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\jc\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.13.20)  from:C:\Program Files (x86)\AutoIt3\Beta  input:C:\Users\jc\Documents\AutoMAT\tmp\tst.au3
+>08:41:09 AU3Check ended.rc:0
>Running Au3Stripper (15.503.1200.0)  from:C:\Program Files (x86)\AutoIt3\SciTE\Au3Stripper cmdline: /Beta
+> 0 Au3Stripper v15.503.1200.0 finished merging 4217 lines of code, stripped 1195 comment lines and Merged 7 Continuation lines.
+>        Created:C:\Users\jc\Documents\AutoMAT\tmp\tst_stripped.au3 with 3015 lines.
+>08:41:10 Au3Stripper ended.rc:0
>Running AU3Check (3.3.13.20)  from:C:\Program Files (x86)\AutoIt3\Beta  input:C:\Users\jc\Documents\AutoMAT\tmp\tst_stripped.au3
+>08:41:10 AU3Check ended.rc:0
>Running:(3.3.13.20):C:\Program Files (x86)\AutoIt3\Beta\aut2exe\aut2exe.exe  /in "C:\Users\jc\Documents\AutoMAT\tmp\tst_stripped.au3" /out "C:\Users\jc\AppData\Local\AutoIt v3\Aut2exe\~AU3lbgytcl.exe" /nopack /comp 2
+>08:41:11 Aut2exe.exe ended.C:\Users\jc\AppData\Local\AutoIt v3\Aut2exe\~AU3lbgytcl.exe. rc:0
+>08:41:11 Created program:C:\Users\jc\Documents\AutoMAT\tmp\tst.exe
+>08:41:11 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.641

I note that in the console output you pasted I don't have the 2nd line (@@Debug...). Also this was output for Compile, not Run.

Yet, every time I can see the output line shown as expected:  Nie można uruchamiać programu wielokrotnie w tym samym czasie

 

[mLipok]

Despite this problem it is in Au3Stripper or Aut2exe.
"One of them works differently from the other."

Ultimately, it may cause in the future new questions and problems of other members of our community.

 

Another problem that I have now about 20 different own UDFs. Changing the master file does not help right away I have to change all the files.
At the moment I have not done it yet, I will inform you of the progress and possible further results, in this thread.

Edited May 6, 2015 by mLipok
wording

[jchd]

Sorry but I still don't see a problem. My installation is "vanilla" and works as expected AFAICT.

[mLipok]

I make some converter:

#AutoIt3Wrapper_Version=B
#include <File.au3>
#include <array.au3>

_Encoding_Converter(@ScriptDir)
_Encoding_Converter(RegRead("HKEY_LOCAL_MACHINE\SOFTWARE" & ((@OSArch = 'X64') ? "\Wow6432Node" : "") & "\AutoIt v3\AutoIt", "InstallDir")  & '\Beta\Include\')

; and my own UDF folder
_Encoding_Converter('z:\TOOLs\Macro\PROGRAMY\__UDF\')


Func _Encoding_Converter($sFolder)
    Local $aIncludeList = _FileListToArray($sFolder, '*.au3', $FLTA_FILES, True)
;~  _ArrayDisplay($aIncludeList,'$aIncludeList')
    Local $sFileContent
    Local $hFile
    For $iFile = 1 To $aIncludeList[0]
        $sFileContent = FileRead($aIncludeList[$iFile])
        $hFile = FileOpen($aIncludeList[$iFile], $FO_OVERWRITE +  $FO_UTF8)
        FileWrite($hFile, $sFileContent)
        FileClose($hFile)
    Next
EndFunc   ;==>_Encoding_Converter

 

Here is my next  observation:

  Quote

>"C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /NoStatus /prod /in "Z:\TOOLs\Macro\PROGRAMY\KK_EPU\KK_EPU.au3"
+>09:43:30 Starting AutoIt3Wrapper v.14.801.2025.6 SciTE v.3.5.2.0   Keyboard:00000415  OS:WIN_7/Service Pack 1  CPU:X64 OS:X86    Environment(Language:0415)
+>         SciTEDir => C:\Program Files\AutoIt3\SciTE   UserDir => C:\Users\user\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user\AppData\Local\AutoIt v3\SciTE
>Running Tidy (14.801.2025.3)  from:C:\Program Files\AutoIt3\SciTE\tidy
+>09:43:30 Tidy ended.rc:0
>Running AU3Check (3.3.13.19)  from:C:\Program Files\AutoIt3  input:Z:\TOOLs\Macro\PROGRAMY\KK_EPU\KK_EPU.au3
! Au3check doesn't support input files encoded as UTF8 with BOM: Z:\TOOLs\Macro\PROGRAMY\__UDF\KK_constants.au3
! Au3check doesn't support input files encoded as UTF8 with BOM: Z:\TOOLs\Macro\PROGRAMY\__UDF\KK_Licences.au3

 

 

EDIT:
Small change

;~      $hFile = FileOpen($aIncludeList[$iFile], $FO_OVERWRITE +  $FO_UTF8)
        $hFile = FileOpen($aIncludeList[$iFile], $FO_OVERWRITE +  $FO_UTF8_NOBOM )

and some CleanUp, and finally here is  Au3_encoding_converter.au3 working script to change encoding.

Edited May 6, 2015 by mLipok

[czardas]

I'm quite convinced that it's a bug of some kind: deleting a comment changes the behaviour of the interpreter! Deleting other parts of the code also changes the behaviour and I don't see any unicode in the modifications that could trigger such a change.

Edited May 6, 2015 by czardas
Add more info

[mLipok]

  On 5/6/2015 at 8:00 AM, czardas said:

I'm quite convinced that it's a bug of some kind: deleting a comment changes the behaviour of the interpreter!

​I agree.

I think it is Au3Stripper that mixes character encoding, while combining different files.
By creating a linked file AU3, should take the character encoding, as it is in the main AU3 file .

But let's leave the last word to Dev, especially for @Jos.

 

Edited May 6, 2015 by mLipok
wording

[czardas]

I'm not sure how that can be the issue: there are no includes in your first example. I need to perform a deeper analysis.

Edited May 6, 2015 by czardas

[mLipok]

  On 5/6/2015 at 8:11 AM, czardas said:

I'm not sure how that can be the issue: there are no includes in your first example.

​

  On 5/3/2015 at 11:44 PM, czardas said:

Could you give a very short code example that demonstrates the problem, please?

​

  On 5/4/2015 at 3:27 AM, mLipok said:

The problem arose when using Au3Stripper

#Au3Stripper_Parameters=/SO

But I do not think it is related to it.

I strip manualy my script file from 9000 line to 1441 lines.
First 1420 lines comes from Standard UDF.

​

CONCLUSION: it was a one of my programs

EDIT:

Additional explanations:

The problem arose when using Au3Stripper

#Au3Stripper_Parameters=/SO

Au3Stripper prepared the resulting file scirpt_stripped.au3 (some about 10,000 lines)
Then additionally: To make smallest reproduction script I strip manualy strip my script file to 1441 lines. (just deleting lines in SciTE)

 

Au3_encoding_converter.au3

Edited May 6, 2015 by mLipok

[czardas]

In your conclusion (It was one of my programs), are you now saying you don't think it is a bug?

[czardas]

  On 5/6/2015 at 8:17 AM, mLipok said:

​CONCLUSION: it was a one of my programs

​Are you now saying it is not a bug?

Arghh - what happened to that forum post delete button.

Edited May 6, 2015 by czardas

[jchd]

Indeed, Au3Check doesn't cope with UTF8-encoded include files. That needs fixing someday because it's a real limitation: I never use it for that precise reason.

More broadly, AutoIt and its tools should now support transparently the most widely used Unicode file encodings (UTF8, UTF16) over the full Unicode range, main scripts and includes (standard or user). Of course use of a BOM considerably helps disambiguate things for correct processing and should be strongly encouraged.

I've found that for small to moderately large text input, converting everything read in UTF32-LE internally is the easiest. I mean for C or similar programs like AutoIt tool suite.

[mLipok]

  On 5/6/2015 at 8:29 AM, jchd said:

Indeed, Au3Check doesn't cope with UTF8-encoded include files. That needs fixing someday because it's a real limitation: I never use it for that precise reason.

More broadly, AutoIt and its tools should now support transparently the most widely used Unicode file encodings (UTF8, UTF16) over the full Unicode range, main scripts and includes (standard or user). Of course use of a BOM considerably helps disambiguate things for correct processing and should be strongly encouraged.

I've found that for small to moderately large text input, converting everything read in UTF32-LE internally is the easiest. I mean for C or similar programs like AutoIt tool suite.

​Thanks for all your explanation @jchd.
Thanks also to @BrewManNH for his accurate comments.

At the moment, thanks to you guys, I solved the problem with my programs.

 

 

and in the meantime, continuing thread of "mixing AU3 files" by the program Au3Stripper

  On 5/6/2015 at 8:24 AM, czardas said:

​Are you now saying it is not a bug?

Arghh - what happened to that forum post delete button.

​So what you finally now think - is it a bug in your opinion ?

 

Of course it is still valid:

  On 5/6/2015 at 8:05 AM, mLipok said:

​But let's leave the last word to Dev, especially for @Jos.

 

​

[czardas]

I still think there is a bug and I never use au3Stripper myself. Deleting a comment should not change the way the code behaves. It seems most likely to have to do with encoding changes although I haven't thoroughly checked everything. If this is the case, it is still unclear to me what triggers the changes because deleting other parts of the code also fixes the unicode in the edit control. That in itself is a conflict which needs explaining (no explanation for this has been presented). The current stable release works out of the box.

Edited May 6, 2015 by czardas

[Jos]

Couple of remarks without reading all details of all posts:

Au3Stripper will create an outputfile that has the same encoding as the main sourcefile.
Au3Check indeed doesn't support UTFxx with BOM, but AutoIt3Wrapper "works around that" because it will copy the original main script into a tempfile which is ascii encode and then runs the au3check against that. The reported errors are translated back for the original file/linenumber.

EDIT: the new au3check in Beta does support UTF8 with BOM.

Jos

Edited May 9, 2015 by Jos

[czardas]

Sorry for the delay in looking at this again (British election ). I just tried jchd's suggestion and it seems to fix the issue. However it is still very odd that deleting one region, or alternatively another region of code, would cause the same buggy behaviour. I conclude that it cannot be being triggered by the actual characters within the deleted regions, but has to be caused by something else (perhaps file size). If this is a known (or expected) result of not following jchd's exact solution, then I guess there is no cause for concern. On the other hand, this might bite someone in the posterior at some point in the future.

I also don't believe this has anything to do with SciTE or any addons, but rather it has to do with recent changes in the beta: the stable release is not affected. It does concern me that previous beta releases were not behaving this same way (according to mLipok). I still have to check the change logs.

Edited May 9, 2015 by czardas

[mLipok]

@czardas

Check here:

It is fixed now

I mean I can not again produce this kind of strange file, with latest Au3Stripper.

[czardas]

Okay, I'm glad to hear this. I wish I had more time to spend on the forum.