UTF8/UTF16 Testing

[Jon]

Looking at fixing the memory overflow in this report https://www.autoitscript.com/trac/autoit/ticket/2901

Found a few logic errors around creating blank/1 char files and BOMs - especially when opening, closing, reopening those files. They could ignore the user flags. I've made some changes that I'll test myself tomorrow but if anyone can test the attached exe beforehand that would be great.

 

Edited July 8, 2015 by Jon
Removed attachment - use 3.3.13.21 Beta

[jaberwacky]

Maybe I'm missing something but when I run this exe, it then asks me for a script.  I choose a script and I get an error message which says:

#include <StringConstants.au3>

Error: Error opening the file.

 

I'm on Win 7 x64 with the latest AutoIt prod and beta.  What else you need? Or did I totally do this wrong?

 

Edit: just in case, here is the script:

expandcollapse popup

#include <StringConstants.au3>

#include <Misc.au3>

Const $title = "Volume Mixer"

Switch _Singleton(@ScriptName, 1)
  Case 0
    WinClose($title)
    Exit
EndSwitch

Opt("WinDetectHiddenText", True)

_main()

Func _main()
  Local Const $volume_mixer_pid = Run("SndVol.exe")

  Switch $volume_mixer_pid <> -1
    Case True
      WinWaitActive($title)
      
      Local Const $default_width = WinGetPos($title)[2]
      
      Local Const $control_threshold = 3      

      Local $control_count = 0

      While ProcessExists($volume_mixer_pid)        
        Switch _get_text($control_count)
          Case True
            $control_count += 1
            
          Case False
            Switch $control_count >= $control_threshold
              Case True   
                _resize_window($default_width, $control_count, $control_threshold)                                
                
                If @error Then 
                  Return SetError(1, 0, False)
                EndIf
                
                $control_count = 0
            EndSwitch
        EndSwitch
      WEnd 
  EndSwitch
EndFunc

Func _resize_window(Const $default_width, Const $control_count, Const $control_threshold)                        
  Local Const $x = WinGetPos($title)
  
  If @error Then 
    Return SetError(1, 0, False)
  EndIf
                  
  Local Const $new_width = $default_width + (($control_count - $control_threshold) * 110)    
                
  Switch $new_width <= (@DesktopWidth - $x[0])
    Case True
      Switch WinMove($title, '', Default, Default, $new_width)
        Case 0
          Return SetError(2, 0, False)
      EndSwitch
  EndSwitch
  
  Return False
EndFunc

Func _get_text(Const $control_count)
  Local Const $text = StringStripWS(WinGetText($title), ($STR_STRIPLEADING + $STR_STRIPTRAILING + $STR_STRIPSPACES))
  
  Return StringInStr($text, "Volume for", 0, ($control_count + 1)) > 0
EndFunc

 

Edited July 8, 2015 by jaberwacky
script

[mLipok]

Looking at fixing the memory overflow in this report https://www.autoitscript.com/trac/autoit/ticket/2901

Found a few logic errors around creating blank/1 char files and BOMs - especially when opening, closing, reopening those files. They could ignore the user flags. I've made some changes that I'll test myself tomorrow but if anyone can test the attached exe beforehand that would be great.

 

I'll check over the weekend (I am currently very busy working)

Already checked - results in track ticket.

Edited July 8, 2015 by mLipok

[Jon]

Your script works fine for me even on the last beta. I see the text with the correct accents in it in each MsgBox and it's in UTF8. You did save the script in UTF8 didn't you - rather than codepage?

Edited July 8, 2015 by Jon

[jaberwacky]

Ohhhh, yeah, knew I was in over my head.  Sorry!

[mLipok]

Your script works fine for me even on the last beta. I see the text with the correct accents in it in each MsgBox and it's in UTF8. You did save the script in UTF8 didn't you - rather than codepage?

Yes

I used one of UTF8 encoding in my Au3 scirpt file

 

[Jon]

Can anyone else reproduce the issue with this script?

expandcollapse popup

#include <FileConstants.au3>
#include <MsgBoxConstants.au3>

_Test()

Func _Test()
    Local $sFileFullPath = @ScriptDir & '\test_file_encoding.txt'

    ; FileWrite with $FO_UTF8_NOBOM
    _FileWrite_StringContent($sFileFullPath, 'poniedziałek, wtorek, środa, czwartek, piątek, sobota, niedziela')

    ; FileRead with FileGetEncoding()
    Local $sResult = _FileRead_StringContent($sFileFullPath)
    MsgBox(0, '$sResult', $sResult)

    ; FileRead - simply with out : FileOpen + FileClose
    Local $sResult2 = FileRead($sFileFullPath)
    MsgBox(0, '$sResult2', $sResult2)

EndFunc   ;==>_Test

Func _FileWrite_StringContent($sFileFullPath, $sContent)
    Local $hFile = FileOpen($sFileFullPath, $FO_OVERWRITE + $FO_CREATEPATH + $FO_UTF8_NOBOM)
    FileWrite($hFile, $sContent)
    FileClose($hFile)
EndFunc   ;==>_FileWrite_StringContent

Func _FileRead_StringContent($sFileFullPath)
    ConsoleWrite(FileGetEncoding($sFileFullPath) & @CRLF)
    Local $hFile = FileOpen($sFileFullPath, $FO_READ + FileGetEncoding($sFileFullPath))
    Local $sContent = FileRead($hFile)
    FileClose($hFile)
    if IsBinary($sContent) then
        ConsoleWrite('IsBinary -- converting to string')
        $sContent = BinaryToString($sContent,1 )
    EndIf
    Return $sContent
EndFunc   ;==>_FileRead_StringContent

 

[mLipok]

Can anyone else reproduce the issue with this script?

One remarks:
using AutoIt 3.3.13.20 and AutoIt3.exe from OP 

and with script encoded in UTF8

I attach this script  test_UTF.au3.

EDIT:

I just notice there is new Beta version 3.3.13.21

@Jon please precise how you exactly want to test (which Au3 version ? )

 

Edited July 8, 2015 by mLipok

[Jon]

The beta 3.3.13.21.

That script worked fine here too.

Edited July 8, 2015 by Jon

[mLipok]

using 3.3.13.21 works correctly

 

[mLipok]

btw.

using this:

ConsoleWrite('poniedziałek, wtorek, środa, czwartek, piątek, sobota, niedziela' & @CRLF)

I get in console this:

poniedziaԥk, wtorek, ݲoda, czwartek, piڴek, sobota, niedziela
+>14:19:55 AutoIt3.exe ended.rc:0
+>14:19:55 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.5069

 

is this issue related to Au3 or SciTE Console ?

 

[Jon]

That's a console issue. Not unique to SciTe, but ConsoleWrite can't output utf8.

[mLipok]

 

That's a console issue. Not unique to SciTe, but ConsoleWrite can't output utf8.

It is not clear to me.

so I ask in this way:

Is the repair this problem is within your possibilities ?
If so, could I ask you to repair this ?
I think it's one of the last problems that are now in support for UTF, I mean in use AutoIt3 + SciTE4AutoIt (recently was many changes made in Au3Stripper and Au3Check)

 

[Jon]

I've not looked at it for years but if I recall it is an OS limitation of console windows they only work in the codepage of the host OS.

[jchd]

mLipok,

Place these two line in SciTEUser.properties

code.page=65001
output.code.page=65001

(to be honest I don't remember which one applies to the SciTE console, but anyway). Then use this version to display UTF8:

; Unicode-aware ConsoleWrite
Func __ConsoleWrite($s)
    ConsoleWrite(BinaryToString(StringToBinary($s, 4), 1))
EndFunc   ;==>__ConsoleWrite

 

[mLipok]

code.page=65001
output.code.page=65001

I was used this before

 

but this is

; Unicode-aware ConsoleWrite
Func __ConsoleWrite($s)
    ConsoleWrite(BinaryToString(StringToBinary($s, 4), 1))
EndFunc   ;==>__ConsoleWrite

el perfecto

Edit:

Still I wondering if AutoIt can not detect the fact that the script is encoded in UTF and accordingly immediately change this string ?

Edited July 8, 2015 by mLipok

[jchd]

You're being confused by the situation here.

Whatever source file encoding you use, the (native) AutoIt strings are encoded in UCS2-LE (look at it as if it was UTF16-LE) which uses only one 16-bit encoding unit per Unicode character. The corresponding type in DllStruct is wstr, the type that Windows uses for Unicode-aware applications.

Now let's look at ConsoleWrite. The allowable encodings for consoles are all "multibytes", that is either your Windows charset (aka ANSI even if this is a seriously bending of the "norms") or UTF8, but always using bytes as encoding units: ANSI uses 1 byte = 1 ANSI char while UTF8 uses 1-4 bytes = 1 Unicode char.

Anyway, to be able to display sensible strings to the console, you have to make a match between the encoding used for the strings to be displayed and the encoding that the console expects. By default, ConsoleWrite converts native (UTF16-LE Unicode) AutoIt strings to ANSI (the expected charset by consoles) but as you've seen you can make the console interpret strings in UTF8 by setting the charset to 65001. What _ConsoleWrite does is convert native AutoIt strings to UTF8 and this way everything works well.

I'm one tenant that every part in AutoIt tool chain should work in Unicode only, but to answer your last question, there is no "detection" by AutoIt, just historical conversions.

Edited July 9, 2015 by jchd
Typos, sorry!

[mLipok]

Thanks for such a broad explanation.