Latest Beta

[jchd]

There is a bug in StringToASCIIArray with UTF-16 parameter: values in the returned array are ANDed with 0x00FF.

Replicate script:

expandcollapse popup

;
;; correct console display needs decent programming font with Unicode characters (e.g. DejaVu Sans Mono) and Unicode codepage in SciTE:
;
; in SciTEGlobalProperties change to code.page=65001:
;~ # Internationalisation
;~ # Japanese input code page 932 and ShiftJIS character set 128
;~ #code.page=932
;~ #character.set=128
;~ # Unicode
;~ code.page=65001                  <<<<<<<<<<<<<<<<<<<<<<<<<<<
;~ #code.page=0


; $str = e-grave 0x00E8, e-sharp 0x00E9, e-circumflex 0x00EA, space 0x0020, A-caron 0x01CD, a-caron 0x01CE, I-caron 0x01CF, fi ligature 0xFB01

Local $str = "éêè ǍǎǏfi"

__ConsoleWrite('StringLen("' & $str & '"): ' & StringLen($str) & @LF)   ; correct but only for characters < 0x010000 (~UCS-2 charset)

Local $a = StringToASCIIArray($str, Default, Default, 0)                ; length correct but values incorrectly masked with 0x00FF
Local $b = StringSplit($str, '', 2)
__ConsoleWrite('Glyph ' & @TAB & _ArrayToString($b, @TAB) & @LF)
ConsoleWrite('UTF-16 ' & @TAB & _ArrayToString($a, @TAB) & @LF)
For $i = 0 To UBound($a) - 1
    $a[$i] = Hex($a[$i], 4)
Next
ConsoleWrite('  ' & @TAB & _ArrayToString($a, @TAB) & @LF)

;; UTF-8 is all correct
$a = StringToASCIIArray($str, Default, Default, 2)                      ; length and contents are correct
ConsoleWrite('UTF-8 ' & @TAB & _ArrayToString($a, @TAB) & @LF)
For $i = 0 To UBound($a) - 1
    $a[$i] = Hex($a[$i], 2)
Next
ConsoleWrite('  ' & @TAB & _ArrayToString($a, @TAB) & @LF)

Exit

Func __ConsoleWrite($sText)
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sText, "int", -1, _
                                "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    Local $tText = DllStructCreate("char[" & $aResult[0] & "]")
    DllCall("Kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sText, "int", -1, _
                            "ptr", DllStructGetPtr($tText), "int", $aResult[0], "ptr", 0, "ptr", 0)
    ConsoleWrite(DllStructGetData($tText, 1))
EndFunc ;==>__ConsoleWrite

You may have to follow included instruction for correct display.

Here's the result:

;>Running:(3.3.5.4):C:\Program Files\AutoIt3\beta\autoit3.exe "D:\XLequit\AutoMAT\Test\try.au3" 
;StringLen("éêè ǍǎǏfi"): 8
;Glyph  é ê   è Ǎ   ǎ Ǐ fi
;UTF-16     233 234 232 32 205 206 207 1
 ;  00E9 00EA 00E8 0020 00CD 00CE 00CF 0001
;UTF-8  195 169 195 170 195 168 32  199 141 199 142 199 143 239 172 129
 ;  C3  A9  C3  AA  C3  A8  20  C7  8D  C7  8E  C7  8F  EF  AC  81

XP SP3 x86 (if that matters)

Edit: I didn't find the right markup that won't destroy spaces alignment in the result above. Giving up trying for today.

Edited February 15, 2010 by jchd

[jchd]

This is not the same issue as previous post!

I follow up here to Valik answer

Yes, there are issues with muli-'code units' for Unicode characters >= 0x010000 and, like you said, they are probably AutoIt-wide issues. It seems that string native functions use a class (or equivalent) where one character = one 16-bit codeunit in all cases. So besides the different terminology, current AutoIt UTF-16 is in fact essentially UCS-2.

There is nothing wrong with this as long as the product doesn't claim Unicode compliance under UTF-16 representation, or at least that the documentation clearly says so. Changing all the documentation to replace Unicode by UCS-2 would be technically correct, but certainly potentially more confusing to most users. A clear note in the features and datatype sections could be enough to warn concerned people.

Anyway AutoIt and its 'natural companion' (customized SciTE) can't be said to be horribly wrong. As I previously mentionned, there are little editors that treat full Unicode range under UTF-16 correctly, even today. FYI, I found only one that does: the VC++ editor (I use 2008 Express but no doubt newer versions do as well) but there are obviously many others.

It's likely that changing AutoIt core to support the full UTF-16 encoding with multi code units could reveal a delicate task and lead to even more issues (DllCall, DllStructs, ...). OTOH AutoIt doesn't appear to be under much pressure from users to support the 'exotic' (said with full respect to people who live there) character ranges, the new huge asian block or language tags, for instance.

Again, I'm not asking for full support tomorrow! I simply wanted to let you know.

I someone needs a ready-to-use script + font + instructions to look at that more closely, just ask, I'll be pleased to help.

[Valik]

Jon is retarded. I knew that already. Now you do, too. I've fixed his silly casts.

[jchd]

;-) OK this one was most probably very easy. BTW I suppose you mean the ushort -> uchar emasculation (Jon called the encoding joys "balls breaking" for a reason!).

[Valik]

Yes, I was referring to that. He was casting everything through UCHAR instead of UCHAR for ANSI/UTF-8 and USHORT for UTF-16.

[Jon]

Yes, I was referring to that. He was casting everything through UCHAR instead of UCHAR for ANSI/UTF-8 and USHORT for UTF-16.

omg, I did the hard stuff and forgot the most basic cast. FFS.

[Valik]

Yeah... that was some messed up shit.

[Jon]

AutoIt v3.3.5.5 (Beta) Released:

There has been a significant rewrite of the Send/ControlSend code to better cope with Unicode characters. Those using characters <127 (USA/English/UK/etc) shouldn't notice any difference unless something has gone very wrong. Extended/Unicode users should test this release to see if there ary any improvements/disasters...

AutoIt:

- Fixed #1475: TrayItemSetState($Value, $Tray_Checked) incorrectly enables a disabled TrayItem.

- Fixed: StringToASCIIArray() was incorrectly truncating UTF-16 values at 8 bits.

UDFs:

- Fixed #1445: Documentation updated for _MemGlobalFree().

- Fixed #1469: _GDIPlus_BitmapCloneArea() documentation updated.

- Fixed #1466: _GUICtrlEdit_GetLine() returns an unexpected character.

The following changes are script breaking changes:

Discuss the beta here.

Report issues here.

Download here.

[GEOSoft]

It's broken Jon.

On the first script I tried it returned an Error of Unknown Function Name for StringRegExpReplace()

[Jon]

Uploading a fixed version now.

[GEOSoft]

Uploading a fixed version now.

Thanks.

EDIT: Working fine now.

Edited February 28, 2010 by GEOSoft

[wraithdu]

I guess that would be 3.3.5.6 then. Might want to update the announcements as well.

[Jon]

AutoIt v3.3.7.0 (Beta) Released:

AutoIt:

- Fixed #1599: TraySetItemText() regression for default menu item.

- Fixed #1282: WinMove before GUICtrlSetPos during GUICreation.

- Fixed #1397: Bad HotKeySet() not detected on keyboard as Russian one.

- Fixed #1531: default txtcolor for iput edit list combo updown with black theme.

- Fixed #1617: GuiCreate failure after GuiDelete can lead to a loop on Autoit exit.

- Fixed #1596: GUICtrlSetPos() default = no change.

- Fixed #1485: Crash on ContinueCase.

- Fixed #1626: TCPRecv()/TCPSend() doc about Unicode transmission.

- Fixed #1653: Doc for WM_KEYLAST Windows 2000 message.

- Fixed #1669: StringRegExpReplace() doc about doubling of "\" in replace string.

- Fixed #1673: WinGetProcess() doc example.

- Fixed #1677: Invalid ContinueLoop with multiple levels.

- Fixed #1684: FileRead() binary read memory allocation error.

- Fixed #1685: BitRotate() shift parameter.

- Fixed #1734: GUICtrlCreateAVI() crash with negative subfield.

- Fixed #1923: Memory consumption while FileOpen/FileClose.

- Fixed #1883: IsHWnd(), Return Value. (Nitpick: Value = Bool).

AutoItX:

- Fixed #1686: AU3_PixelSearch crashes.

UDFs:

- Fixed #1040: _ScreenCapture_Capture(): GDI object leak with cursor capture.

- Added: _DebugSetup() can report to a Notepad window.

- Added #1371: Allow Default keyword in _TempFile().

- Added #1527: Test example to have doc example working.

- Added #1636: _Security__LookupAccountSID() for Remote Systems.

- Added #1569: _ArraySearch() $iPartial (->$iCompare) extended to match on variables of same type.

- Added #1557: VK_xBUTTON in Constants.au3.

- Fixed #1542: _DebugSetup() closing when use with GUI handling events.

- Fixed #1549: _SQLite_Escape() crash for strings > 64K (> 3.3.0.0).

- Fixed #1517: _GUICtrlListView_simpleSort with checkbox.

- Fixed #1588: AUtoIt3.exe stay active for all processes using the Window report.

- Fixed #1615: _GUICtrlTreeView_SetStateImageIndex() with index = 0.

- Fixed #1620: _DebugOut() or _DebugReportVar() containing '.

- Fixed #1513: Allow _GUI...() to use notification callback (LPSTR_TEXTCALLBACK).

- Fixed #1608: _Crypt_EncryptFile() for file >1Mb.

- Fixed #1644: _InetMail() with Windows Live mail.

- Fixed #1453: _Net_Share_ShareCheck return always 0.

- Fixed #1664: _GUICtrlTab_GetItem() does not return text.

- Fixed #1671: _WinAPI_WideCharToMultiByte() doc.

- Fixed #1672: _WinAPI_GetObject() reference ANSI version.

- Fixed #1665: _ScreenCapture_CaptureWnd() when running with Aero theme.

- Fixed #1689: _Debug...() cannot be obfuscated.

- Fixed #1712: _FileWriteFromArray() crash on array.

- Fixed #1754: _PathFull() optional parameter doc.

- Fixed #1756: _GDIPlus_Startup() Flaw on error.

- Changed: _SQLite 3.6.22 -> 3.7.2.0

Au3Check:

- Added: #forcedef directive to force var definition as after Assign().

- Added: -w 7 to check ByRef parameter passing.

- Fixed: Const Enum wrong ERROR.

- Fixed: Global declared in func not detected.

- Fixed #1051: no ERROR if keyword not followed by a separator as Local$a.

[jvanegmond]

Think about putting the changes there as:

- Fixed #1599: TraySetItemText() regression for default menu item.

- Fixed #1282: WinMove before GUICtrlSetPos during GUICreation.

- Fixed #1397: Bad HotKeySet() not detected on keyboard as Russian one.

with [.ticket] tags. The feature is there, might as well use it.

A simple regular expression will convert it.

Edited May 23, 2011 by Manadar

[UEZ]

I'm getting an em when compiling with this beta:

"The program can't start because MSVCP100.dll is missing from your computer. Try reinstalling the program to fix this problem."

Br,

UEZ

[Jon]

I'm getting an em when compiling with this beta:

"The program can't start because MSVCP100.dll is missing from your computer. Try reinstalling the program to fix this problem."

Hmm, sounds like some static library settings got messed up in the compiler upgrade. Which .exes do this?

[UEZ]

Aut2exe.exe is causing the error.

[Jon]

Uploaded a fixed version 3.3.7.1

[UEZ]

Seems to work properly now. Btw, the exe size is smaller now...

Thanks,

UEZ

Edit: you should change the version number on the announcement thread, too.

Edited May 23, 2011 by UEZ

[skyteddy]

Thank to all for the new version 3.3.7.1!

I can't compile anything, because the au3check.exe crashes every time. OS: Win XP, SP3+, x64, German

>Running AU3Check (1.54.20.0)  params:-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6  from:C:\Programme\AutoIt3\Beta
!>11:26:24 AU3Check ended.rc:-1073741819