Graphical AutoIt Debugger

[ldub]

When the debugger (ver 0.27) load an .au3 file, all french accented characters are replaced by squares.

Then, if I save the file, all the accentued characters are modified !

What can you do ?

Thanks so much for this wonderful debugger.

Edited January 5, 2010 by ldub

[Stumpii]

@corgano

The debugger (like Scite via View -> Parameters) should allow you to send command line parameters to the uncompiled script. However this seems to be broken in v0.27.

@mkornuser

Yes, the feature seems to be currently broken, but it should do exactly what you think it should.

@Stumpii

Can you take a look?

You can add command line arguments to an non-compiled script. The Options>Script Settings is the right place to add the command line arguments. That feature is broken at the moment and will be fixed in the next release.

I will look at debugging a compiled script. It should be possible, but may take a little work.

[Stumpii]

Stumpii, whats the size limit of the tracked variables? I had a website source code stored in a variable, but it got cut off. This was quite some time ago, just remembered.

I don't really know. If you experience the problem again, let me know and I will look into it some more.

[Stumpii]

When the debugger (ver 0.27) load an .au3 file, all french accented characters are replaced by squares.

Then, if I save the file, all the accentued characters are modified !

What can you do ?

Thanks so much for this wonderful debugger.

Can you send me an example?

Thanks,

Stumpii

[ldub]

Here are the samples :

1 - Before.jpg = original

2 - Reading.jpg = reading under Graphical Autoit Degugger

3 - After.jpg = once saved by Graphical Autoit Degugger

I hope you'll find a solution for these accentued characters !

Thanks a lot

[jchd]

Bonsoir Paris

Under which encoding are you saving your file? Probably UTF-8 with BOM is better than other and easier to deal with. The latest beta corrected wrong file encoding. Only catch with Unicode source files:

! ***************************************************************************************************

! * Input file is UTF8 encoded with BOM, Au3Check does not support UNICODE and will be skipped. *

! ***************************************************************************************************

[ldub]

Bonsoir Paris

Under which encoding are you saving your file? Probably UTF-8 with BOM is better than other and easier to deal with. The latest beta corrected wrong file encoding. Only catch with Unicode source files:

! ***************************************************************************************************

! * Input file is UTF8 encoded with BOM, Au3Check does not support UNICODE and will be skipped. *

! ***************************************************************************************************

Hi, South of France !

I am not competent enough to solve all the problems associated with accented characters. It's pretty complicated.

I run under Windows XP SP3. All my software, including AutoIt, work perfectly. I never have problems with accented characters.

What should I do, practically, that Autoit Graphical Debugger works normally with accented characters?

[jchd]

I am not competent enough to solve all the problems associated with accented characters. It's pretty complicated.

No, not "pretty complicated": it's rather _very_ complicated!

More seriously, it's not all the big fuss you believe it to be. There are much complex solutions when working with ANSI codepages and the like than working with standard Unicode. AutoIt uses standard Unicode natively so it's an ally rather than an ennemy in this battle to have things going straight.

What should I do, practically, that Autoit Graphical Debugger works normally with accented characters?

From your point of view, I'd say you only have to use Unicode to store your source files. Switch to Unicode with Files > Encoding > UTF-8 with BOM then modify your source anyhow (add space + remove space) and CTRL-S to save it. If you have thousand of *.au3 files, then a "foreach $file in *.au3 do recode $file ..." kind of solution is better than hours of "hand job"

The counterpart is making the author of Autoit Graphical Debugger understand the benefit of handling files encoded in Unicode and treat them accordingly, if it isn't the case already.

Once all that's done, I believe the only problems will be with Console(Read|Write) which currently only handle ANSI (user codepage) out of the box. As I found this to be a Royal Pain In The Ass®™, I use a slitghtly modified version of ConsoleWrite to display Unicode correctly:

Func _ConsoleWrite($sString)
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sString, "int", -1, _
                                "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    If @error Then Return SetError(1, @error, 0)
    Local $tText = DllStructCreate("char[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sString, "int", -1, _
                            "ptr", DllStructGetPtr($tText), "int", $aResult[0], "ptr", 0, "ptr", 0)
    If @error Then Return SetError(2, @error, 0)
    ConsoleWrite(DllStructGetData($tText, 1))
EndFunc

Also, change Scite codepage to Unicode: in Global.Properties, change lines 230-231 from

#code.page=65001

code.page=0

to

code.page=65001

#code.page=0

You also may want to use a more programmer-oriented fixed-pitch font able to display a minimum subset of common glyphs. I've selected DejaVu Sans Mono (available for free, Google for it) to be very readable and "international" enough. Of course, this is only a personal choice and others may have different preferences. For consistency, I also change _ArrayDisplay to use this font since it greatly simplifies my life.

I've spent undecent time sorting out various Unicode representation issues with parts of AutoIt, SQLite and the like. I now have a reasonable understanding of the issues and the hints I give above are mostly result of bad experience with other solutions. By no mean am I an absolute reference: I can only offer some guidance in this "Babel mess" that I leant the hard way.

[ldub]

Hi, South of France,

Thank you very much for your very detailed help.

Your approach scares me a little. I'm afraid to spend much time trying to adapt Unicode and to go from the frying pan into the fire.

I am not very talented and I don't like the "Royal Pain" ...

However, I'll try to understand your solutions.

Rgds

Edited January 15, 2010 by ldub

[jchd]

Your approach scares me a little. I'm afraid to spend much time trying to adapt Unicode and to go from the frying pan into the fire.

French frieds?

It would be great if Stumpii would comment on this and confirm his code can cope with (now standard) Unicode sources in, say, UTF-8 with BOM format. That or go public saying how much work is needed and if such support can be made available soon.

I see this as a prerequisite to any serious use for those having to deal with comments or litterals outside pure 7-bit ASCII. Relying on local codepage for correct handling is highly unstable/insecure and very likely to cause weird problems someday.

[Stumpii]

French frieds?

It would be great if Stumpii would comment on this and confirm his code can cope with (now standard) Unicode sources in, say, UTF-8 with BOM format. That or go public saying how much work is needed and if such support can be made available soon.

I see this as a prerequisite to any serious use for those having to deal with comments or litterals outside pure 7-bit ASCII. Relying on local codepage for correct handling is highly unstable/insecure and very likely to cause weird problems someday.

@jchd and ldub

First, I am in no way an expert on encoding, so bear with me!

I tried the following example and could not replicate the problem. I first created the file using Notepad++ which saved the file in 'UTF8 without BOM' format. The file opened fine in the debugger and saved fine as well. Give it a try. If you can send me a text file of your problem text, I will try opening it my end and see how it is encoded. The ScintillaNet control has a codepage setting the same as SciTE (because both use Scintilla), so anything that is possible in SciTE should be possible in the debugger.

test.au3

Stumpii

[jchd]

First, I am in no way an expert on encoding, so bear with me!

But you're on your way of becoming one!

I tried the following example and could not replicate the problem. I first created the file using Notepad++ which saved the file in 'UTF8 without BOM' format. The file opened fine in the debugger and saved fine as well. Give it a try. If you can send me a text file of your problem text, I will try opening it my end and see how it is encoded. The ScintillaNet control has a codepage setting the same as SciTE (because both use Scintilla), so anything that is possible in SciTE should be possible in the debugger.

Copy - paste this into a UTF-8 + BOM encoded file.

Local $str[10] = [ _
 "Sant Julià de Lòria", _
 "Skrýchov u Opařan", _
 "Žíšov", _
 "БОЛЬШОЕ ГРИДИНО", _
 "МЫТИЩИ-ДТИ", _
 "歴史的仮名遣", _
 "変体仮名", _
 " فرنسيّ عربيّ", _
 "सभी मनुष्यों को गौरव और अधिकारों के मामले में जन्मजात स्वतन्त्रता और समानता प्राप्त है। उन्हें बुद्धि और अन्तरात्मा की देन है और परस्पर उन्हें भाईचारे के भाव से बर्ताव करना चाहिये।", _
 "เขาจะได้ไปเที่ยวเมืองลาว" _
]

Process it then save it back in a reply here. It should look the same.

The reason I do this way it that to really test some Unicode, you have to use characters outside your local codepage. So testing with àéèçÔë and the like isn't that much a good test since those chars are found in the Windows Western codepage that many of us are using.

The example above uses chars in more than one codepage. Only Unicode can process them correctly, but there isn't one Windows font (that I know of) able to display all of them at the same time. You're likely to see only some of those characters in your editor or messagebox. Only using html on a forum like this one shows everything easily.

NOTE: I grabed text from my own ZIP database and also some scripts examples from Wikipedia. I'm not responsible if anything above really is a terrible insult!

[Stumpii]

But you're on your way of becoming one!

Copy - paste this into a UTF-8 + BOM encoded file.

Local $str[10] = [ _
 "Sant Julià de Lòria", _
 "Skrýchov u Opařan", _
 "Žíšov", _
 "БОЛЬШОЕ ГРИДИНО", _
 "МЫТИЩИ-ДТИ", _
 "歴史的仮名遣", _
 "変体仮名", _
 " فرنسيّ عربيّ", _
 "सभी मनुष्यों को गौरव और अधिकारों के मामले में जन्मजात स्वतन्त्रता और समानता प्राप्त है। उन्हें बुद्धि और अन्तरात्मा की देन है और परस्पर उन्हें भाईचारे के भाव से बर्ताव करना चाहिये।", _
 "เขาจะได้ไปเที่ยวเมืองลาว" _
]

Process it then save it back in a reply here. It should look the same.

The reason I do this way it that to really test some Unicode, you have to use characters outside your local codepage. So testing with àéèçÔë and the like isn't that much a good test since those chars are found in the Windows Western codepage that many of us are using.

The example above uses chars in more than one codepage. Only Unicode can process them correctly, but there isn't one Windows font (that I know of) able to display all of them at the same time. You're likely to see only some of those characters in your editor or messagebox. Only using html on a forum like this one shows everything easily.

NOTE: I grabed text from my own ZIP database and also some scripts examples from Wikipedia. I'm not responsible if anything above really is a terrible insult!

I paste it into the debugger, save and open again and it looks fine (att). What am I doing wrong (or what is the program doing right?)!

[jchd]

I paste it into the debugger, save and open again and it looks fine (att). What am I doing wrong (or what is the program doing right?)!

That's a confirmation your debugger works flalessly with Unicode source files.

Your part is done!

Users just need to supply UTF-8 + BOM - encoded files with whatever Unicode characters they feel pretty.

Again, the only drawback if that Au3Check won't process it. But careful coders don't need it, do they?

[Stumpii]

That's a confirmation your debugger works flalessly with Unicode source files.

Your part is done!

Users just need to supply UTF-8 + BOM - encoded files with whatever Unicode characters they feel pretty.

Again, the only drawback if that Au3Check won't process it. But careful coders don't need it, do they?

Hurray!

So what does ldub need to do to get file viewing properly? Maybe start by entering the text into the debugger and saving, which will hopefully save as unicode? I mean, that was the post that started all of this.

[jchd]

So what does ldub need to do to get file viewing properly?

Probably only save it in UTF-8 + BOM encoding from within Scite. Note that once File>Encoding has been selected, it needs a (dummy) modification (add space then remove space) to cause he next save file to use he new encoding. Changing only in the menu isn't enough.

[ldub]

Probably only save it in UTF-8 + BOM encoding from within Scite. Note that once File>Encoding has been selected, it needs a (dummy) modification (add space then remove space) to cause he next save file to use he new encoding. Changing only in the menu isn't enough.

I have opened the text under SciTE and converted it to UTF 8 - BOM.

Then I successfully read it in Autoit Debugger (see screenshot).

Though, the debug file must also be converted to be readable.

Finally, it works after all these manipulations.

My text is not as perfect as the one of jchd but is sufficient for me.

I'll maybe use Autoit to automate all this!

Thanks again for your help

Edited January 17, 2010 by ldub

[jchd]

I have opened the text under SciTE and converted it to UTF 8 - BOM.

Then I successfully read it in Autoit Debugger (see screenshot).

Though, the debug file must also be converted to be readable.

Excellent news. Realize that it opens a new perspective to users of AutoIt that need to manipulate non-latin scripts (I mean 'languages') in their scripts (I mean 'programs'). I may overlook the need for it, but I feel it's important and beneficial to everyone.

[Stumpii]

I have opened the text under SciTE and converted it to UTF 8 - BOM.

Then I successfully read it in Autoit Debugger (see screenshot).

Though, the debug file must also be converted to be readable.

Finally, it works after all these manipulations.

My text is not as perfect as the one of jchd but is sufficient for me.

I'll maybe use Autoit to automate all this!

Thanks again for your help

I will look at the 'debug' file. It is possible that this file created by the debugger may not be unicode. If not, I will try and make it so.

[Stumpii]

New update of the AutoIt Debugger (0.28.0):

Fixed: Crash caused by comments at the end of continuation rungs. Thanks Datenshi.

Fixed: Command Line arguments now saved with file. Thanks wraithdu.

Changed: Auto Tidy is now performed before auto context check.

Changed: Debug file is now created with same encoding as source file. Thanks ldub.

Changed: Upgraded to VS2008. Should not affect anything.

Added: Separated ClassViewer window to three windows so Variables, Functions and Includes can be docked separately.

As usual, link is in the first post.