(RESOLVED) How to find the error by the line number returned from the compiled EXE?

[dv8]

Hi guys,

please bare with me for a moment here...

My goal: To be able to take the error line number returned by the EXE file and locate the actual line in the source.

I believe most of the AutoIt users are coming to this issue at some point.

I realize the final source, before the compilation is done, is the combined source of all the include files and the main script etc. So I figured if I have a way to get this combined source, that would give me a solution to this problem. And I spent the last couple of days (more than 10 hours a day) searching in the forum and asking some questions in different threads. And with the help of few people, I found a way (using the Obfuscator) to get this source and the line numbers were actually MATCHING!

However this solution introduced a WHOLE LOT of other problems for me and broke some of the functionality in my script since it has a lot of Call() and ObjEvent() functions. My code turned out to be un-obfuscatable. And for the record, I do not need any obfuscation done at all, nor I need the unused included functions stripped down (even though it made my EXE a 30% smaller). I needed the Obfuscator just to get this final piece of code to be able to achieve My goal.

So all this got me thinking... Please correct me if I'm wrong, because I don not have any knowledge of the internals of a compiler.

Since Aut2exe is the one that combines all the code together and then working it's magic to create the EXE, there should be a pint in this process where that code is available in its final form. Then why isn't there a command line option to dump the final combined source in a file just before it is compiled?

Once again, I do not know the internals of the compiler but my logic dictates that this would be the most reasonable and straightforward solution.

Please share your thoughts and also, if any of you knows a way to achieve my goal without using the Obfuscator, I would LOVE to here it!

EDIT: And here is the solution: ( I've tried to explain it as simply as I can, so even the beginners can understand it.)

With the huge help from Jos, the goal in this post is now achieved, by using the (version 1.0.29.8 and above) and the /MergeOnly (or /MO) option.

So, if you want to be able to find the errors reported by your compiled (EXE) scripts easily, you will need to add this line:

#Obfuscator_Parameters=/mergeonly

at the beginning of your script and run it through the Obfuscator (version 1.0.29.8 and above).

The option in the line above will not do any obfuscation to your code. It will only merge all the included files into just one, called YourScriptInitialName_Obfuscated.au3.

Now if you compile the obfuscated file, the error line numbers reported by your EXE file should match the lines in this obfuscated file. And once you find the right line of code, it should be a piece of cake to locate this line in your original source code. So make sure you keep this file somewhere for future error reference.

IMPORTANT NOTE: You should pay attention to the messages that Obfuscator gives, or check the Obfuscator.log file after the process is completed for any messages like this one:

-Line: 2. Encountered a continuationline which would be longer than 4095 when merge into 1 line, so spliting this continuation line into multiple lines..

If there is a similar line displayed, the line numbers from the error report and the obfuscated source will NOT match. In order to match them, you will need to merge any lines broken into smaller lines with the continuation character ("_") in the obfuscated source. But you will have to do that AFTER you have compiled your EXE file from that obfuscated script.

Here is how the process works:

1. Add #Obfuscator_Parameters=/mergeonly at the beginning of your AU3 script

2. Obfuscate your AU3 script using the Obfuscator.exe (version 1.0.29.8 and above)

3. Compile the obfuscated script.

4. Check the Obfuscator.log file (it is in the same folder where the Obfuscator.exe is installed) for a line similar to the bulded one in the above text.

5. If no similar lines are found - you are good to go and the line numbers should match.

6. If there are similar lines, you will need to merge the lines splitted with the continuation ("_") character in order for your line numbers to match the number in the error report.

If you come to the 6th point in the above list, you can either search for these lines manually, or you can use the following script to do the changes for you:

If $CmdLine[0]=1 AND FileExists($CmdLine[1]) Then
   $SourceFileName=$CmdLine[1]
Else
    $SourceFileName = FileOpenDialog("Specify the file to be processed", @ScriptDir, "AutoIt3 Script (*.au3)", 3)
   If @error Then Exit
EndIf

$ScriptData = Fileread($SourceFileName)
;Merge Continuation line into one line
$ScriptData = StringRegExpReplace($ScriptData, "( _rn)", "")
$TargetFileName = StringReplace($SourceFileName,".","_N.",-1)
FileDelete($TargetFileName)
FileWrite($TargetFileName, $ScriptData)

Edited February 25, 2012 by dv8

[jvanegmond]

All it does is read out any include files and simply copy-paste them in your script. If you do this manually, it will take you 5 minutes. If you write a program to do this, it will take you 10 minutes.

Good luck.

[dv8]

Thank you. Is this the way you do it? If it is, why don't you share your program?

I will sure try to do that and I may solve my problem this way. However this will not do much good to the next user who bumps into this problem.

And may I ask, what good is that line number for, if this line number does not exists anywhere in the scripts for reference?

The fact that this number is there suggests that there should be a way to get to it, right?

I'll do some programming after my lunch and post the solution here if I manage to get satisfactory results.

[jvanegmond]

This is a trivial problem. Anyone can solve it in 10 minutes. I don't have the code to do it or I would have shared it with you. Who do you think I am? Some kind of person who doesn't share his code... No. There is probably a configuration for obfuscator to do this as well. Fortunately for you, obfuscator is well documented.

I say again: Good luck.

This is such a trivial thing to do that it's not worth annoying me or anyone about. Starting thinking before doing and you'll do just fine.

[dv8]

As I mentioned above I spent the A LOT of time reading through the Obfuscator documentation and testing different configuration options. And again as I stated above it only brought more problems on the table instead of solving this "trivial" one for me.

And maybe because it is so "trivial", there is no solution to it offered by anyone yet.

I say again: Thank you!

Believe me, I do A LOT of thinking and reading, then testing, before doing and I do just fine.

Please do not annoy yourself any more with this issue. I would really like to see what other people think about this. Thank you.

[dv8]

All it does is read out any include files and simply copy-paste them in your script. ...

OK,

here is what I did, following Manadar's suggestion:

expandcollapse popup

;Au3CodeAppender.au3
;Written by dv8 on Feb. 24 2012
;Any suggestions and improvements are welcome!

#Include <string.au3>

If $CmdLine[0]=1 AND FileExists($CmdLine[1]) Then
   $SourceFile=$CmdLine[1]
Else
    ConsoleWrite("ERROR: The input file does not exist!"&@CRLF)
   Exit
EndIf

ConsoleWrite(StringTrimRight(@ScriptFullPath,4)&".ini"&@CRLF)
GLOBAL $IncludeLibrariesPath=INIRead(StringTrimRight(@ScriptFullPath,4)&".ini","General","IncludeLibrariesPath","")
If $IncludeLibrariesPath<>"" AND StringRight($IncludeLibrariesPath,1)<>'\' Then $IncludeLibrariesPath&='\'
If Not FileExists($IncludeLibrariesPath) Then
   ConsoleWrite("ERROR: The IncludeLibrariesPath ("&$IncludeLibrariesPath&") does not exist. Please set it in the INI file."&@CRLF)
    Exit
EndIf
GLOBAL $IncludedFiles=""                                        ;This is to keep track of the included files so they do not get included twice.
GLOBAL $NewContent=""

RecursiveAddFile($SourceFile)

$DestFile=FileOpen(StringTrimRight($SourceFile,4)&"_Merged.au3",2)
FileWrite($DestFile,$NewContent)
FileClose($DestFile)

Func RecursiveAddFile($File)
ConsoleWrite("Currently adding: "&$File&@CRLF)
   $tmp=StringSplit($File,"\")
    $SourceFileName=$tmp[$tmp[0]]
    $SourceFilePath=$tmp[1]&'\'
    For $i=2 To $tmp[0]-1
       $SourceFilePath&=$tmp[$i]&'\'
    Next
    $F=FileOpen($File,0)
    While 1
        $Line=FileReadLine($F)
        If @error<>0 Then ExitLoop
        If StringLower(StringLeft(StringStripWS($Line,1),13))="#include-once" Then
            ContinueLoop
        ElseIf StringLower(StringLeft(StringStripWS($Line,1),8))="#include" Then
            $TrimmedLine=StringStripWS(StringTrimLeft($Line,8),3)
ConsoleWrite("Found #include "&$TrimmedLine&@CRLF)
            If StringInStr($TrimmedLine,'<') AND StringInStr($TrimmedLine,'>') Then
            $aName=_StringBetween($TrimmedLine,"<",">")
                $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$IncludeLibrariesPath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            ElseIf StringInStr($TrimmedLine,'"') AND StringInStr($TrimmedLine,'"') Then
                $aName=_StringBetween($TrimmedLine,'"','"')
            $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$SourceFilePath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            ElseIf StringInStr($TrimmedLine,"'") AND StringInStr($TrimmedLine,"'") Then
                $aName=_StringBetween($TrimmedLine,"'","'")
            $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$SourceFilePath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            Else
                ConsoleWrite("ERROR: Unrecognized #Include line ("&$Line&")"&@CRLF)
            FileClose($F)
                Exit
            EndIf
        Else
         $NewContent&=$Line&@CRLF
        EndIf
    Wend
    FileClose($F)
EndFunc

It also needs an INI file with the same name to read the path to the AutoIt "Include" library:

;Au3CodeAppender.ini
[General]
IncludeLibrariesPath=D:\Tools\AutoIt\Include\

This program generates a new YourScriptName_Merged.au3 script with no #include directives, wich contains all the include files that were reffered in the main script.

However, when I tested it with an intentionally added error line in my script, the reported line was 35427 and the line in the source is actually 75549.

So apparently this approach is not working. So any suggestions, anyone?

[guinness]

Why not use the parameters of Obfuscator e.g. /StripOnly?

[dv8]

Please read my first post carefully! I already stated there that the Obfuscator "...broke some of the functionality in my script since it has a lot of Call() and ObjEvent() functions."

[BrewManNH]

Did you remove the #include lines from the modified script? Sounds like it might be adding them twice.

[guinness]

I did but saying you used Obfuscator doesn't really tell much. Oh well.

[dv8]

Guys, do you read the posts at all? This is frustrating.

I said: "This program generates a new YourScriptName_Merged.au3 script with no #include directives, which contains all the include files that were referred in the main script."

You are also welcome to try it and give any suggestions...

[jvanegmond]

Now strip empty lines and comments. Then you're done. I don't think compiler directives are removed. Maybe #cs is removed, but I'm pretty sure that #include-once isn't.

I just stripped string.au3 by hand to remove all comments and empty lines and it had 216 lines. My script: #include<string.au3> Kek() fails on line 217.

Edited February 24, 2012 by Manadar

[BrewManNH]

I tried your script just now, it doesn't have any way of including any UDFs that aren't in the include folder that I can see. Not to mention, it includes the entire file that you're including even functions you're not using.

[jvanegmond]

Not to mention, it includes the entire file that you're including even functions you're not using.

That's what AutoIt does as well. Edited February 24, 2012 by Manadar

[dv8]

BrewManNH - It does include the #Include "filename.au3" files. Check again...

Manadar - we are getting closer. I stripped the empty lines and left the #include-once directives. The reported line number was 35427 and the actual line is now 39207. Only about 3800 lines more to get rid of.

Here is the new version.

expandcollapse popup

;Au3CodeAppender.au3
;Written by dv8 on Feb. 24 2012
;Any suggestions and improvements are welcome!

#Include <string.au3>

If $CmdLine[0]=1 AND FileExists($CmdLine[1]) Then
   $SourceFile=$CmdLine[1]
Else
    ConsoleWrite("ERROR: The input file does not exist!"&@CRLF)
   Exit
EndIf

ConsoleWrite(StringTrimRight(@ScriptFullPath,4)&".ini"&@CRLF)
GLOBAL $IncludeLibrariesPath=INIRead(StringTrimRight(@ScriptFullPath,4)&".ini","General","IncludeLibrariesPath","")
If $IncludeLibrariesPath<>"" AND StringRight($IncludeLibrariesPath,1)<>'' Then $IncludeLibrariesPath&=''
If Not FileExists($IncludeLibrariesPath) Then
   ConsoleWrite("ERROR: The IncludeLibrariesPath ("&$IncludeLibrariesPath&") does not exist. Please set it in the INI file."&@CRLF)
    Exit
EndIf
GLOBAL $IncludedFiles=""                                        ;This is to keep track of the included files so they do not get included twice.
GLOBAL $NewContent=""

RecursiveAddFile($SourceFile)

$DestFile=FileOpen(StringTrimRight($SourceFile,4)&"_Merged.au3",2)
FileWrite($DestFile,$NewContent)
FileClose($DestFile)

Func RecursiveAddFile($File)
ConsoleWrite("Currently adding: "&$File&@CRLF)
   $tmp=StringSplit($File,"")
    $SourceFileName=$tmp[$tmp[0]]
    $SourceFilePath=$tmp[1]&''
    For $i=2 To $tmp[0]-1
       $SourceFilePath&=$tmp[$i]&''
    Next
    $F=FileOpen($File,0)
    While 1
        $Line=FileReadLine($F)
        If @error<>0 Then ExitLoop
        If StringStripWS($Line,8)="" Then
            ContinueLoop
        ElseIf StringLeft(StringStripWS($Line,1),1)=';' Then
         ContinueLoop
        ElseIf StringLower(StringLeft(StringStripWS($Line,1),8))="#include" AND StringLower(StringLeft(StringStripWS($Line,1),13))<>"#include-once"Then
            $TrimmedLine=StringStripWS(StringTrimLeft($Line,8),3)
ConsoleWrite("Found #include "&$TrimmedLine&@CRLF)
            If StringInStr($TrimmedLine,'<') AND StringInStr($TrimmedLine,'>') Then
            $aName=_StringBetween($TrimmedLine,"<",">")
                $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$IncludeLibrariesPath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            ElseIf StringInStr($TrimmedLine,'"') AND StringInStr($TrimmedLine,'"') Then
                $aName=_StringBetween($TrimmedLine,'"','"')
            $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$SourceFilePath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            ElseIf StringInStr($TrimmedLine,"'") AND StringInStr($TrimmedLine,"'") Then
                $aName=_StringBetween($TrimmedLine,"'","'")
            $FileN=StringStripWS($aName[0],3)
                $FileToAdd=$SourceFilePath&$FileN
                If Not StringInStr($IncludedFiles,$FileToAdd) Then
             $IncludedFiles&=$FileToAdd&"|"
             RecursiveAddFile($FileToAdd)
                Else
ConsoleWrite("Skipping: "&$FileToAdd&" (Already added)"&@CRLF)
                EndIf
            Else
                ConsoleWrite("ERROR: Unrecognized #Include line ("&$Line&")"&@CRLF)
            FileClose($F)
                Exit
            EndIf
        Else
         $NewContent&=$Line&@CRLF
        EndIf
    Wend
    FileClose($F)
EndFunc

[BrewManNH]

That's what AutoIt does as well.

But with Obfuscator you can strip them out.

[BrewManNH]

BrewManNH - It does include the #Include "filename.au3" files. Check again...

No actually it doesn't. I used it on a small script that has #include lines in that point to files that are in the same folder as the script, this script doesn't include them when it's run on my script. I used the include directive #include <myincludefile.au3> which points to a file in the script folder, even though I'm not using the quotes to tell the script that it's in the same folder, AutoIt will find and include them, you script ignores them and doesn't tell me it can't find them.

[dv8]

Manadar,

I hope you see now that this is not exactly a 10 minutes process and it involves A LOT of guessing what exactly Aut2Exe does. And imagine that we manage to guess it right and make this program match the line numbers, then the next version of Aut2Exe is changed due to a bug fix, then what? Start the guessing game again?

And since Aut2Exe already does what we are trying to do here, I don't see why we can't use this in our advantage. I believe it won't be too much of a trouble for the developer to add some kind of command line switch (/DumpSource for example) and once the Aut2Exe gets to the point where the whole source is prepared for compilation, it checks for that switch and dumps the source into InputFilename_Dump.au3 for example.

So I would like to put this a feature request. I believe it is reasonable and will make finding bugs in the compiled scripts a lot easier.

[dv8]

I was not aware that you can use the #include <myincludefile.au3> directive to include files from the script folder. I thought only #include "myincludefile.au3" is used for local includes.

OK, I'll account for this in the next modification... Thanks for letting me know.

[jvanegmond]

Compile the file that you create from your script. The line numbers almost match. (Try to get rid of all the directives.)

It's actually not such a big deal to do this, considering the alternatives. My initial 10 minute script observation was based on the old compiler. It has obviously changed. But it's still not a big deal at all. (Again: considering the alternative).