AutoIt parser in AutoIt

[genius257]

Hi all!

I am working on a AutoIt grammar in the Peggy syntax, and have recently started working towards being able to output the generated parser in AutoIt syntax via a Peggy plugin.

The parser/grammar is not yet 100% AutoIt syntax compatible, but I think the generated parser is now in an acceptable state.

If you have any performance optimization suggestions, let me know! 😄

When the plugin is ready for release, I will add a GitHub link to latest GitHub release generated asset, for the parser, but for now i will just attach the script here, and the grammar with modified code blocks.

The parser outputs an abstract syntax tree, representing the parsed input.

Example:

expandcollapse popup

#include <Array.au3>
#include "parser.au3"

$ast = peg_parse("2 * (3 + 4)")

If Not (@error = 0) Then
    ConsoleWrite("Error: "&@error&@CRLF)
    Exit 1
EndIf

$result = Process($ast)
_ArrayDisplay($result)

Func Process($ast)
    Switch $ast.type
        Case 'Program'
            Local $body = $ast.body
            Local $results[UBound($body)]
            For $i = 0 To UBound($body) - 1
                Local $node = $body[$i]
                $results[$i] = Process($node)
                If Not (@error = 0) Then Return SetError(@error)
            Next
            Return $results
        Case 'ExpressionStatement'
            Local $result = Process($ast.expression)
            If Not (@error = 0) Then Return SetError(@error)
            Return $result
        Case 'BinaryExpression'
            Local $left = Process($ast.left)
            If Not (@error = 0) Then Return SetError(@error)
            Local $right = Process($ast.right)
            If Not (@error = 0) Then Return SetError(@error)

            Switch $ast.operator
                Case '+'
                    Return $left + $right
                Case '-'
                    Return $left - $right
                Case '*'
                    Return $left * $right
                Case '/'
                    Return $left / $right
                Case '&'
                    Return $left & $right
                Case Else
                    ConsoleWriteError("Unknown operator: "&$ast.operator&@CRLF)
                    Return SetError(1)
            EndSwitch
        Case 'Literal'
            Return $ast.value
        Case 'ParenthesizedExpression'
            Local $result = Process($ast.expression)
            If Not (@error = 0) Then Return SetError(@error)
            Return $result
        Case Else
            ConsoleWrite("Unknown node type: "&$ast.type&@CRLF)
            Return SetError(1)
    EndSwitch
EndFunc

Link to grammar: autoit3.pegjs

Link to Peggy plugin: peggy-au3.ts

 

parser.au3 autoit3-modified.pegjs

Edited February 9 by genius257
Replaced files with fixed ones

[SOLVE-SMART]

Thanks @genius257, but I cannot run the example code above 😔 .

I created a folder, pasted your two attached files in and created a example.au3 file with the code above.
When I try to run this code (in VSCode) or try to use CTRL+F5 to call the Au3Check, I get several warnings and errors.

⚠ I also have to say: At the moment I use the VSCode extension of Loganch, yours is disabled for now. Is that a problem 🤔 ?  Are there other possible dependencies like your "vscode-autoit3-debug"?

Here my VScode OUTPUT when I try to run the example above:

Spoiler

"C:\Program Files (x86)\AutoIt3\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "c:\Store\Repositories\playground\autoit-parser-by-genius257\example.au3" [PID 31520]
+>21:29:13 Starting AutoIt3Wrapper (21.316.1639.1) from:Code.exe (1.97.0.0)  Keyboard:00000407  OS:WIN_10/2009  CPU:X64 OS:X64  Environment(Language:0407)
>Running AU3Check (3.3.16.1)  from:C:\Program Files (x86)\AutoIt3  input:c:\Store\Repositories\playground\autoit-parser-by-genius257\example.au3
"c:\Store\Repositories\playground\autoit-parser-by-genius257\parser.au3"(3879,66) : warning: $TYPES_TO_PROPERTY_NAMES: possibly used before declaration.
        ($tail[$i])[$TYPES_TO_PROPERTY_NAMES[($tail[$i])["type"]]]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"c:\Store\Repositories\playground\autoit-parser-by-genius257\parser.au3"(3879,77) : error: Statement cannot be just an expression.
        ($tail[$i])[$TYPES_TO_PROPERTY_NAMES[($tail[$i])["type"]]] = $result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"c:\Store\Repositories\playground\autoit-parser-by-genius257\parser.au3"(6109,37) : warning: $a: possibly used before declaration.
     $m["body"] = $body = Null ? $a :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"c:\Store\Repositories\playground\autoit-parser-by-genius257\parser.au3"(6109,37) : error: $a: undeclared global variable.
     $m["body"] = $body = Null ? $a :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
c:\Store\Repositories\playground\autoit-parser-by-genius257\example.au3 - 2 error(s), 2 warning(s)
!>21:29:14 AU3Check ended.rc:2
+>21:29:14 AutoIt3Wrapper Finished.
!>Exit code 2 Time: 0.947

 

I guess at least the two errors should not exist, right?

Best regards
Sven

Edited February 8 by SOLVE-SMART

[genius257]

Hi @SOLVE-SMART,

I am sorry for the problems you found, that's my bad. 😞

5 hours ago, SOLVE-SMART said:

I created a folder, pasted your two attached files in and created a example.au3 file with the code above.
When I try to run this code (in VSCode) or try to use CTRL+F5 to call the Au3Check, I get several warnings and errors.

• The warning with $TYPES_TO_PROPERTY_NAMES possibly being used before declaration, i have fixed with a #forcedef.
• The error with "Statement cannot be just an expression", will pass at runtime, but would not work as expected, so I've fixed it with a new helper method "__setKeyValue"
• The warning/error with $a being an undeclared global variable and possibly used before declaration, was an oversight by me and have been fixed.

6 hours ago, SOLVE-SMART said:

⚠ I also have to say: At the moment I use the VSCode extension of Loganch, yours is disabled for now. Is that a problem 🤔 ?  Are there other possible dependencies like your "vscode-autoit3-debug"?

No problem at all My vscode-autoit3-debug extension simply does not use the Au3Check tool, and therefore i only notice issues at runtime, currently (I should have manually used Au3Check before uploading 🤡). If i had reached the non declared variable area, the code would have crashed and I would have noticed, but had not yet, and the AutoIt runtime itself therefore does not care

 

---

 

But I've updated the uploaded files, after fixing the issues.

You only need the parser.au3 file for testing, the autoit3-modified.pegjs is just for showing the source grammar that generated the AutoIt3 code

Thanks for the heads up, and once again my apologies for my lack of quality control 😅

[SOLVE-SMART]

17 hours ago, genius257 said:

hanks for the heads up, and once again my apologies for my lack of quality control 😅

Please don't. Please do not apologize for your great work. Yes it does not work in the first place, but to me it does not matter that much. What matters, and this is the point what I appriciate the most, your answers, your code and your explanations (here or on GitHub) makes me a better programmer and lead me to learn new things.

I will test the new parser example tomorrow (or in the next days) and will let you know about possible questions etc 🤝 .

Best regards
Sven

[IronFine]

I just looked at the parser.au3 (just out of curiosity) and while scrolling, I discovered many calls of

Execute("")

Is this just due to obfuscation or does it serve another purpose?

 

[genius257]

1 hour ago, IronFine said:

I just looked at the parser.au3 (just out of curiosity) and while scrolling, I discovered many calls of

Execute("")

Is this just due to obfuscation or does it serve another purpose?

Hi @IronFine ,

It's for debugging:

I use execute for dynamic function calls. When an syntax error occurs, no error information is given. Calling Execute within an Execute execution, allows AutoIt3 to report errors again.

 

During my testing it seems to have no tangible performance impact, so I've left it in for now  

[genius257]

Just an update regarding performance. This parser is slower than i want, but from my own testing, it seems the biggest problem is how AutoIt works with Arrays and Maps.
It seems AutoIt always copies arrays and maps when re-assigning to another variable and/or returning from a function.
I have compared my parser to the one hand made by @Mat (https://github.com/MattDiesel/Ault) and my parser seems at least is 100x slower.
Due to how the expected behavior of pegjs/peggy grammars work, i cannot use the same trick of defining one big array that never copied (unless number of branches reaches the array limit, and it will then redim +100 elements) and passed via ByRef.

I have plans on making an executor, with som JIT asm creation, that MIGHT help me solve this the long way around at some point, but for now it seems the performance is stuck with the current implementation.

I still plan on continuing with my current implementation, simply to allow both JS and au3 parsing to work identically.

 

Edit:

When testing big, i parse the parser itself. On my machine it takes ~11minutes and takes up ~2.6GB of ram 😵

So just to be clear, 11minutes is not acceptable to me, but as stated above I'll stick with this approach anyway.

Edited February 10 by genius257

[jaberwacky]

Not having delved too deeply into your code, I was wondering if ByRef could be your solution? Pass an array to a function and process it without having to return the modified array?

[genius257]

7 minutes ago, jaberwacky said:

Not having delved too deeply into your code, I was wondering if ByRef could be your solution? Pass an array to a function and process it without having to return the modified array?

ByRef can help, but not with my current implementation. Trouble is this code is a port of the peggy JS generated output. Passing in an array without returning it, while still keeping the same expected behavior is tricky. I am not sure i can, honestly.