A cross-platform implementation of the AutoIt language

[TheDcoder]

I am back with another bi-weekly (almost ) update, as usual I have been busy with life so progress is slow, but I have jumped another small yet-somewhat-significant hurdle, I have mostly figured out the basic skeleton of the syntax, to achieve this I had to spend 100% of my brain power for a couple of minutes almost every day for a two weeks

Here is a sneak peak of the code:

enum Operation {
	/* Addition, Substraction, Multiplcation, Division, Exponentiation */
	OP_ADD, OP_SUB, OP_MUL, OP_DIV, OP_EXP,
	/* Concatenation */
	OP_CAT,
	/* Equal to, Strictly equal to, Less than, Less than or equal to, Greater than, Greater than or equal to */
	OP_EQU, OP_SEQU, OP_LT, OP_LTE, OP_GT, OP_GTE,
	/* Conditional */
	OP_CON,
};

struct Token {
	enum TokenType type;
	char *data;
	size_t data_len;
	void *info;
};

struct TokenList {
	size_t length;
	struct TokenListNode *head;
	struct TokenListNode *tail;
	bool dirty;
};

struct TokenListNode {
	struct Token *token;
	struct TokenListNode *prev;
	struct TokenListNode *next;
};

struct Primitive {
	enum {
		PRI_NUMBER,
		PRI_STRING,
		PRI_BOOLEAN,
		// ...
	} type;
	union {
		//int number;
		//char *string;
		//bool boolean;
	};
};

struct Operand {
	enum {
		OPR_PRIMITIVE,
		//OPR_VARIABLE,
		//OPR_MACRO,
	} type;
	union {
		struct Primitive *value;
		//struct Variable *variable;
		//struct Macro *macro;
	};
};

struct Expression {
	enum Operation op;
	struct Operand operands[];
};

struct Declaration {
	enum {SCO_LOCAL, SCO_GLOBAL} scope;
	// ...Const, Static
	char *name;
	bool is_function;
	union {
		// Variable or constant
		struct Expression *initializer;
		// Function
		struct {
			struct Statement *block;
			size_t size;
		} code;
	};
};

struct Statement {
	enum StatementType {
		SMT_DECLARATION,
		SMT_EXPRESSION,
	} type;
	union {
		struct Declaration *declaration;
		struct Expression *expression;
	};
};

struct Unit {
	enum UnitType {
		UNT_COMMENT,
		UNT_DIRECTIVE,
		UNT_STATEMENT,
	} type;
	union {
		struct Token *token;
		struct Statement *statement;
	};
};

The above code is basically the struct declarations that I extracted from my WIP code, these structures reference each other to form a single "code unit" which roughly corresponds to a single line of code in the AutoIt syntax. I have tried to arrange the declarations in the ascending order, where the top-most structure roughly represents the most basic element of the syntax (the token in this case) which is contained in a more informative and complex structure, all the way to the bottom where the "Unit" structure has a fully formed meaning.

 

This should be enough for me to start writing code to construct the syntax tree, but there are some other things that I would like to work on first. Right now I am looking into incorporating debug data into the final binary, so that we can have a nice debugger to debug our scripts .

It is not necessary that I look into this right now, but it will come in handy if I study and understand the basic concepts of how the debugger maps the final instructions to strings in the source code, so that I can make modifications to the syntax tree's design right now, as opposed to in the future to prevent inconvenience of re-writing of related code. Basically leaving gaps in the tree which I can fill later when actually implementing the debugging functionality.

A good example to study in my opinion is the format used by C debuggers (gdb and co.), and I found out that the format is called DWARF (Name FAQ), I tried to read their technical specification but it is too thick for me right now... but luckily I found an article called "Introduction to the DWARF debugging format" written by the Chairman of the standardization committee, lucky me .

So I am reading that right now. Hopefully I will have another update for you in 2 weeks, see you until then!

[jpm]

It will good to use VS code as the debuger as the extension done for AutoIt itself.

I hope that make sense with your implementation.

[TheDcoder]

Hi @jpm,

Thanks for the suggestion, I assume you are suggesting I use the same technique for the debugger that is used in the AutoIt extension for VS Code? I had a look at it and the feature list doesn't mention debugging functionality, and I did a basic search of the code to see if it has support but I couldn't find anything.

Are you sure that the extension supports interactive debugging of AutoIt code?

[jpm]

not sure

[TheDcoder]

@jpm I see. Thanks for your suggestion regardless. I am pretty positive that the extension does not support interactive debugging, but I will try out VS Code myself just to be 100% sure

[TheDcoder]

BONUS UPDATE! A minor bonus update for you guys, I have implemented parsing of keywords while working on the statement parser:

Spoiler

commit bec465aba26af50297cf0b6c828ff8c89d146db8
Author: Damon Harris <TheDcoder@protonmail.com>
Date:   Thu Oct 15 07:33:13 2020 +0530

    Implement identification of keywords

diff --git a/parse.c b/parse.c
index 5db1d62..fcc6025 100644
--- a/parse.c
+++ b/parse.c
@@ -48,6 +48,48 @@ char STRING_CE[] = "ce";
 char STRING_COMMENT_START[] = "comments-start";
 char STRING_COMMENT_END[] = "comments-end";
 
+struct KeywordMap {
+       char *string;
+       enum Keyword symbol;
+};
+
+struct KeywordMap KEYWORD_MAP[] = {
+       {"Dim", KWD_DIM},
+       {"Local", KWD_LOCAL},
+       {"Global", KWD_GLOBAL},
+       {"Enum", KWD_ENUM},
+       {"Const", KWD_CONST},
+       {"Static", KWD_STATIC},
+       {"ContinueCase", KWD_CONT_CASE},
+       {"ContinueLoop", KWD_CONT_LOOP},
+       {"Default", KWD_DEFAULT},
+       {"Null", KWD_NULL},
+       {"Do", KWD_DO},
+       {"Until", KWD_UNTIL},
+       {"While", KWD_WHILE},
+       {"WEnd", KWD_END_WHILE},
+       {"For", KWD_FOR},
+       {"In", KWD_IN},
+       {"To", KWD_TO},
+       {"Step", KWD_STEP},
+       {"Next", KWD_NEXT},
+       {"Exit", KWD_EXIT},
+       {"ExitLoop", KWD_EXITLOOP},
+       {"Func", KWD_FUNC},
+       {"Return", KWD_RETURN},
+       {"EndFunc", KWD_END_FUNC},
+       {"If", KWD_IF},
+       {"Else", KWD_ELSE},
+       {"ElseIf", KWD_ELSE_IF},
+       {"EndIf", KWD_END_IF},
+       {"ReDim", KWD_REDIM},
+       {"Select", KWD_SELECT},
+       {"Switch", KWD_SWITCH},
+       {"Case", KWD_CASE},
+       {"EndSelect", KWD_END_SELECT},
+       {"EndSwitch", KWD_END_SWITCH},
+};
+
 static void print_token(struct Token *token) {
        puts("---### TOKEN ###---");
        char *token_type;
@@ -119,6 +161,7 @@ struct Token token_get(char *code, char **next) {
                .type = TOK_UNKNOWN,
                .data = NULL,
                .data_len = 0,
+               .info = NULL,
        };
        size_t length;
        char *next_code = NULL;
@@ -193,6 +236,12 @@ struct Token token_get(char *code, char **next) {
                token.type = TOK_WORD;
                token.data = code;
                token.data_len = length;
+               
+               // Identify keywords
+               for (size_t i = 0; i < sizeof KEYWORD_MAP / sizeof(struct KeywordMap); ++i) if (strncmp(KEYWORD_MAP[i].string, code, length) == 0) {
+                       token.info = &(KEYWORD_MAP[i].symbol);
+                       break;
+               }
        } else if (*code == CHR_MACRO || *code == CHR_VARIABLE){
                // Macro or Variable
                token.type = *code == CHR_MACRO ? TOK_MACRO : TOK_VARIABLE;
diff --git a/parse.h b/parse.h
index 64dbf60..5704ad1 100644
--- a/parse.h
+++ b/parse.h
@@ -39,10 +39,48 @@ enum TokenType {
        TOK_COMMA,
 };
 
+enum Keyword {
+       KWD_DIM,
+       KWD_LOCAL,
+       KWD_GLOBAL,
+       KWD_ENUM,
+       KWD_CONST,
+       KWD_STATIC,
+       KWD_CONT_CASE,
+       KWD_CONT_LOOP,
+       KWD_DEFAULT,
+       KWD_NULL,
+       KWD_DO,
+       KWD_UNTIL,
+       KWD_WHILE,
+       KWD_END_WHILE,
+       KWD_FOR,
+       KWD_IN,
+       KWD_TO,
+       KWD_STEP,
+       KWD_NEXT,
+       KWD_EXIT,
+       KWD_EXITLOOP,
+       KWD_FUNC,
+       KWD_RETURN,
+       KWD_END_FUNC,
+       KWD_IF,
+       KWD_ELSE,
+       KWD_ELSE_IF,
+       KWD_END_IF,
+       KWD_REDIM,
+       KWD_SELECT,
+       KWD_SWITCH,
+       KWD_CASE,
+       KWD_END_SELECT,
+       KWD_END_SWITCH,
+};
+
 struct Token {
        enum TokenType type;
        char *data;
        size_t data_len;
+       void *info;
 };
 
 struct TokenList {

--

I have also created a chatroom on [matrix], so if anyone is interested they can easily join using the web client: https://app.element.io/#/room/#EasyCodeIt:matrix.org

The chatroom is fully public and all chat is visible to everyone (including unregistered guests!), so you can have a peek 👀 before joining. I am the only participant at the moment but I will respond to everyone who wants to chat... because I love to talk, perhaps too much

[TheSaint]

6 hours ago, TheDcoder said:

because I love to talk, perhaps too much

I can attest to that. 

[seadoggie01]

Ping! Just in case you forgot you bi-weekly update

[TheDcoder]

@seadoggie01 I missed my bi-weekly updated... isn't that surprising?

Seriously though, I did remember it but I didn't have any working or complete code, so I couldn't share it. In other words, it is still a work in progress. There has been progress since the last post:

I have started work on the statement parser, I am currently working on parsing declarations, as well as making other required tweaks in the tokenizer. Oh, and I also implemented a rudimentary error handling method using longjmp, because it gets complex in C.

There is a lot of new code that I wish I could share, but it is all a mess so I can't do it in a meaningful way.

--

And I am doing all that while dealing with some issues on my professional end, no doubt many would be in the same situation thanks to the deadly cough going around.

I am resuming work on ECI now, after taking a break for 2 day, so hopefully I will have a more proper update out soon next week

[TheDcoder]

Sorry about the lack of updates, I have been busy with life (as usual) and wasn't able to properly formulate a plan on how to proceed forward with parsing statements, I decided that it would be better to tackle expression parsing now, but that threw me into a whole new tangent about various different kinds of techniques for parsing expressions... luckily I have made up my mind with what approach I will use in the initial implementation, I am not 100% sure it will work, but it should.

In the mean-while I created an "official" thread for the project:

This thread was never really meant to be used for actual technical discussion of the implementation, and it has gotten too big and ugly anyway, it is impossible for new users to get a brief glance about the whole thing without reading through all the pages. So everyone please follow the new thread for updates! I should have an update out soon if all goes according to plan