EasyCodeIt - cross-platform AutoIt implementation

[E1M1]

Would be cool to have examples directory in your git where with every new feature you also post example code. Like if you add switch statement then also add switch.au3 to your examples that does simple printing or something. you could also just add it in form of md file so that code is in code tags and output is in some preformat tags.

[TheDcoder]

@E1M1 That does sound cool, but I am a bit hesitant to do something like that for several reasons. I mostly want to keep the tree clean for now and since the code is at the rapid evolution stage, starting to write documentation can become a tedious task. I am having enough trouble as-is

And since ECI is mostly the same syntax as AutoIt, there wouldn't be much trouble looking up AutoIt documentation in the interim.

[E1M1]

That would allow others to see how far have you get with this project. I mean not eveyone who is interested in project is interested enough to try it out for themselves. It's kinda like with YouTube - It allows me to quickly check out new interesting and fun Doom mods without having to install and play these myself. And so is with examples - it helps you to get more "views". Which means better chance that someone would actually try to do something with your language and upload their script to their github which in return ends up in more click on your project.
I understand how busy you are - I would probably also feel that way. But you also dont want to end up having 100% implementation of Autoit and only then star to figure out how to get user base.
And These example scripts could also serve as some sort of unit tests - you refactor something and then see which example scripts start to print something else than they used to.

Edited February 20, 2021 by E1M1

[JockoDundee]

Fledgling Startup EasyCodeIt Faces Division Over Direction  (Feb 20, 2021 - Wire Services)

Citing concerns about a perceived dearth of example code, a small but vocal faction of the burgeoning ECI community shared their belief that greater resources invested towards example scripts would, in the end lead to greater “views”, and ultimately “more click on [the] project”.  With a shout-out to the iconic, if aging Id title, poster E1M1, speaking for the faction argued:

Quote

It's kinda like with YouTube - It allows me to quickly check out new interesting and fun Doom mods without having to install and play these myself.

This latest provocation threatens to widen the bifurcation started previously when E1M1 noted simply that such a directory might be “cool”, and was decidedly vague about implementation details. The response from the reclusive and enigmatic “TheDcoder” was swift and unambiguous, countering “cool” with “tedious” and ending with a classic punctuating emoji.

What TheDcoder’s response to this latest more aggressive salvo is anyone’s guess, known for being wildly unpredictable, there’s just no telling what his next move may be as the fate of ECI hangs in the balance 

Stay tuned - this is breaking news...

[TheDcoder]

@E1M1 I definitely understand what you are saying. I will most likely consider an "out of tree" examples section in the future, maybe it can even be maintained by the community if it gets enough momentum. Anyway, thanks for your input

@JockoDundee Whoa, that's some A-level reporting, were you perhaps a journalist at any point in your life?

[TheDcoder]

I have posted a progress update about the current state of things in code at the new forum: https://forum.dtw.tools/d/5-easycodeit-progress-parsing-statements

For those who missed the last interim update (not surprising for those who did, because the post was unfortunately the last post before the page break), I basically made a new forum where I am planning to post more shorter and frequent updates:

On 2/18/2021 at 6:31 AM, TheDcoder said:

There is another major-ish annoucement, I have created a new forum at my website (forum.dtw.tools) and I intend to post more frequent shorter updates there as I plan, investigate and code. I don't want to spam the AutoIt forum with that stuff so a dedicated forum should be a good place!

The forum is effectively in alpha stage at this moment so there isn't a particular theme to it, so you can post questions, chat discussions, memes and pretty much anything else. I'll be waiting for you there

Here is the same progress update for archival purposes:

Spoiler

Hello everyone,

I am writing the current state of progress and what I am doing in these blog-like posts, hopefully these will receive regular updates as I make progress. Feel free to post in this thread to offer comments, suggestions, insights, questions etc.

Currently I am working on parsing statements as I finished working on parsing expressions.

The issue I am tackling with at the moment is dealing whitespace between tokens and safely traversing through the token array.

As ECI is written in C, there are no high-level programming features like objects which can hold a dynamic state or safeguards to prevent the program from crashing if reading beyond the array. So I need to come up with some method to safely and conveniently access tokens and skip whitespace when needed.

I have thought about writing a peek function which will automatically raise an error, thanks to the somewhat smart error system in the parser which is implemented via setjmp (basically a beefed up version of "GoTo" which can jump across functions).

Here is the current code I have for parsing statements:

struct Statement statement_get(struct Token *token, struct Token **next) {
	struct Statement statement;
	struct Token *next_token = NULL;
	
	bool function, declaration = false;
	if (token->type == TOK_WORD && kwd_is_declarator(token->keyword)) {
		function = token->keyword == KWD_FUNC;
		declaration = true;
	}
	
	if (declaration) {
		statement.type = SMT_DECLARATION;
		statement.declaration = malloc(sizeof *statement.declaration);
		if (statement.declaration == NULL) raise_mem("parsing declaration statement");
		
		statement.declaration->is_function = function;
		if (function) {
			// ...
		} else {
			// Variable Declaration
			statement.declaration->scope = SCO_AUTO;
			statement.declaration->is_static = false;
			statement.declaration->is_constant = false;
			statement.declaration->name = NULL;
			statement.declaration->initializer = NULL;
			
			// Metadata
			do {
				if (!token->info) /* Not a keyword*/ break;
				enum Keyword kwd = *(enum Keyword *)(token->info);
				if (!kwd_is_declarator(kwd)) break;
				switch (kwd) {
					case KWD_GLOBAL:
						statement.declaration->scope = SCO_GLOBAL;
						break;
					case KWD_LOCAL:
						statement.declaration->scope = SCO_LOCAL;
						break;
					case KWD_STATIC:
						statement.declaration->is_static = true;
						break;
					case KWD_CONST:
						statement.declaration->is_constant = true;
						break;
				}
			} while (TOK_WORD == (++token)->type);
			
			// Name
			if (token->type != TOK_VARIABLE) raise_unexpected_token("a variable", token);
			
			statement.declaration->name = malloc(token->data_len + 1);
			if (!statement.declaration->name) raise_mem("storing variable name");
			strncpy(statement.declaration->name, token->data, token->data_len);
			
			// Initializer
			if (token[1].type != TOK_OPERATOR) goto next;
			if (token[1].op_info.sym != OPR_EQU) /* ... */;
			
			// ... parse expression and store it as initializer
			
		}
	} else {
		statement.type = SMT_EXPRESSION;
		statement.expression = malloc(sizeof *statement.expression);
		if (!statement.expression) raise_mem("parsing expression statement");
		size_t token_count = 0;
		while (true) {
			if (token[token_count].type == TOK_WHITESPACE && token[token_count].newline || token[token_count].type == TOK_EOF) break;
			++token_count;
		}
		*statement.expression = expression_get(token, token_count);
		next_token = token + token_count + 1;
	}
	
	// Set the next token
	next: *next = next_token ? next_token : token + 1;
	return statement;
}

As shown in the code, the tokens array is being accessed directly and that is a safety hazard, I have implemented a basic safeguard in the form of dummy padding tokens at both the start and end of the array, but it only protects accessing 1 step beyond the valid range, so another solution is needed.

I'll post updates here on what strategy I am going to use. Thanks for reading my technical rambling

As I have mentioned in my post, please feel free to post any queries and suggestions in the forum. I plan to use the forum to share an increasing number of posts about many things, having a dedicated forum has it's advantages.

For starters, I don't have to spam the AutoIt forum with all of those things and I can keep this thread short for more important updates. Not to mention that all of the discussion would be more organized with multiple threads.

I'll be looking forward to see you there

Edited February 22, 2021 by TheDcoder

[TheDcoder]

This is an open invitation to anyone who is interested in getting involved with development. I think it is a good time to start seeking help from other experienced programmers to speed up development on EasyCodeIt.

Tagging all users who voted "I am willing to work on the code" on the original thread's poll: @genius257 @StandardUser @KhalidAnsari @seadoggie01 @Surya @MattHiggs

Knowledge of C is not required, sharing ideas and suggestions also counts

I request those who are interested to make an account in the dedicated forum (link in my signature, see last post for details) and post in the latest progress thread

You can also join the chatroom in Matrix, see the first post for details.

[TheDcoder]

Hi everyone,

I have been busy with taxes so I have not done much coding work recently, it is currently the last month of the financial year in India (and in many countries around the world), so I need to go through accounting and do my taxes, fun stuff... NOT

In the meanwhile, I thought I could do some research and interact with you guys on the new forum, so I created a topic about "Non-Windows Experiences" where you guys can share your experience with Non-Windows systems, this project is about getting freedom from Windows after all!

Do check it out and reply if you would like to participate in the discussion. Oh, and you can also use Discord, Google, GitHub etc. to login into the forum without having to manually create an account!

In any case, for those who are not aware, I am planning to post short and frequent updates on the dedicated forum, see my previous posts for more info. I'll keep you guys updated, hopefully we will have a major update soon once I do my taxes.

Edited March 4, 2021 by TheDcoder

[TheDcoder]

Quote

Hey everyone, apologies for not giving any updates for the entire month of march, I was busy with a work project and I also had to deal with taxes.

I recently started working on ECI again and I am currently working on the statement parser, I found that I was frequently in need of "dynamic arrays" to store an indeterminate amount of information, so to tackle the problem from the root without using a one-off solution, I decided to make a new library in C for just that (I didn't like the other libraries I could find).

Here it is: https://github.com/TheDcoder/dynarr

The code is very simply and is bascially a wrapper around realloc.

Fun Fact: I like pronouncing the name as "Dinner", though I guess one can also call it "Diner" depending on how they pronounce the "y" 🙂

Expect more updates in the coming days!

Source: https://forum.dtw.tools/d/5-easycodeit-progress-parsing-statements/14

[TheDcoder]

Quote

EasyCodeIt Update: Our hand-written parser is going away

Hi everyone,

It has been roughly 2 months since I have provided any sort of update... certainly one of the longer gaps I took while working on this project.

I haven't done much work in this period, as I kept stumbling across little things that pop up while working on the parser, and that has been a major source of procrastination for me... and honestly, I didn't have the mental energy to deal with all of the small details involved in writing a parser by myself. All my mental energy was going towards other stuff in life, mainly my day job which puts food the my table... and the remaining I had to split on recreation and working on this project.

The problem is that this work has been more tedious than fun, so for that reason, I have decided to abandon the hand-written parser (for now at-least).

The sad part ends there, because I have no intention to stop work on this project, which is to finally bring a simple and easy to use scripting language to Linux 😁

---

After some discussion with more experienced members of the C community, I have come to know that I have been reinventing the wheel mostly with my parser. Initially I have been hesitant to use external tools to generate a parser, partly because I didn't want to learn how to use them.

Now after my raw experience with doing it the old straight-forward way, I have decided that being practical is more important... obviously I under-estimated what it would take to write a full parser by myself, it is task comparable to actually implementing the rest of the language, which actually does the work... and it is the fun bit, which parsing is not.

The tools I chose to use for the job are lex and yacc, these are the OG tools that were (and in some, still are) used in popular C compilers, so if those are good enough for that, they should definitely be more than sufficient for us!

And obviously, they are tightly integrated with C and produce C code for parsing, so that is easy to integrate them into our existing code.

I have been reading the book lex & yacc, 2nd Edition by John Levine, Doug Brown, Tony Mason to learn how to use them, the first chapter (available to read for free) gives a very good introduction, I was blown away at how simple it was, compared to reading all those technical research papers on parsing techniques 😵.

I regret not looking into it earlier, I probably would have made this decision sooner if I had.

I have still yet to get into the second chapter which explains it all in detail, and then I will start experimenting with code to replace our existing code (RIP).

Hopefully this means we can skip the hard parts of parsing and get into the meat, which is actually executing the code and making it do things.

I will keep you guys updated, hopefully the next one won't take another two months 🤭

Source: https://forum.dtw.tools/d/26-easycodeit-update-our-hand-written-parser-is-going-away

Also don't forget to make an account at the forum if you want to catch all of my updates on the latest... you can also do other stuff, but that is up to you.

[JockoDundee]

28 minutes ago, TheDcoder said:

I regret not looking into it earlier, I probably would have made this decision sooner if I had.

Nah, by coming to terms with some of the nuances of writing a parser from tabla rasa, you'll be better equipped to integrate someone elses.

32 minutes ago, TheDcoder said:

Initially I have been hesitant to use external tools to generate a parser, partly because I didn't want to learn how to use them.

I know what you mean.  I hate learning someone else's holier than thou, be-all end-all, over-generalized abstraction with its inscrutably whimsical syntax, when I can be creating my own self-indulgent, turgid and pompus opus

[Jos]

Funny...lex & yacc is what is used already for parsing au3 files in au3check.

Jos

[JockoDundee]

10 minutes ago, Jos said:

Funny...lex & yacc is what is used already for parsing au3 files in au3check.

Rare is the moment when a humble member can seize on an apparent staff contradiction in an effort to trigger a game of mod vs. mod.

Rarer still is the chance to lay the groundwork for an epic dev vs. dev clash...

 

 

[Jos]

10 hours ago, JockoDundee said:

Rare is the moment when a humble member can seize on an apparent staff contradiction in an effort to trigger a game of mod vs. mod.

What again is the contradiction?

[TheDcoder]

10 hours ago, Jos said:

Funny...lex & yacc is what is used already for parsing au3 files in au3check.

I recall you mentioned that a while ago when I was still looking for information to get started on writing the parser. Not really a surprise that the tools are common, because they are the best and somewhat de facto for writing parsers according to my research, still cool though

@JockoDundee Thanks for your words... even though I didn't understand half of them

10 hours ago, JockoDundee said:

Rarer still is the chance to lay the groundwork for an epic dev vs. dev clash...

Unfortunately I don't think it is the case here... you might have misunderstood what jpm said in the mentioned post.

He said that Jon could give them "a copy of what is used in Au3Check" as AutoIt is not based on Lex and Yacc... which means while AutoIt itself doesn't use it, Au3Check (a separate program) does!

Which is the same thing as what Jos said

P.S Quite the coincidence that Jon decided to update the forum on the same day that I decided to post an update on the project

[JockoDundee]

8 minutes ago, Jos said:

What again is the contradiction?

No actual contradiction, as eloquently pointed out by @TheDcoder.
Only an apparent contradiction, as interpreted by yours truly, in misreading @jpm’s response partially due to my conflation of Au3Check with AutoIt proper, and partially being blinded by an all-consuming giddiness over the prospect of all out lexical conflict.

tl dr;

Apologies, @Jos

*And let  this be a lesson to those who wish to wield weasel words wisely.  Note the seemingly casual use of “apparent” in my OP, but now coming in quite handy by giving me the crux of my implied defense.  

 

[TheDcoder]

Quote

I have finished reading all of the main chapters of the book (Lex & Yacc)... except the Chapter about writing a parser for the whole SQL language, which was just repeating what the previous chapters already established anyway 😒

I glossed over the bits in the secondary chapters, so I guess I am an expert in Lex & Yacc now 🤠

Now to actually start writing something with all the knowledge I have amassed, LOL. Expect an update in the upcoming weekend 😁

Source

[TheDcoder]

Quote

I just dumped my entire uncommitted code into a single commit for archival purposes.

Now I can work with a clean working directory and start porting the lexer (tokenizer) into Lex and the rest of it into Yacc for parsing.

Wish me luck guys! 🤞

(Source)

Sorry for the second post, but it was kind of important so I had to mirror it here. You should really join the dedicated forum if you wish to get all updates and would like to participate in any way

[Earthshine]

This project is NNNNN weeks LATE! our customers are going to be furious! LOL, nah, kidding, good job

[TheDcoder]

Quote

I'd done a terrible thing, and I am cursing myself right now 🤬

Turns out there was a sequel to the "Lex & Yacc" book, called "flex & bison" which is 2 decades newer than the first book, which I read ☹️

I searched for a newer book, but couldn't find any... probably because I kept searching for "Lex" and "Yacc", not "Flex" and "Bison". I was aware of them, they are newer implementations of the old tools, but apparently they are now considered separate and people stopped calling bison grammar "yacc" 🤬

Geez, now I need to go through that book and find out what new stuff they added... it is probably for the best so that I can write a better parser.

Sorry guys but it looks like I need a couple more days before I can start work on the parser.