A cross-platform implementation of the AutoIt language

[TheDcoder]

@argumentum That is an interesting choice, can you share the reasoning for 8 kilobytes for each chunk?

I chose 1 kilobyte because most text files are in that range and thus are nicely dividable without wasting too much space in the last chunk.

[argumentum]

if you're gonna read from the disk then, what is the cluster. You're gonna find that 4 k is becoming standard. Should optimize for bigger and faster hardware. Memory is abound. But like everything else, try different values and build a table of results. You may find that is standard may very well be based in old solutions for old hardware. In any case the OS will read ahead so no big deal. But if the slowest part is reading from spinning rust ( HDD ), the best is to optimize that. The rest of the function is CPU and memory and that is not a bottle neck. Try and see. Try in old hardware and the newest you can get.

And since I have no experience with what I'm presenting, I hope to not have wasted you time or worse, mislead you.

Edit: from https://stackoverflow.com/questions/10698339/what-would-be-an-ideal-buffer-size 

Optimum buffer size is related to a number of things: file system block size, CPU cache size and cache latency.

Most file systems are configured to use block sizes of 4096 or 8192. In theory, if you configure your buffer size so you are reading a few bytes more than the disk block, the operations with the file system can be extremely inefficient (i.e. if you configured your buffer to read 4100 bytes at a time, each read would require 2 block reads by the file system). If the blocks are already in cache, then you wind up paying the price of RAM -> L3/L2 cache latency. If you are unlucky and the blocks are not in cache yet, the you pay the price of the disk->RAM latency as well.

This is why you see most buffers sized as a power of 2, and generally larger than (or equal to) the disk block size. This means that one of your stream reads could result in multiple disk block reads - but those reads will always use a full block - no wasted reads.

..that sounds better than my writing   

Edited June 12, 2020 by argumentum
better argument

[TheDcoder]

@argumentum Those are some very interesting insights, thanks for bringing them up

For now I would like to keep it simple, and not try to overly optimize... we can do that once we have the basics working

I may or may not change the default chunk size to 8 KB, as it seems to ideal when taking the block sizes of the HDD into account. We are looking at memory vs. disk speed trade-off here, and I imagine the difference would be minuscule practically.

The most important part, having an efficient "algorithm", is done. There is only one more method which is better, that is to use platform-native API to query the file size and then read all of that at once, no chunks

The current code is a good balance of efficiency combined with portability as everything is written in standard C.

[TheSaint]

I'd call that cross-dressing or cross referencing at the crossroads of good cross-platform implementation ... if you get what I mean. 

At least you're not working at cross-purposes or crossing over. Keep up the good work. 

[seadoggie01]

9 hours ago, TheDcoder said:

everything is written in standard C

Drat. Time to learn another language. Probably good anyways I understand very little low level code. I don't want to rescind my vote to work on code

[TheDcoder]

25 minutes ago, seadoggie01 said:

I don't want to rescind my vote to work on code

You don't have to, working with code doesn't mean only writing low-level C code, there are many other things which can be worked on... such as writing scripts and documentation

[TheDcoder]

No progress on coding today, but I am studying the syntax and trying to come up with the basic logic I would use to parse the code. I also realized that I lost a never-published update to my example reference script, which is a shame

However I lifted the AutoIt help file and the code examples from my VM into a local folder for easy access, this should be good enough to get me started. I guess the main syntax features which must be implemented are:

1. Variables
2. Functions
3. Common expressions

 

[TheDcoder]

I am working on a tokensizer which will be the first stage in parsing, some call it the scanner phase. It is basically taking the raw text and converting it into tokens (variable, string, number, operator, keyword etc.), which are much easier to work with in code, the next step would be syntactic analysis, which is just a fancy word to describe to process of checking if the tokens are in the right order (e.x. checking if 123 = $number is in the right order... spoiler alert: it is not in the right order).

As usual, not much work code wise but I have a plan in my mind now, and I spent most of my time today researching how enumerators work in C and if linked-lists are really the best way to store dynamically allocated data.

Another nice thing is that I have finally created a repository with the code: https://github.com/DcodingTheWeb/EasyCodeIt

The code is very basic right now, it just prints the contents of the script to the standard output, I will update it as I work on features. Don't forget to give the repository a 🌟 star if you like what I am doing

[TheDcoder]

Sorry guys, it was kind of a bad day today, the day started with a power outage which lasted for a good 4 hours, which left me muggy and irritated. After that I got busy with lunch, catching up with other stuff, watching videos etc. So couldn't really take out time for coding.

I did make some minor progress, wrote a structure for storing tokens and a skeleton function. I now have to figure out how to deal with white-space, which should be relatively easy. Hope to have something working by tomorrow

[TheSaint]

Bad luck bud .... that's the Universe(s) for you. 

I've had a word with Miss Universe, and she now promises to leave you alone (bad luck).

However, Mr. Universe might not be so obliging .... especially if he thinks you have been paying undue attention to his daughter.

And bud ...... he's got arms of steel .... so be careful when you are thinking of swearing at the Universe.

Anyway bud, good to see you are making at least some forward progress ... all sounds quite promising ... all this work on ImagineIt. 

[TheDcoder]

Sorry guys, I really dropped the ball. I could not write satisfactory code for parsing so I kept delaying work on it and then things happened. One of those things was the opportunity to work again, I have not done any freelance work since I lost my regular client due to the effect of COVID on their business, so I was excited.

I finally convinced myself to make some progress today, and I did

I have some not-so-good code which is able to parse comments... and only comments. That is the most basic thing I could find so I worked on it. Here is how it currently works:

J:\Projects\EasyCodeIt\build_win>eci test.au3
---### TOKEN ###---
Type: Comment
Data: ; This is a single line comment
---### TOKEN ###---
Type: Comment
Data: ; Mary had a little lamb
---### TOKEN ###---
Type: Comment
Data: ; Wow comments
Unknown token encountered

and the source for test.au3:

; This is a single line comment

; Mary had a little lamb
; Wow comments

MsgBox

Here is the current parsing code that I have written:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2020 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#include <stdbool.h>
#include <stdio.h>
#include "parse.h"
#include "utils.h"

const char CHR_WHITESPACE[] = {' ', '\t', '\n'};
const char CHR_COMMENT = ';';
const char CHR_DIRECTIVE = '#';

struct TokenCharMapElem {
	enum TokenType type;
	union {
		const char chr;
		const char *chr_arr;
	};
};

static void print_token(struct Token *token) {
	puts("---### TOKEN ###---");
	char *token_type;
	switch (token->type) {
		case TOK_WHITESPACE:
			token_type = "Whitespace";
			break;
		case TOK_COMMENT:
			token_type = "Comment";
			break;
		case TOK_DIRECTIVE:
			token_type = "Directive";
			break;
	}
	fputs("Type: ", stdout);
	puts(token_type);
	fputs("Data: ", stdout);
	for (size_t c = 0; c < token->data_len; c++) putchar(token->data[c]);
	putchar('\n');
}

void parse(char *code) {
	tokenize(code);
}

void tokenize(char *code) {
	char *curr_char = code;
	
	while (true) {
		// Check if whitespace
		if (chrcmp(*curr_char, (char *) CHR_WHITESPACE, sizeof CHR_WHITESPACE)) {
			// Advance to the next character
			++curr_char;
			continue;
		}
		
		// Do it all manually
		
		// Comment
		if (*curr_char == CHR_COMMENT) {
			struct Token tok = {
				.type = TOK_COMMENT,
				.data = curr_char,
			};
			for (++curr_char; *curr_char != '\n' && *curr_char != '\0'; ++curr_char);
			tok.data_len = (curr_char - tok.data);
			
			print_token(&tok);
			
			// Advance to the next character
			++curr_char;
			continue;
		}
		
		// EOF (Null terminator)
		if (*curr_char == '\0') break;
		
		// Error
		die("Unknown token encountered");
	}
}

There is definitely room for improvement, and I am working on it. Wish me luck everyone

[TheSaint]

I'm not everyone, but I wish you luck bud anyway ... get those juices flowing, climb those mountains, leap those hurdles, get down & dirty ... and most importantly have a beer believe in yourself. 

[TheDcoder]

Hello again,

I apologize for the delay, life got in the way again... as it always does

Today I made some good progress, I redesigned my code to handle one token at a time, instead of all tokens at once, this has made things simpler. I now have a tangible scanner framework, and I have added support for quite a few types of tokens!

enum TokenType {
	TOK_UNKNOWN,
	TOK_WHITESPACE,
	TOK_COMMENT,
	TOK_DIRECTIVE,
	TOK_NUMBER,
	TOK_WORD,
	TOK_OPERATOR,
	TOK_BRACKET,
	TOK_COMMA,
};

I still have to add support for macros, variables, strings etc. but that shouldn't be very hard. I also forgot to mention that handling multi-line comments turns out to be not as simple as I thought, it requires special handling and it has unique behavior, I believe it is the only multi-line (pre-processor) directive.

Anyway, here is the complete code of the parser:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2020 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#include <ctype.h>
#include <stdbool.h>
#include <stdio.h>
#include "parse.h"
#include "utils.h"

const char CHR_COMMENT = ';';
const char CHR_DIRECTIVE = '#';
const char CHR_COMMA = ',';

char CHRSET_WHITESPACE[] = {' ', '\t', '\n'};
char CHRSET_OPERATOR[] = {
	'+', '-', '*', '/', '^',
	'&',
	'=', '<', '>',
	'?', ':',
};
char CHRSET_OPERATOR_EQUABLE[] = {'+', '-', '*', '/', '^', '&', '='};
char CHRSET_BRACKET[] = {'(', ')'};

struct TokenCharMapElem {
	enum TokenType type;
	union {
		const char chr;
		const char *chr_arr;
	};
};

static void print_token(struct Token *token) {
	puts("---### TOKEN ###---");
	char *token_type;
	switch (token->type) {
		case TOK_UNKNOWN:
			token_type = "Unknown";
			break;
		case TOK_WHITESPACE:
			token_type = "Whitespace";
			break;
		case TOK_COMMENT:
			token_type = "Comment";
			break;
		case TOK_DIRECTIVE:
			token_type = "Directive";
			break;
		case TOK_NUMBER:
			token_type = "Number";
			break;
		case TOK_WORD:
			token_type = "Word";
			break;
		case TOK_OPERATOR:
			token_type = "Operator";
			break;
		case TOK_BRACKET:
			token_type = "Bracket";
			break;
		case TOK_COMMA:
			token_type = "Comma";
			break;
		default:
			token_type = "Unnamed";
			break;
	}
	fputs("Type: ", stdout);
	puts(token_type);
	fputs("Data: ", stdout);
	for (size_t c = 0; c < token->data_len; c++) putchar(token->data[c]);
	putchar('\n');
}

void parse(char *code) {
	while (true) {
		struct Token token = token_get(code, &code);
		if (!code) break;
		if (token.type != TOK_WHITESPACE) print_token(&token);
		if (token.type == TOK_UNKNOWN) die("!!! Unknown token encountered !!!");
	}
	return;
}

struct Token token_get(char *code, char **next) {
	struct Token token = {
		.type = TOK_UNKNOWN,
		.data = NULL,
		.data_len = 0,
	};
	size_t length;
	
	// Identify the token
	if (length = scan_string(code, char_is_whitespace)) {
		// Whitespace
		token.type = TOK_WHITESPACE;
		token.data = code;
		token.data_len = length;
	} else if (*code == CHR_COMMENT || *code == CHR_DIRECTIVE) {
		// Comments and Directives
		token.type = *code == CHR_COMMENT ? TOK_COMMENT : TOK_DIRECTIVE;
		token.data = code;
		token.data_len = scan_string(code, char_is_not_eol);
	} else if (length = scan_string(code, char_is_num)){
		// Numbers
		token.type = TOK_NUMBER;
		token.data = code;
		token.data_len = length;
	} else if (length = scan_string(code, char_is_alphanum)){
		// Words
		token.type = TOK_WORD;
		token.data = code;
		token.data_len = length;
	} else if (char_is_opsym(*code)) {
		// Operator
		token.type = TOK_OPERATOR;
		token.data = code;
		
		// Include the trailing `=` if possible
		token.data_len = code[1] == '=' && chrcmp(*code, CHRSET_OPERATOR_EQUABLE, sizeof CHRSET_OPERATOR_EQUABLE) ? 2 : 1;
	} else if (char_is_bracket(*code)) {
		// Bracket (Parenthesis)
		token.type = TOK_BRACKET;
		token.data = code;
		token.data_len = 1;
	} else if (*code == CHR_COMMA) {
		// Comma
		token.type = TOK_COMMA;
		token.data = code;
		token.data_len = 1;
	} else {
		// Unknown
		token.data = code;
		token.data_len = 1;
	}
	
	// Set the next code
	*next = *code == '\0' ? NULL : code + token.data_len;
	
	// Return the token
	return token;
}

size_t scan_string(char *str, bool (cmpfunc)(char)) {
	size_t len = 0;
	while (true) {
		if (!cmpfunc(*str)) break;
		++len; ++str;
	}
	return len;
}

bool char_is_whitespace(char chr) {
	return chrcmp(chr, CHRSET_WHITESPACE, sizeof CHRSET_WHITESPACE);
}

bool char_is_alpha(char chr) {
	return isalpha(chr);
}

bool char_is_num(char chr) {
	return isdigit(chr);
}

bool char_is_alphanum(char chr) {
	return char_is_alpha(chr) || char_is_num(chr);
}

bool char_is_opsym(char chr) {
	return chrcmp(chr, CHRSET_OPERATOR, sizeof CHRSET_OPERATOR);
}

bool char_is_bracket(char chr) {
	return chrcmp(chr, CHRSET_BRACKET, sizeof CHRSET_BRACKET);
}

bool char_is_not_eol(char chr) {
	return chr != '\n' && chr != '\0';
}

Example output from my test:

expandcollapse popup

J:\Projects\EasyCodeIt\build_win>eci test.au3
---### TOKEN ###---
Type: Directive
Data: #include <Motivation.au3>
---### TOKEN ###---
Type: Comment
Data: ; This is a single line comment
---### TOKEN ###---
Type: Comment
Data: ; Mary had a little lamb
---### TOKEN ###---
Type: Comment
Data: ; Hello from EasyCodeIt!
---### TOKEN ###---
Type: Word
Data: MsgBox
---### TOKEN ###---
Type: Bracket
Data: (
---### TOKEN ###---
Type: Number
Data: 0
---### TOKEN ###---
Type: Comma
Data: ,
---### TOKEN ###---
Type: Number
Data: 0
---### TOKEN ###---
Type: Comma
Data: ,
---### TOKEN ###---
Type: Number
Data: 1
---### TOKEN ###---
Type: Operator
Data: +
---### TOKEN ###---
Type: Number
Data: 2
---### TOKEN ###---
Type: Operator
Data: -
---### TOKEN ###---
Type: Number
Data: 3
---### TOKEN ###---
Type: Operator
Data: *
---### TOKEN ###---
Type: Number
Data: 4
---### TOKEN ###---
Type: Operator
Data: /
---### TOKEN ###---
Type: Number
Data: 5
---### TOKEN ###---
Type: Operator
Data: ^
---### TOKEN ###---
Type: Number
Data: 6
---### TOKEN ###---
Type: Bracket
Data: )
---### TOKEN ###---
Type: Comment
Data: ; The result is 2.999232 if anyone is wondering

And here is the script that I tested with:

#include <Motivation.au3>

; This is a single line comment

; Mary had a little lamb

; Hello from EasyCodeIt!

MsgBox(0, 0, 1 + 2 - 3 * 4 / 5 ^ 6)
; The result is 2.999232 if anyone is wondering

 

I hope to have more updates soon and hopefully the scanner/tokenizer will be completed! The next step would be actually analyzing the tokens and constructing a source tree from it (syntactic analysis is the buzz word)

Edited June 25, 2020 by TheDcoder

[TheSaint]

I'd like to give a token of my appreciation, but I see you already have many. 

But still, if you ever run out, you know where you can find me. 

By the way ......... good work ......... keep it up ......... for science.

[rcmaehl]

Let me know when you have an installer for the alpha build.

[TheDcoder]

@rcmaehl Not likely soon, I am busy with other freelance projects right now (as well as a few other personal things), and it will take a bit of time before I have a fully working interpreter... even then I would most likely just do a zip release without an installer

Thanks for showing your interest, I will definitely let you and everyone know when I have an alpha build

[TheDcoder]

I just finished adding support for all of the leftover types of tokens, and now I have a fully functional tokenizer!

The latest code is available on GitHub, and I have also uploaded the latest binary builds here so that you guys can test and give me feedback:

There are two files, one each for Windows and Linux, and you guys are smart enough to figure out which is for which

Right now the interface is very simple, just download the binary and supply it an .au3 file as the first command-line argument, and it will print out all the tokens. Please report any unknown token errors which occur in valid scripts!

Have fun,
TheDcoder.

[jpm]

Hi,

Just looking at the tokens I don't know if having only one token for [ and ] for example will not make the use of the token more complicated.

more why not have a token for Integer different from the float (ex 1.2e+4)

more how do you plan to support unicode

Good luck for the next steps

[TheDcoder]

Hi @jpm! It is a privilege to have you commenting here

6 minutes ago, jpm said:

Just looking at the tokens I don't know if having only one token for [ and ] for example will not make the use of the token more complicated.

Indeed, currently I am using a generic Token structure, it is just meant to make scanning the tokens easier for the next step in the parser. I will make specialized Token structures for each token type which will contain the primitive data.

9 minutes ago, jpm said:

more why not have a token for Integer different from the float (ex 1.2e+4)

I have not added support for floats yet, partly because I have rarely seen them being used in scripts. I will add support for it if someone asks for it though, right now I am concentrating on getting a simple and functioning proof-of-concept interpreter running, as opposed to a full-featured one. We can build on the PoC to add more features and eventually support most of them in AutoIt

14 minutes ago, jpm said:

more how do you plan to support unicode

I have given some thought to this, but at this point it is just up in the air, I am using standard C functions (which are locale-sensitive) to identify the types of characters (along with some hardcoded characters), but in the future I will eventually use a robust UTF-8 text processor.

23 minutes ago, jpm said:

Good luck for the next steps

Thank you!

[argumentum]

S:\198464-a-cross-platform-implementation>eci test.au3
---### TOKEN ###---
Type: Word
Data: MsgBox
---### TOKEN ###---
Type: Bracket
Data: (
---### TOKEN ###---
Type: Number
Data: 0
---### TOKEN ###---
Type: Comma
Data: ,
---### TOKEN ###---
Type: String
Data: this
---### TOKEN ###---
Type: Comma
Data: ,
---### TOKEN ###---
Type: String
Data: that
---### TOKEN ###---
Type: Bracket
Data: )

took me some time to "get it". Do include a sample script and a batch file ( for those like me ? )

@Echo OFF
REM example batch
eci.exe test.au3
pause

and a bash file for those in Linux. ( i don't know Linux, I think is called a bash file )

Also the reference in the downloads to "https://www.autoitscript.com/forum/topic/198464-a-cross-platform-implementation-of-the-autoit-language/?do=findComment&comment=1459435" so those like me can find what is the download related to.

Other than that, kudos.