Setting up a Queuing System

[jchd]

Trying is not the answer. You may well "try" for a week and not fall into the trap. That doesn't prove anything. You might fail just seconds after terminating the test, or days.

The question is: do Ini* function use file locking at all and does this file locking work correctly in all cases?

My guess: no. Because .ini files are not supposed to be in concurrent R/W access context: it isn't their purpose and they are fading away, so little chance to see things improving in this area.

On the contrary, something like the registry or SQLite are designed to stand concurrent accesses, with the registry not offering transactions AFAIK. Being a big fan of SQLite I'd go for a local DB in WAL journaling mode (which allows one writer and multiple readers concurrently).

EDIT: transactional registry operations are reserved to Windows Server editions >= 2008, so not a good choice for Joe Doe PC.

Edited August 5, 2013 by jchd

[Edano]

;Script1 writes
$count=0
While 1
    Sleep(200)
    Do
        $x=IniWrite("temp.ini","test",$count,@SEC)
    Until $x>0
    $count+=1
WEnd

.

;script 2 reads and deletes
While 1
    Sleep(50)
    Do
        $x=IniReadSectionNames("temp.ini")
    Until @error=0
    Do
        $y=IniReadSection("temp.ini",$x[1])
    Until @error=0
    Do
        $z=IniDelete("temp.ini",$x[1],$y[1][0])
    Until $z>0
    MsgBox(0,$y[1][0],$y[1][1])
WEnd

.

better make a longer sleep if you find an empty file .... until it gets written again

Edited August 6, 2013 by Edano

[Edano]

Trying is not the answer. You may well "try" for a week and not fall into the trap. That doesn't prove anything. You might fail just seconds after terminating the test, or days.

The question is: do Ini* function use file locking at all and does this file locking work correctly in all cases?

My guess: no. Because .ini files are not supposed to be in concurrent R/W access context: it isn't their purpose and they are fading away, so little chance to see things improving in this area.

On the contrary, something like the registry or SQLite are designed to stand concurrent accesses, with the registry not offering transactions AFAIK. Being a big fan of SQLite I'd go for a local DB in WAL journaling mode (which allows one writer and multiple readers concurrently).

EDIT: transactional registry operations are reserved to Windows Server editions >= 2008, so not a good choice for Joe Doe PC.

.

okay you guess, but this is not very productive here.

[jchd]

Sorry but guessing that concurrent R/W accesses to a flat file will work flawlessly over time is much more dangerous that you seem to believe.

I stand by my words until it's known that AutoIt Ini* function lock files correctly. Currently only Jon can tell for sure.

[Edano]

melba told me recently, that it does. there is no other way without locking, he said.

[jchd]

Maybe. We shall know shortly.

Anyway even if the individual functions use file locking I see no way to protect an update sequence in an application from being fooled by a concurrent program writing to the same file.

[Edano]

i have no doubt that sql would be the more secure way, but it's kind of overkill for a simple desktop chat application. i hope the OP tries it out and reports. we don't yet know the frequency of read write processes.

[jchd]

I see no reason to advise a design subject to "some" failure when there exists a simple approch with zero probability of failure, whatever low the failure rate is.

Edited August 6, 2013 by jchd

[Edano]

in all my years of coding, i've never seen an ini file failing.

[jchd]

The issue is with updates. Like you said in the first post of yours in this thread you need to read (say a section), process it and write it back. If the concurrent process writes in the middle of your processing, you're going to rewrite only the data you know about, ignoring the data just written under your feet.

Because we don't have a way to make the RMW (Read, Modify, Write) operation atomic, you can only rely on chance that such concurrent access don't occur. You say you have a chat with 2 processes, one constantly adding stuff while the other is constantly reading. I agree that it poses no problem in this simple case.

But here we need to invoke ReadSection, then process what was read, then WriteSection with modified data. That's where things get catchy.

[Zeerti]

I was grasping what was going on and suddenly *swoosh* right over my head. Though what I am  reading sounds like there is a chance with the way that Edano provided could end up losing some of the requests.

Would a way around this be to have the script only write to the ini file and when it is opened have it clear the ini so the file doesn't become enormous?

thinking more about this, how would I have the parent program know to use the last entry in the ini opposed to the start?

Edited August 6, 2013 by Zeerti

[jchd]

"Enormous" is very relative in the case of an INI file since sections are limited to 32Kb. Also INI are ANSI-only (whatever that means) and don't natively accept native AutoIt strings which use a large subset of UTF16.

[Edano]

I was grasping what was going on and suddenly *swoosh* right over my head. Though what I am  reading sounds like there is a chance with the way that Edano provided could end up losing some of the requests.

Would a way around this be to have the script only write to the ini file and when it is opened have it clear the ini so the file doesn't become enormous?

thinking more about this, how would I have the parent program know to use the last entry in the ini opposed to the start?

.

it's up to you. you can delete the ini file when you want. you can continuously check the size. you can read the last entry instead of the first, but i thought you definitely wanted queing ?

Edited August 6, 2013 by Edano

[Zeerti]

"Enormous" is very relative in the case of an INI file since sections are limited to 32Kb. Also INI are ANSI-only (whatever that means) and don't natively accept native AutoIt strings which use a large subset of UTF16.

 

AutoIt encodes in UTF16? I know that ANSI is 8 bit, but I'm not entirely sure 

When I say enormous I was implying the total file size, not the individual sections.

 

.

it's up to you. you can delete the ini file when you want. you can continuously check the size. you can read the last entry instead of the first, but i thought you definitely wanted queing ?

 

I most certainly do! I was saying that instead of deleting the ini entry as it is processed let entries build up and then when the program is launched clear out the file so it doesn't get to large and bog the script down.

Edited August 6, 2013 by Zeerti

[Edano]

if the "parent script" also writes something, it would be put in another file. the ini is only for data exchange. and for a data exchange in both directions => use two inis.

Edited August 6, 2013 by Edano

[Zeerti]

if the "parent script" also writes something, it would be put in another file. the ini is only for data exchange. and for a data exchange in both directions => use two inis.

 

Could you elaborate on this? I don't understand why you would need two ini files.

[Edano]

Could you elaborate on this? I don't understand why you would need two ini files.

.

no. i don't need two ini files. but someone mentioned that two processes write in one ini. as far as i understood, one script writes, the other reads and deletes. so one file is enough.

we strayed a bit from the point during that cross-posting.

Edited August 6, 2013 by Edano

[Edano]

sorry, good night for today. it's 3:15 here.

[MHz]

What I would like for it to do is set up a que, what I assume would be done via an array push of some sort, so that it does the requests in the order received. I've tried a couple of different methods without any success so If I can get some assistance on this that would be great! I have included the code below.

An array seems like a good idea to me. I consider that you had a good idea from the start. Or perhaps I am just insane for stating this.

Example from _ArrayAdd() page with a couple of lines added.

#include <Array.au3>

Local $avArray[10]

$avArray[0] = "JPM"
$avArray[1] = "Holger"
$avArray[2] = "Jon"
$avArray[3] = "Larry"
$avArray[4] = "Jeremy"
$avArray[5] = "Valik"
$avArray[6] = "Cyberslug"
$avArray[7] = "Nutster"
$avArray[8] = "JdeB"
$avArray[9] = "Tylo"

_ArrayDisplay($avArray, "$avArray BEFORE _ArrayAdd()") ; show queue
_ArrayAdd($avArray, "Brian") ; add Brian to end of queue
_ArrayDisplay($avArray, "$avArray AFTER _ArrayAdd()") ; show queue
_ArrayPush($avArray, "Zeerti") ; push JPM off the top of the queue and push Zeerti onto the queue
_ArrayDisplay($avArray, "$avArray AFTER _ArrayPush()") ; show queue

Look at the same page and view the related functions to also help with the queue in the array. Is there a need for a hard copy of the queue while the script is running unless perhaps on the closure of the script?

[Zeerti]

An array seems like a good idea to me. I consider that you had a good idea from the start. Or perhaps I am just insane for stating this.

Example from _ArrayAdd() page with a couple of lines added.

#include <Array.au3>

Local $avArray[10]

$avArray[0] = "JPM"
$avArray[1] = "Holger"
$avArray[2] = "Jon"
$avArray[3] = "Larry"
$avArray[4] = "Jeremy"
$avArray[5] = "Valik"
$avArray[6] = "Cyberslug"
$avArray[7] = "Nutster"
$avArray[8] = "JdeB"
$avArray[9] = "Tylo"

_ArrayDisplay($avArray, "$avArray BEFORE _ArrayAdd()") ; show queue
_ArrayAdd($avArray, "Brian") ; add Brian to end of queue
_ArrayDisplay($avArray, "$avArray AFTER _ArrayAdd()") ; show queue
_ArrayPush($avArray, "Zeerti") ; push JPM off the top of the queue and push Zeerti onto the queue
_ArrayDisplay($avArray, "$avArray AFTER _ArrayPush()") ; show queue

Look at the same page and view the related functions to also help with the queue in the array. Is there a need for a hard copy of the queue while the script is running unless perhaps on the closure of the script?

this is what I was initially thinking of doing, but I like the idea of an ini file doing it simply because I don't know how it works and my programming is rather limited. Have to expand my horizons for the future!