Multiprocessing UDF that likes same name of module in Python

[oceanwaves]

  We all know Autoit has no threading concept. And I searched the forums, some peoples have written multiple process framework, but they look not very popular.

  Python support threading, but Python also have a module, it names multiprocessing. According to Python PEP: Python's GIL causes only one threading can be run at same time (CPython interpreter). So Python provide multiprocessing module, it is real parallel running. So I decide to simulate the Python's module and import into Autoit. It also my first UDF in forums.

  In shortly, I write 4 UDFs, Multiprocessing.au3, SimpleQueue.au3, SimpleSemaphore.au3 and SimpleLock.au3. Multiprocessing.au3 provides method that let your function will run in another process rather than main process. SimpleQueue.au3 provides a queue (FIFO) that can be used in IPC. SimpleLock.au3 as its name, if you familiar threading concept.

  Limitation: Pass Autoit type data into other process or queue, I use JSMN.au3 to data serialization. According to JSMN, multiple dimensions array, COM object and Dll struct etc. You can check at '?do=embed' frameborder='0' data-embedContent>>  The UDFs has added at main script head also must add at Multiprocessing.au3. Another attention is please place Multiprocessing.au3 to end of #include, then Multiprocessing.au3 can find your what UDF need to use.

  Detailed information in these UDFs:

  When you include a UDF in your script, actually the UDF will run before your main script running. So I just determine the current process is main process or not (according to CmdLine[]). If it is main process, it will do some prepare action, pass your function parameter into memory then call the same script or exe --------child process. When child process startup, the Multiprocessing.au3 also running first and get parameter from memory, then use Autoit function Call().

  This UDF also provide some methods to manger child process, it used MS jobs object to manager them. You can see the information at http://msdn.microsoft.com/en-us/library/windows/desktop/ms684161%28v=vs.85%29.aspx

 But interesting for this, A include B, B can find function in A but can't find other UDFs that are referred by A, so must add these UDF at B head. In attention, interpreters for 3.3.10.2 and 3.3.11.4 are different. Perhaps Autoit develop team can explain it,

  For SimpleQueue.au3, just using file mapping concept of MS, it also used JSMN.au3 to data serialization. Support data are limited by JSMN.au3. The queue is stored in file mapping. Set up a specific size memory and a lock, then each process can access it without conflict.

SimpleLock.au3 and SimpleSemaphore.au3 can get detail information at MSDN.

At last, I hope Autoit has own data serialization UDF that can support multiple dimension array and other can’t be supported by current JSMN.au3.

If you find any bug, please point out to me, I’ll do my best to fix it.

Sorry for my poor English.

-----------------------History----------------------------------------

2014.5.21. Update Multiprocessing.au3 ---- add new function AllSubProcess_Close

                  Update SimpleSemaphore.au3 ---- add function head for each function and rewrite some function, remove redundant virables.

                  Upload SimpleSemaphore_testing.au3 example. The example shows it has 3 semaphores and 6 child process wants to connect to a sqlite database, but I limit in same time maximum connection is 3, so if one child process has not gotten a semaphore, it must be waiting unless other child process release a semaphore. When you run the example script, maybe must know some standard SQLite UDF, you can check it on the Autoit help file.

Multiprocessing.zip

Multiprocessing_testing.zip

Multiprocessing.au3

SimpleSemaphore.au3

SimpleSemaphore_testing.au3

Edited May 21, 2014 by oceanwaves

[legend]

Might be a stupid question, but is it true multi threadding? I saw many attempts to do multi threading with autoit, but none of them

were true multi threading.

[spudw2k]

Might be a stupid question, but is it true multi threadding? I saw many attempts to do multi threading with autoit, but none of them

were true multi threading.

In short, no; Multithreadding is not the same as multiprocessing.

Multithread allows simultaneous, parallel operations within a single process.

Multiprocessing spawns individual processes that uses timining to coordinate them.  I like to think of it as poor mans multithreading through timing.

[oceanwaves]

Might be a stupid question, but is it true multi threadding? I saw many attempts to do multi threading with autoit, but none of them

were true multi threading.

No, thread concept you can refer to here: http://en.wikipedia.org/wiki/Thread_%28computer_science%29. Multiprocessing consists with process. In other words, you can think thread from your body, like hand, arms. Multiprocessing likes human cooperation. And this UDF is Multiprocessing UDF, not Multithreading UDF. Autoit doesn't support thread concept.

[MagicSpark]

oceanwaves,

nice. I'm just playing around with the scripts at the moment.

Can you please give an example with Semaphore's ?

Oh, just one question:

Can I terminate (say: send some close command) to all processes in job from master process?

Thank you.

Edited May 10, 2014 by MagicSpark

[oceanwaves]

oceanwaves,

nice. I'm just playing around with the scripts at the moment.

Can you please give an example with Semaphore's ?

Oh, just one question:

Can I terminate (say: send some close command) to all processes in job from master process?

Thank you.

HI MagicSpark,

    For your 1st question, I'll give you a example, maybe tomorrow?

    Terminate all child process, yes, it can do to it, but not recommand to do it. The action can be deemed ProcessClose functionI (it is a force kill action).  I copy from Python's document to explain : "if the process has acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock." .

     MS has provided related API to do it : TerminateJobObject, detailed you can see at http://msdn.microsoft.com/en-us/library/windows/desktop/ms686709%28v=vs.85%29.aspx

     If you think the function is working for you, I'll add it to the UDF, soon. XD

[MagicSpark]

Hello oceanwaves,

an example script for semaphores would be very nice.

For teminating or shutdown childs the TerminateJobObject looks good. Maybe you implement it into the UDF.

Thank you in advance.

[oceanwaves]

Sorry for my lazy. update Multiprocessing.au3 and SimpleSemaphore.au3. Add example for SimpleSemaphore.au3

[drego]

This sort of thing is good though because even though it isn't multithreading within in the same process it does allow the OS to make use of multiple cores across the multiple processes you spawn. I'm currently in the middle of making my own forking/job queueing system in AutoIt when I came across this. I might use it. What are some other popular UDFs for this? I would like to look them all over to see where we're at with this sort of thing.