[Solved] IPC between system and user

[argumentum]

There is this topic on Examples about IPC. My question is: What is the best IPC to work with between a script running as SYSTEM level, User level, and Administrator level to interact with each other ? 

Thanks

Edited April 1, 2018 by argumentum

[RTFC]

Hi, interesting problem. TCP/UDP should work. NamedPipes and regular shared memory likely won't work. I would guess Mailslot and injecting Windows message queue won't work either (or only one-way at best), but haven't tested this. I did just run a successful test with my HighMem  UDF (link in sig, Test 3 scripts A and B, any number of processes can share access to the same region(s) ) where one script is ADMIN and the other is USER  (note that Scite output is not visible for admin when starting Scite as user and script has #RequireAdmin), but comms and beeps upon successful I/O still work, as does user output in the other Scite window); haven't ever tested HighMem with SYSTEM account, but it might be worth a shot (I'd be interested to hear how it went); it does require x64-mode (machine/OS/Au3) though.

EDIT: So I couldn't resist, and hooked up wraithdu's adaptation of arcker's great services UDF to elevate script HighMem_Test3_A.au3 to  SYSTEM level, and to my delight and surprise, communicating to/from user-level script works fine both ways. And because the shared mem is virtual, access is fast, and there's no size restrictions (easily >4GB).

Edited March 30, 2018 by RTFC
more testing

[Bilgus]

Depending on how much data you are transfering you could use the eventlog functions..

I only mention it because you can get notifications with _EventLog__Notify ( $hEventLog, $hEvent )

Edited March 30, 2018 by Bilgus

[argumentum]

4 hours ago, RTFC said:

Hi, interesting problem

yeap, and the "Sorry, this test ONLY works if the following three conditions are all met:", none are.

Is all 32bit compile as it should work in as many OS versions as possible. 

In what I'm looking to do is to send commands only, so, one-way only is fine, as I can make each script send the one-way back, by including the originating "address" as part of the message. Even as UDP would be the obviously simple solution, the memShare would be perfect to avoid the firewalls and whatnot, interrupting communications between processes.

I was looking at _WinAPI_CreateFileMapping() and I was thinking of having a go at it, but then again, I have no idea what all those DllStruct are. So I'm not well suited to make an IPC UDF out of it. If you'd like to give it try, I figure, it would only take rearranging the example to be a UDF ( but other than copy/paste, I have no idea of the working of the code ).

Thanks

[RTFC]

23 minutes ago, argumentum said:

Is all 32bit compile as it should work in as many OS versions as possible. 

Luckily for you, I just tested the original HighMem Test3 scripts running as 32-bit and everything still works; of course, you're shared memory region should be significantly less than 2 GB in that case. Just disable "#AutoIt3Wrapper_UseX64=Y" everywhere, and remove the @Autoitx64 macro checks.

23 minutes ago, argumentum said:

looking at _WinAPI_CreateFileMapping()

Definitely the way to go, IMHO. Under the hood, that's exactly what HighMem uses too.

Edited March 30, 2018 by RTFC

[AdamUL]

Have you looked at the MailSlot UDF?  It may work for what you need.  

 

Adam

 

[argumentum]

1 minute ago, AdamUL said:

Have you looked at the MailSlot UDF?  It may work for what you need. 

it does not  

[argumentum]

2 minutes ago, RTFC said:

running as 32-bit and everything still works; of course, you're shared memory region should be significantly less than 2 GB

cool, All I'll ask my other script is to "restart" or, "whatYouUpto", so that count as less than 2GB  
I'll do the mods. and try right now.

[RTFC]

7 minutes ago, AdamUL said:

MailSlot

is restricted to 64KB (minus 29-byte header) on localhost per transfer, and to datagrams (424 bytes, minus 12-byte header?) between machines on a LAN (AFAIK). And if you're going to send datagram-sized packets, you might as well use TCP. But both MailSlot and TCP ports can be blocked by AV software/firewalls, so more prone to unexpected failure in (unknown/changing) user environments. And I have no clue how to tell AV to allow MailSlot traffic if it doesn't already by default. So it comes with caveats. Nevertheless, great UDF, and a bedrock-reliable form of IPC if it is not blocked.

[argumentum]

25 minutes ago, RTFC said:

Just disable "#AutoIt3Wrapper_UseX64=Y" everywhere, and remove the @Autoitx64 macro checks

Did that and worked  
Now, these examples write an INI file, in my case, declaring 1 MB, the files has:

[PID]
PID=16120
[ALLOC1]
sizeInBytes=512
relativeOffset=0x00000000
[ALLOC2]
sizeInBytes=1024
relativeOffset=0x00010000

So I can assume that only the PID is needed and the ALLOC 1 & 2 will always be the same. If that is so, all that is needed is the PID of a process, in a one-way-ish ( as in I send "234" and I can get an "ACK" back as to, get acknowledgment of data received and change the data and "flag" ready for next msg.
Along with _Crypt_DecryptData() to make it not accept data from where the CryptKey is obviously wrong. ..... 

If all this is so, would you like to put it together as, a universal ( for the AutoIt universe anyway, tho, it will be interoperational with any other language in Windows ), UDF in the example forum, to have a final and definitive solution to the IPC, that always come up every so often ?

[argumentum]

@RTFC, looking at the HighMem UDF examples, I see:

If @error Then
    ConsoleWrite("Error: Allocation failed with error: " & @error & @CRLF)
    Exit(1) ;   on Exit, _HighMem_CleanUp() is automatically invoked
EndIf

and when AutoIt crashes, it returns 1. Could you change it to "2", to know that it failed the script and is not a crash ?

PS: I just realized that what I posted here is not related to the UDF and ... is just nonsense, sowy  

Edited March 30, 2018 by argumentum
oops

[RTFC]

15 hours ago, argumentum said:

Did that and worked

Regarding those  structs I used in that Test3 example, the first one is just a dummy buffer to illustrate the relative offset you need to access the second struct. Note that the absolute address of the allocation in virtual memory is PID-specific, so each process obtains its own mapping of the same virtual region, and the (same) offset of a particular struct will be relative to that PID-specific base within each process. With me still? I'm also using the PID to generate a process-specifc unique name for the allocated region, so multiple processes can each allocate multiple regions (and multiple structs within those, if necessary), and each process can predict a memory-mapped region's name once it knows what process (PID) created it. The first allocation within each region of course has an offset of zero, so if you only need it for basic comms, maybe that's enough (two processes each create one write buffer, read by the other, so then you don't need to bother with mutex negotiation to ensure you're not both writing to the same allocation at the same time).

15 hours ago, argumentum said:

would you like to put it together as, a universal ( for the AutoIt universe anyway, tho, it will be interoperational with any other language in Windows ), UDF

Errm, not really, sorry, but thanks for asking. I only cobbled HighMem together for my E4A matrix computing environment, because I need to manipulate large datasets (>4GB, hence x64-mode required). If you think it might be of more general use in an IPC context, just write your own wrapper UDFs and include HighMem.au3 (I think only the test scripts use x64-mode anyhow, the main UDF does not require it, as we just discovered). It shouldn't take more than a rewrite of the Test3 scripts in UDF form. Maybe provide some snazzy user options such as storing strings in a 64K string struct, or some predefined codes for basic comms operations. Have a look at my Pool environment for some examples, if you wish.

It's definitely doable and probably fun (basic functionality is already present), but I simply cannot spare the time for new UDFs these days, drowning in CUDA at the moment... But it's great that you find it useful!

Edited March 31, 2018 by RTFC
typos

[argumentum]

7 minutes ago, RTFC said:

With me still?

 

7 minutes ago, RTFC said:

Maybe provide som snazzy user options such as storing strings in a 64K string struct, or some predefined codes for basic comms operations

I only wish I had an idea of what you are saying, code wise.

I'll wait for someone to put it together as I'm not the one that can tackle this. But is good to know that is doable. That means that is just a matter of time, until such UDF popup in the forum.

meanwhile I'll do it the UDP way. Thanks for your help, as you've been quite enlightening for me on this subject.  

[RTFC]

13 hours ago, argumentum said:

I'll do it the UDP way

In that case, be aware of firewall/AV restrictions on UDP port usage that may affect user environments. You may have to actively enable access first; but since your processes run at admin/system level anyhow, they have the required permissions to do so.

 

13 hours ago, argumentum said:

I only wish I had an idea of what you are saying, code wise

Apologies for being obscure. I hope you're not abandoning the HighMem option because of unfamiliarity with structs.

Structs are easy. You can think of them as a database record containing one or more fields, with each field defined (e.g., by you) to hold a specific type of data (integer, string, double precision value, and so forth). Or easier still, a struct is like a big IKEA wardrobe you design yourself, with drawers of different sizes (different variable types) and each as deep as you want it to be: one may hold a single coin (integer), another a big pile of books (strings). They can be tiny (a few bytes) to humongous in size (GBs). Best of all, you have the option of deciding exactly where you want that wardrobe to be placed in the room (memory), for example in the SW corner, 5 cm from the west wall and 10 cm from the southern one (using the optional second parameter in DllStructCreate, a 32/64 bit memory pointer). You can even define multiple wardrobes to physically overlap, mapping to the same space, but each with a different internal structure, so the same content (data) can be interpreted differently depending on whether you open a book drawer (string) in wardrobe 1, or a numeric one (e.g., integers) in wardrobe 2. So despite being fixed (structures), they're actually very flexible (to work with). In the IPC example, several processes are mapping the same-shaped wardrobe(s) as defined in the Test3 .ini file, and then each one in turn opens a drawer, takes something out (DllStructGetData), and sticks something else back in (DllStructSetData). It's no different than accessing an internal $variable, other than interacting with a memory address (DllStructGetPtr) directly rather than indirectly. As long as you know where the wardrobe sits and what (type and size of) drawers it contains (and in what order), you can manipulate their content to your heart's content. The only issue in IPC is that virtual memory is remapped to a different base address by each process, so the room is (nominally) different, but the wardrobe's position relative to its SW corner is the same. That's why those relative offsets were parsed in the example scripts (clumsily through an .ini, because it was just for illustration). DllCalls also rely heavily on them, whenever more than a single piece of data needs to be parsed or returned. Much of AutoIt's functionality, and indeed Windows OS itself, relies on dll calls (to various Windows kernel dlls), and many of those in turn rely on structs to get the job done. Therefore it might be worth your time to get to know them, especially since many programming languages provide access to them (my E4A matrices are structs, and I manipulate them from three very different programming environments). Of course, it's up to you...

Edited March 31, 2018 by RTFC

[argumentum]

6 hours ago, RTFC said:

Apologies for being obscure

You were not obscure at all, actually, well presented. It is I, that is in a hurry, and "looking for a rounder circle" without much experience in drawing a line.
@Michel Claveau posted a dll to do the _WinAPI_CreateFileMapping() and @trancexx pointed him to the DLL calls she used in her code.
You yourself use those calls in your code, so what I'll do is to use a mix of what I have in hand, and that is the example from the help file, mainly due to that in the help files example, looks to me ala mailslot, by just declaring, what in that UDF is called "address", that is in fact a name of sorts, and in this is called "The name of the file mapping object to be opened" in the FileMapping example file. So by looking to read that name object, no other info is needed. To make it a unique mapping name, I'll use "ScriptName + PID" and that would be all that is needed to talk to that script, very much so, as only the PID is needed to for the WM_COPYDATA ( if one can find the window ).

That is the approach I'm taking until I get more familiar with WinAPI FileMapping. I've been using mailslots for the longest time and I know what I can get out of it, and it's caveats, but I have no clue of the caveats in FileMapping.

So I've dropped the UDP idea and I'm jumping to this, as its the best approach, for a A/V proof, system/Admin/User proof, IPC. Unless I find caveats that send me back to the drawing board  

PS: ..and get to understand DllStructs, that in comparison, is like "why should I learn to drive a car, I already know how to walk there", but I now have to walk an ocean and if I knew how to drive a car, it would be easy to drive a boat. So yes, in due time, due to my "I don't have time to get into that now", I'll take the time to learn and build experience working with DllStructs. And thank you for your push towards it   

Edited March 31, 2018 by argumentum
expand

[argumentum]

ok, 1st draft. Working on making it flawless but has its problems  
I get a lot of @extended 183 when on "rapid fire" mouse clicking. 

<snip>

...work in progress  

P.S.: Writing the other way around, Sender as Receiver, as it make more sense  

Edited March 31, 2018 by argumentum
writing the other way around

[argumentum]

<snip>

Edited April 1, 2018 by argumentum
better code and comments

[argumentum]

The above code is, unless any of you find a flaw, perfect.   

Now my question is:  can it be a GUIRegisterMsg($something , 'FileMappingRecvRead')  implemented ?

That'd be beautiful      

[argumentum]

anyway, I figure that, that, was nonsense too. So I turned it into a UDF at 

 

[RTFC]

Excellent, well done! Glad it worked out (and fast too).