Mouse click and sends in the background

[Gal]

I've been searching these past few days for anything that would help me automate in the background, I've made a few things work but for a couple decade old programs I can't get the control IDs, which is resulting in difficulties.

 

I've tried using the _MouseClickPlus method which calls the user32 dll sendmessage with the mouse opcodes, this worked like a charm for a new adobe apollo runtime environment I was checking.

In that case, it didn't have any control IDs or anything else, and somehow all it needed to work was the title to obtain the winhandle from.

However this doesn't work for many windows apps, not sure why.

 

All of the following problems are under windows 7

1) In paint, if I use

WinGetHandle("Untitled - Paint")

it returns zeroes, anyone have any idea why? I've gotten around this by using the title "[CLASS:MSPaintApp]", however I'm still interested in this response.

2) Next I was looking to make it click somewhere, this:

ControlClick("[CLASS:MSPaintApp]", "", "[CLASS:Afx:00000000FF0C0000:8; INSTANCE:1]", "left", 1, 275, 152)

worked, although a better solution for me would be supplying only the title / handle and coords from the window start and not from the control start, and having it click on any control.

Is there any way to do this?

3) In Paint, if I click / hold for a second the alt key, I can see the keyboard shortcuts, for example, alt + H would trigger the home tab. to what control do I use ControlSend to so that "!hp" would call "Alt h" and then select the pencil with "p", I couldn't make this work when the main whiteboard is the controlID and neither when the top toolbar is the control ID, like so:

ControlSend("[CLASS:MSPaintApp]", "", "[CLASS:NetUIHWND; INSTANCE:1]", "!hp")

4) Continuing the previous topic, in one of my older programs, alt + F for example doesn't trigger the File at the top, so I would need to send a ControlClick to it, however it doesn't have any control info, you can look at the SciTE editor, going over the file/edit/search.. with the window info tool, there's nothing about the control.

How can I send a ControlClick to the Search button in this case? Simply supplying the window is not enough, and I have a feeling ControlClick never works without specifying a control ID.

A solution to my second question would be a solution to the fourth as well :/

 

Any tips will be greatly appreciated,

Thanks!

g3trans

 

 

[Jfish]

For #1:  I think you may not have the window activated.  Try this to get the handle and see if it works for you ...

run("mspaint.exe")
sleep(1000)
WinActivate("Untitled - Paint")
$paintHandle=WinGetHandle("Untitled - Paint")
ConsoleWrite(@CRLF&$paintHandle&@CRLF)

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

Edited August 7, 2015 by Jfish

[Jos]

Why would the Window need to be active to get its handle ?

This is working fine for me so what do you get?:

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : WinGetHandle("Untitled - Paint") = ' & WinGetHandle("Untitled - Paint") & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

Jos

[Jfish]

I was getting 0x00000 before winactive - and a result after it was activated.  In hindsight it was the sleep(1000) that made it work. 

[Gal]

You're right, not sure why that didn't work for me the other day. 

Any idea on my other questions? 

 

Basically I would ideally be able to send a ControlClick anywhere in the window, but for now, the simplified problem would be in the SciTE editor, sending a click to open the "Search" options and then clicking "Find" for example

This would be like sending Ctrl + F if the window was on top and activated, but in the background I'm trying to go about it with ControlClick.

 

Any thoughts? 

Gal

 

 

 

[Gal]

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

I'm checking out the IUI now, as for sending send instead of controlsend, that's not an option for me as I need to perform the whole automation in the background.

I'll update with my impression of the IUI

[Gal]

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

This is pretty cool, the simplyspy, but it doesn't work in the background, to requires me to send one of their methods _UIA_Action with setfocus before it can perform a click on the object I recorded.

Maybe there's a way to use the object I found with the simplespy and send it to the controlclick?

 

Like using the below methods to find an object by title and class and control type, and then forwarding that to control click?

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=C:\Users\King\Desktop\simplyspy\exploratoryAutoit.au3 - SciTE-Lite [4 of 4];controltype:=UIA_WindowControlTypeId;class:=SciTEWindow", $treescope_children)
 ;_UIA_Action($oP1,"setfocus")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Application;controltype:=UIA_MenuBarControlTypeId;class:=", $treescope_children)
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Search.mainwindow", "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Search;controltype:=UIA_MenuControlTypeId;class:=#32768", $treescope_children)
 ;_UIA_Action($oP0,"setfocus")
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Replace...  Ctrl+H.mainwindow", "title:=Replace...  Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Replace... Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")

 

[junkew]

Why run it in the background? Probably easier to create a virtualmachine to handle the application in the foreground and if needed from a local autoit you send commands with tcpip or other comm protocols.

[Gal]

The programs I want to be running in the background need to talk to com ports and actual hardware, 

I can imagine a setup where the physical computer translates the com ports input / output into tcp/ip commands to talk to a virtual machine and then that virtual machine translates it back into comm port emulation but that is too complex for me to even grasp how I would go about getting that to happen

 

Right now there's a machine dedicated to running the autoit, which is less than idea because if I were to want to run several of these at the same time I'd need a machine for each run, and more external hardware which is pricier as well.

 

 

I'm sure that it's possible to send these clicks / sends in the background, I just can't find the solution.

 

Thanks for the tips though 

 

[junkew]

What are the programs you are trying to click on. Sounds easie to talk to com ports directly from autoit sending the needed commands directly to your hardware. 

[Karthi]

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

Edited February 4, 2016 by Karthi

[takebacker]

6 hours ago, Karthi said:

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

 

I haven't found a way to do something like this. If you're automating, you eventually are forced to use blockinput because if you use the mouse or keyboard you're going to directly interfere with whatever you're trying to do. You need mouse in X,Y position? Move your mouse while this is happening and it won't be where you need it. Automating while doing work yourself isn't something that autoit can do.

Maybe a virtual machine would work, but i have no experience working with it. I don't think it would work.

[JLogan3o13]

@Karthi traditionally you would use ControlSend or ControlClick to interact with a window, even when it is no active. I believe the suggestion for IUI was because the OP was unable to do this. This is another reason why it is better to begin your own thread rather than tagging a question onto the end of someone else's; your needs may be different from those of the OP.

I would suggest you start with a detailed explanation on what application you're trying to automate, what the window looks like, what you find when you use the AutoIt Window Info Tool (located in the same directory where you installed AutoIt) etc. The more info you give us the better we can offer assistance.