Simple automation using OpenCVs match feature

[BBs19]

I needed a function to automate programs at work that can't be fully automated via Autoits built in functions.

For example a virtual machine running on your physical machine, meaning you would need to run an extra script within the virtual machine (if it is even running Windows) in order to automate everything.

I came across OpenCV which allows matching/finding a picture in another picture. This would also allow searching for a button/text on the screen in order to press the exact position. Fortunately @mylise already translated all the required OpenCV functions to Autoit, I just had to remove all unnecessary functions to make the script as small as possible.

The problem:

Using this method, you will never be able to fully automate everything dynamically, as it will only work on the machine with same resolution/dpi settings, same theme etc.. This is only a last resort for programs that can't be automated using the built in Autoit functions.

Features:

• Find a given picture on the entire screen (all monitors) or a certain area on the screen and execute mouse clicks on this position.
• Adjust the threshold so that the picture doesn't have to match 100%.
• Debugging options like logging and marking the screen where the picture was found etc.
• Includes a Snapshot-Tool that creates snapshots of a certain area(buttons, text etc.) on the screen and generates the code that is required for the matching. It can also be used to get the coordinates to a marked area on the screen in order to check only on a certain area for the match picture.

Example:

Note: The example will probably not work on your computer, depending on the display resolution and dpi settings, as the picture has to match the exact same size on the screen. Please use the included Snapshot-Tool to generate new match pictures and code very easily.

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "OpenCV-Match_UDF.au3"

_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output. 

;Please note that these examples might not work as the match pictures have to be found with the exact same size on your screen.

;Example 1
ShellExecute("http://www.tv.com/");Open Website tv.com
$Match1 = _MatchPicture(@ScriptDir&"\Match\1.png", 0.70,False,10,500);Try to find the match picture on the screen. Number of tries: 10, Sleep between each try: 500ms. 
If Not @error Then
    _MarkMatch($Match1) ;Debugging: Draws a rect on the screen/coordinates of the match to show the user where the match was found
    Sleep(100)
    _ClickMouse($Match1, "left",1) ;Calculates the center of the match and clicks the left mouse once on click position
EndIf

Sleep(1000)

;Example 2, matching on a specific area of the screen
ShellExecute("notepad.exe");open nodepad
WinWait("[CLASS:Notepad]","",5)
WinMove("[CLASS:Notepad]","",0,0,500,500)

Local $sCoords[4] = [0, 0, 500,500]
$Match2 = _MatchPicture(@ScriptDir&"\Match\2.png", 0.80,$sCoords,3,500)
If Not @error Then
    _MarkMatch($Match2) 
    Sleep(100)
    _ClickMouse($Match2, "left", 1) 
EndIf

_OpenCV_Shutdown();Closes DLLs

So basically, all you need to do is provide a path to the match picture and the function will return you the coordinates (x1,y1,x2,y2) of where the picture has been found on the screen. With these, you can either calculate an exact position for the mouse click or use the "_ClickMouse" function which will execute a mouse click on the center of the coordinates where the picture was found. 

Credits:

@mylise for the OpenCV UDF

 

Download:

Includes the required .DLL files of OpenCV. You can also manually download them on the website of OpenCV (Version 3.x doesn't include these anymore, you need to download 2.x).

OpenCV_Match.zipFetching info...

Edited February 8, 2017 by BBs19

[Remie9]

Hi,

Its a very good implementation of the OpenCV image search and works like a charm. However how can the new OpenCV 4.1.2 be used with this?

Regards,

Jeremie

[FredMP]

@mylise

@BBs19

I have installed OpenCV and am trying to automate a Java x64 app on windows 10 64-bit. My script is not working.

I am simply trying to get the cursor to move to screen coordinates and click. for now.

  Quote

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "test.au3"

_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output.

HotKeySet( "{SPACE}", "MyFunction")
HotKeySet( "{ESC}", "CloseScript")

Local $x = 342 ; x is first value of mouse position
Local $y = 74 ; y is second value of mouse position

While 1
   sleep(10)
Wend

Func MyFunction()
MouseMove($x, $y, 2)                ; moves the mouse pointer do mouse position with speed 2
Sleep(500)                        ; wait 500 ms half of second ( delay )
MouseClick("left", $x, $y, 1, 2)    ; now click left mouse key on mouse position, 1 click with speed 2
EndFunc

Func CloseScript()
   Exit
EndFunc

Expand  

Error:include depth exceeded

[mLipok]

  On 9/3/2020 at 2:16 AM, FredMP said:

@mylise

@BBs19

I have installed OpenCV and am trying to automate a Java x64 app on windows 10 64-bit. My script is not working.

I am simply trying to get the cursor to move to screen coordinates and click. for now.

Error:include depth exceeded

Expand  

did you try:

?

[FredMP]

Could you elaborate please? What exactly do I need to try from that post and what will that accomplish. I'm new to this. Thanks!

@mLipok

Edited September 6, 2020 by FredMP

[Gianni]

  On 9/3/2020 at 2:16 AM, FredMP said:

I am simply trying to get the cursor to move to screen coordinates and click. for now.

Error:include depth exceeded

Expand  

you are not calling any of the OpenCV-Match_UDF functions in your script
what does the test.au3 file you included at the beginning of your script contain?
... I think if you remove it your snippet will work ....

[FredMP]

@Chimp

Finally got around to fixing this and your suggestion worked, and my 1st script finally works, yesss!

Now, I have two questions:

1.  I need this script to be on standby the whole time the TTS app is active. So, if I press the hotkey (SPACE) the script will run moving the mouse and clicking at the coordinates and then end. But the next time I press SPACE it should do it again. My question would be is it ok to leave the script running on standby like this for hours on end or is there a more efficient way for the script to run? 

2. I need the (SPACE) hotkey to only work in TTS app. So if i go over to a browser window and type SPACE I do not want the script to activate there, only in TTS app. How do I write this into the script?

Here is the current script, I welcome any suggestions!

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "OpenCV-Match_UDF.au3"
_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output.

HotKeySet( "{SPACE}", "MyFunction")
HotKeySet( "{ESC}", "CloseScript")

Local $x = 342 ; x is first value of mouse position
Local $y = 74 ; y is second value of mouse position

While 1
   sleep(10)
Wend

Func MyFunction()
   #RequireAdmin
if winexists("Main@TTS") Then
    winactivate("Main@TTS")
    EndIf
MouseMove($x, $y, 2)                ; moves the mouse pointer do mouse position with speed 2
Sleep(500)                        ; wait 500 ms half of second ( delay )
MouseClick("left", $x, $y, 2, 2)    ; now click left mouse key on mouse position, 2 click with speed 2
EndFunc

Func CloseScript()
   Exit
EndFunc

 

[dmob]

Oops, wrong context, sorry

Edited November 30, 2020 by dmob

[Gianni]

@FredMP

I'm not very clear about your goal, however,
using the hotkey in that way will always activate the function every time you press it, even if the focus is on another window (the browser).
You can limit hotkey activation only when the active window is the "Main@TTS" window using something like:

Func MyHotKeyFunc()
    Local $HotKeyPressed = @HotKeyPressed
    HotKeySet($HotKeyPressed) ; Disable hotkey to prevent looping

    If WinActive("Main@TTS") Then ; hotkey works only if this window is active
        ; do whatever you need
    Else
        Send($HotKeyPressed) ; send key to destination
    EndIf

    HotKeySet($HotKeyPressed, "MyHotKeyFunc") ; enable hotkey again
EndFunc   ;==>MyHotKeyFunc

but in your hot function i see you force focus on "Main@TTS". therefore I assume you want the hotkey to work always, even when "Main@TTS" has no focus (?).
Or, if you want it not to work only when the browser (or another window) is active then you could use some condition like this:

Func MyHotKeyFunc()
    Local $HotKeyPressed = @HotKeyPressed
    HotKeySet($HotKeyPressed) ; Disable hotkey to prevent looping

    If Not WinActive('Browser_hwd') Then ; <-- note the Not operator to exclude the browser
        ; do whatever you need
    Else
        Send($HotKeyPressed) ; send key to destination (the browser)
    EndIf
    HotKeySet($HotKeyPressed, "MyHotKeyFunc")     ; enable hotkey again
EndFunc   ;==>MyHotKeyFunc

 

[junkew]

https://docs.opencv.org/3.4/d1/dc5/tutorial_background_subtraction.html

Should help with the latest version of opencv

 

[skijve]

Hi all,

Would it be possible that this function needs a GPU to works fine ?

It seems that on old PC or on VM without GPU, we encounter 'Failed to load DLL' error.

I'm trying to figure out by checking the open CV website but maybe one of you already knows the answer 🙂

Thx in advance 😉

[skijve]

I'm a bit more far in my investigation.  On 2 of my 3 PC, it works fine.

On the 3rd one, the OPENDLL always return me "-1" as result.  I was wondering if it was not link to any Visual studio install (got that from other forum) but, asexcpected, this does not solve the problem.

I checked the file location, everything is okay (even if I change the dll and put it in the same locaton as the script) but DllOpen("opencv_core2413.dll") always return me -1.

Any idea ?

[skijve]

I downloaded the latest VC for the version (https://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.4.13/) and now it's working. Whouhou