Search the Community

I needed a function to automate programs at work that can't be fully automated via Autoits built in functions. For example a virtual machine running on your physical machine, meaning you would need to run an extra script within the virtual machine (if it is even running Windows) in order to automate everything. I came across OpenCV which allows matching/finding a picture in another picture. This would also allow searching for a button/text on the screen in order to press the exact position. Fortunately @mylise already translated all the required OpenCV functions to Autoit, I just had to remove all unnecessary functions to make the script as small as possible. The problem: Using this method, you will never be able to fully automate everything dynamically, as it will only work on the machine with same resolution/dpi settings, same theme etc.. This is only a last resort for programs that can't be automated using the built in Autoit functions. Features: Find a given picture on the entire screen (all monitors) or a certain area on the screen and execute mouse clicks on this position. Adjust the threshold so that the picture doesn't have to match 100%. Debugging options like logging and marking the screen where the picture was found etc. Includes a Snapshot-Tool that creates snapshots of a certain area(buttons, text etc.) on the screen and generates the code that is required for the matching. It can also be used to get the coordinates to a marked area on the screen in order to check only on a certain area for the match picture. Example: Note: The example will probably not work on your computer, depending on the display resolution and dpi settings, as the picture has to match the exact same size on the screen. Please use the included Snapshot-Tool to generate new match pictures and code very easily. #AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work #include "OpenCV-Match_UDF.au3" _OpenCV_Startup();loads opencv DLLs _OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output. ;Please note that these examples might not work as the match pictures have to be found with the exact same size on your screen. ;Example 1 ShellExecute("http://www.tv.com/");Open Website tv.com $Match1 = _MatchPicture(@ScriptDir&"\Match\1.png", 0.70,False,10,500);Try to find the match picture on the screen. Number of tries: 10, Sleep between each try: 500ms. If Not @error Then _MarkMatch($Match1) ;Debugging: Draws a rect on the screen/coordinates of the match to show the user where the match was found Sleep(100) _ClickMouse($Match1, "left",1) ;Calculates the center of the match and clicks the left mouse once on click position EndIf Sleep(1000) ;Example 2, matching on a specific area of the screen ShellExecute("notepad.exe");open nodepad WinWait("[CLASS:Notepad]","",5) WinMove("[CLASS:Notepad]","",0,0,500,500) Local $sCoords[4] = [0, 0, 500,500] $Match2 = _MatchPicture(@ScriptDir&"\Match\2.png", 0.80,$sCoords,3,500) If Not @error Then _MarkMatch($Match2) Sleep(100) _ClickMouse($Match2, "left", 1) EndIf _OpenCV_Shutdown();Closes DLLs So basically, all you need to do is provide a path to the match picture and the function will return you the coordinates (x1,y1,x2,y2) of where the picture has been found on the screen. With these, you can either calculate an exact position for the mouse click or use the "_ClickMouse" function which will execute a mouse click on the center of the coordinates where the picture was found. Credits: @mylise for the OpenCV UDF Download: Includes the required .DLL files of OpenCV. You can also manually download them on the website of OpenCV (Version 3.x doesn't include these anymore, you need to download 2.x). OpenCV_Match.zip