Lion66 Posted July 13, 2021 Posted July 13, 2021 @smbape I really appreciate your help! I removed the _cvNormalize and the _cvThreshold, and now $tmaxval.max shows the correct results. In a printout I see the maximum value at scale 2, which corresponds to reality: the picture is increased by 2 times. Scale: 1.5 0.263792783021927 Scale: 1.75 0.21775709092617 Scale: 2 0.272976666688919 Scale: 2.25 0.223066240549088 Scale: 2.5 0.226864993572235 But rectangles are not displayed correctly 🙄. Something is wrong with the parameters of the $tmaxloc.x and $tmaxloc.y ? I can not understand. Could you see? Thank you. expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <GDIplus.au3> #include <Memory.au3> #include <GUIConstantsEx.au3> #include <Array.au3> #include <emgucv-autoit-bindings\cve_extra.au3> ;start dll opencv _GDIPlus_Startup() _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $matEmpty = _cveMatCreate() #Region Template ;Load template Local $ptemp = _cveImreadAndCheck("Template.png") ;Opencv function to load image ;// Create a destination image to hold output ;//smooth and converting the original image into grayscale Local $TemplateGray = _cveMatCreate() ;_cveSmoothMat($ptemp, $ptemp, $CV_GAUSSIAN, 3, 3) ; _cveCvtColorMat($ptemp, $TemplateGray, $CV_BGR2GRAY) ; _cveCannyMat($TemplateGray, $TemplateGray, 50, 200) ; kontyre #EndRegion Template #Region Image ;Load image from file Local $pimg = _cveImreadAndCheck("bridge200p.jpg") ;Opencv function to load image ;// Create a destination image to hold output ;//smooth and converting the original image into grayscale Local $pimgGrayScale = _cveMatCreate() ;_cveSmoothMat($pimg, $pimg, $CV_GAUSSIAN, 3, 3) ; _cveCvtColorMat($pimg, $pimgGrayScale, $CV_BGR2GRAY) ; _cveCannyMat($pimgGrayScale, $pimgGrayScale, 50, 200) ; kontyre ;_cvEqualizeHistMat( $pimgGrayScale, $pimgGrayScale ) ; osvetlit' #EndRegion Image ;// Create some CV and Autoit windows to show the input ;_cveNamedWindow("Example") ;// show input image ;_cveImshowMat("Example", $ptemp) ; Local $width2 = _cveMatGetWidth($TemplateGray) Local $height2 = _cveMatGetHeight($TemplateGray) Local $pimgResized, $scale, $width, $height, $rw, $rh, $presult Local $CurMaxVal = 0, $CurLocX = 0, $CurLocY = 0 Local $tBlackColor = _cvScalar(0) Local $tRedColor = _cvRGB(255, 0, 0) Local $tWhiteColor = _cvRGB(255, 255, 255) Local $tDsize = _cvSize(0, 0) For $i = 1.5 To 2.5 Step 0.25 ;//Resize image $scale = $i $pimgResized = _cveMatCreate() $tDsize.width = _cveMatGetWidth($pimgGrayScale) / $scale $tDsize.height = _cveMatGetHeight($pimgGrayScale) / $scale _cveResizeMat($pimgGrayScale, $pimgResized, $tDsize, 0, 0, $CV_INTER_LINEAR) _cveCannyMat($pimgResized, $pimgResized, 50, 200) ; kontyre $width = _cveMatGetWidth($pimgResized) $height = _cveMatGetHeight($pimgResized) If ($width < $width2) Or ($height < $height2) Then ExitLoop EndIf $rw = $width - $width2 + 1 $rh = $height - $height2 + 1 ;// Create Opencv matrix object 32 bit floating $presult = _cveMatCreate() _cveMatCreateData($presult, $rh, $rw, $CV_32FC1) ;// Template matching _cveMatchTemplateMat($pimgResized, $TemplateGray, $presult, 5, $matEmpty) ;// Create minmax variables to pass to opencv Local $tmaxloc = DllStructCreate($tagCvPoint) Local $tminloc = DllStructCreate($tagCvPoint) Local $tmaxval = DllStructCreate("double max;") Local $tminval = DllStructCreate("double min;") ;// create mask to find multiple matches Local $pmask = _cveMatCreate() _cveMatOnes($rh, $rw, $CV_8UC1, $pmask) ;// Find location of maximum _cveMinMaxLocMat($presult, $tminval, $tmaxval, $tminloc, $tmaxloc, $pmask) ;// Mask it to find others if exists and draw red rectangle on input image Local $tMaskRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 5, $height2 + 5) _cveRectangleMat($pmask, $tMaskRect, $tBlackColor, -1, $CV_LINE_8, 0) $tMaskRect = 0 Local $tImgRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 15, $height2 + 15) ;Local $tImgRect = _cvRect($tmaxloc.x, $tmaxloc.y, $width2, $height2) _cveRectangleMat($pimgResized, $tImgRect, $tWhiteColor, -1, $CV_LINE_8, 0) $tImgRect = 0 ;// Update input image _cveImshowMat("", $pimgResized) Sleep(100) If $CurMaxVal < $tmaxval.max Then $CurMaxVal = $tmaxval.max $CurScale = $scale $CurLocX = $tmaxloc.x * $scale $CurLocY = $tmaxloc.y * $scale MsgBox(0, "", "Changed. Scale: " & $scale) EndIf ConsoleWrite("Scale: " & $scale & " " & $tmaxval.max & @CRLF) Sleep(1000) Next ;// Mask it to find others if exists and draw red rectangle on input image ;_cveRectangle($pmask, _cvPoint($CurLocX - 5, $CurLocY - 5), _cvPoint($CurLocX + $width2, $CurLocY + $height2), $tBlackColor, -1, 8, 0) Local $tImgRect = _cvRect($CurLocX - 5, $CurLocY - 5, $width2 + 15, $height2 + 15) _cveRectangleMat($pimg, $tImgRect, $tRedColor, 1, 8, 0) $tImgRect = 0 ;// Update input image ;_cveImshowMat("Example-Resize", $pimgResized) _cveImshowMat("", $pimg) _cveWaitKey() $tDsize = 0 $tWhiteColor = 0 $tRedColor = 0 $tBlackColor = 0 _cveMatRelease($ptemp) _cveMatRelease($TemplateGray) _cveMatRelease($pimg) ; _cveMatRelease($pimgResized) ; _cveMatRelease($pimgGrayScale) ; _cveMatRelease($matEmpty) ; _cveDestroyAllWindows() _Opencv_DLLClose() _GDIPlus_Shutdown() Exit
smbape Posted July 13, 2021 Posted July 13, 2021 (edited) @Lion66 you are welcome. Great catch on the removal of the _cveNormalize. With $CV_TM_CCOEFF_NORMED (5) as matchMethod, an optimum value ($tmaxval.max in this case) which is less than 0.75 can be considered as a false match. Your match is wrong for at least 2 reasons: Canny on the image + Resize + Canny rarely gives expected results The match rectangle has incoherent values. Why - 5 ? Why +15? Here is a modified version that gives expected result. expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <GDIplus.au3> #include <Memory.au3> #include <GUIConstantsEx.au3> #include <Array.au3> #include <emgucv-autoit-bindings\cve_extra.au3> ;start dll opencv _GDIPlus_Startup() _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $matEmpty = _cveMatCreate() #Region Template ;Load template Local $ptemp = _cveImreadAndCheck("Template.png") ;Opencv function to load image ;// Create a destination image to hold output ;//smooth and converting the original image into grayscale Local $TemplateGray = _cveMatCreate() ;_cveSmoothMat($ptemp, $ptemp, $CV_GAUSSIAN, 3, 3) ; _cveCvtColorMat($ptemp, $TemplateGray, $CV_BGR2GRAY) ; _cveCannyMat($TemplateGray, $TemplateGray, 50, 200) ; kontyre #EndRegion Template #Region Image ;Load image from file Local $pimg = _cveImreadAndCheck("bridge200p.jpg") ;Opencv function to load image ;// Create a destination image to hold output ;//smooth and converting the original image into grayscale Local $pimgGrayScale = _cveMatCreate() ;_cveSmoothMat($pimg, $pimg, $CV_GAUSSIAN, 3, 3) ; _cveCvtColorMat($pimg, $pimgGrayScale, $CV_BGR2GRAY) ; ; Canny + Resize + Canny will give wrong matches ; _cveCannyMat($pimgGrayScale, $pimgGrayScale, 50, 200) ; kontyre ;_cvEqualizeHistMat( $pimgGrayScale, $pimgGrayScale ) ; osvetlit' #EndRegion Image ; show input image ; _cveImshowMat("Image", $pimgGrayScale) ; show template image ; _cveImshowMat("Template", $TemplateGray) Local $width2 = _cveMatGetWidth($TemplateGray) Local $height2 = _cveMatGetHeight($TemplateGray) Local $pimgResized, $scale, $width, $height, $rw, $rh, $presult Local $CurMaxVal = 0, $CurLocX = 0, $CurLocY = 0, $CurScale = 1 Local $tBlackColor = _cvScalar(0) Local $tRedColor = _cvRGB(255, 0, 0) Local $tWhiteColor = _cvRGB(255, 255, 255) Local $tDsize = _cvSize(0, 0) For $i = 1.5 To 2.5 Step 0.25 ;//Resize image $scale = $i $tDsize.width = _cveMatGetWidth($pimgGrayScale) / $scale $tDsize.height = _cveMatGetHeight($pimgGrayScale) / $scale If ($tDsize.width < $width2) Or ($tDsize.height < $height2) Then ExitLoop EndIf $pimgResized = _cveMatCreate() _cveResizeMat($pimgGrayScale, $pimgResized, $tDsize, 0, 0, $CV_INTER_LINEAR) _cveCannyMat($pimgResized, $pimgResized, 50, 200) ; Canny after resize will give better results $width = _cveMatGetWidth($pimgResized) $height = _cveMatGetHeight($pimgResized) $rw = $width - $width2 + 1 $rh = $height - $height2 + 1 ;// Create Opencv matrix object 32 bit floating $presult = _cveMatCreate() _cveMatCreateData($presult, $rh, $rw, $CV_32FC1) ;// Template matching _cveMatchTemplateMat($pimgResized, $TemplateGray, $presult, $CV_TM_CCOEFF_NORMED, $matEmpty) ;// Create minmax variables to pass to opencv Local $tmaxloc = DllStructCreate($tagCvPoint) Local $tminloc = DllStructCreate($tagCvPoint) Local $tmaxval = DllStructCreate("double max;") Local $tminval = DllStructCreate("double min;") ;// create mask to ;// Find location of maximum _cveMinMaxLocMat($presult, $tminval, $tmaxval, $tminloc, $tmaxloc, $matEmpty) ; Local $tImgRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 15, $height2 + 15) Local $tImgRect = _cvRect($tmaxloc.x, $tmaxloc.y, $width2, $height2) _cveRectangleMat($pimgResized, $tImgRect, $tWhiteColor, 1, $CV_LINE_8, 0) $tImgRect = 0 ;// Update input image _cveImshowMat("", $pimgResized) ConsoleWrite("Scale: " & $scale & " " & $tmaxval.max & @CRLF) If $CurMaxVal < $tmaxval.max Then $CurMaxVal = $tmaxval.max $CurScale = $scale $CurLocX = $tmaxloc.x * $scale $CurLocY = $tmaxloc.y * $scale MsgBox(0, "", "Changed. Scale: " & $scale) EndIf Sleep(1000) _cveMatRelease($presult) ; _cveMatRelease($pimgResized) ; Next ;// Mask it to find others if exists and draw red rectangle on input image ;_cveRectangle($pmask, _cvPoint($CurLocX - 5, $CurLocY - 5), _cvPoint($CurLocX + $width2, $CurLocY + $height2), $tBlackColor, -1, 8, 0) Local $tImgRect = _cvRect($CurLocX, $CurLocY, $width2 * $CurScale, $height2 * $CurScale) _cveRectangleMat($pimg, $tImgRect, $tRedColor, 1, 8, 0) $tImgRect = 0 ;// Update input image ;_cveImshowMat("Example-Resize", $pimgResized) _cveImshowMat("", $pimg) _cveWaitKey() $tDsize = 0 $tWhiteColor = 0 $tRedColor = 0 $tBlackColor = 0 _cveMatRelease($ptemp) _cveMatRelease($TemplateGray) _cveMatRelease($pimg) ; _cveMatRelease($pimgGrayScale) ; _cveMatRelease($matEmpty) ; _cveDestroyAllWindows() _Opencv_DLLClose() _GDIPlus_Shutdown() Exit Edited July 13, 2021 by smbape Lion66 1
Lion66 Posted July 13, 2021 Posted July 13, 2021 (edited) Perfect! Thank you! It does exactly what I need! I see you made a lot of changes. Later I study it. Now technical question: What files must be left for work from two packages (dll and au3)? Now they are too much. I would like to limit the minimum files. Why - 5 ? Why +15? This is a few pixels on the width of the frame lines so that the frame is outside the sample found. Not essential. Edited July 13, 2021 by Lion66
smbape Posted July 13, 2021 Posted July 13, 2021 (edited) I can't tell. Those files where generated by a script. The dll is needed for sure. cve_extra.au3 has some convinient functions that may not be needed for you. It also includes cve_world.au3 which includes all the generated files. To know the files that really need to be included, remove #include <emgucv-autoit-bindings\cve_extra.au3> and repeat the next process Run the script Find the file which define the missing functions Include it Repeat until there are not more missing functions You can also create your own .au3 which contains only the needed functions Edited July 13, 2021 by smbape
Lion66 Posted July 14, 2021 Posted July 14, 2021 (edited) @ smbape Вопрос был более относится к DLL , и это хорошо , что один достаточно. Для моих примеров был 21 файл AU3 😃 Вы проделали большую интересную работу и полезный UDF. Думаю, вам нужно сделать новую тему и рассказать на примерах своего проекта. Так ваш проект увидит больше людей. Может попросить администратора перенести наши сообщения в новую тему. И из этой темы сделать ссылку. На мой взгляд, ваш проект заслуживает того, чтобы его добавили в Wiki UDF . Удачи в будущем! Edited July 15, 2021 by Lion66
smbape Posted July 14, 2021 Posted July 14, 2021 @Lion66 Thanks for the encouragement. I will make a new topic when I am ready.
Lion66 Posted July 15, 2021 Posted July 15, 2021 @ smbape A new question: Is there in you project a function to search on the screen? As in the file OpenCV-Match_UDF.au3 in this post : ; Name ..........: _MatchPicture ; Description ...: Searches for a picture on screen or on a specific area of the screen and returns the coordinates where the picture has been found. Thank you.
smbape Posted July 15, 2021 Posted July 15, 2021 (edited) There is no function to search for an image. However there is _cveMatchTemplate that can be used for this purpose. For screen search, you can load the screen as a matrix, and from there use any opencv functions. Here is how to load a screen capture as a matrix expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <WindowsConstants.au3> #include <WinAPI.au3> #include "emgucv-autoit-bindings\cve_extra.au3" Local $tDesktopScreenRect = _WinAPI_GetDesktopScreenRect() ConsoleWrite("$tDesktopScreenRect.left: " & $tDesktopScreenRect.left & @CRLF) ConsoleWrite("$tDesktopScreenRect.top: " & $tDesktopScreenRect.top & @CRLF) ConsoleWrite("$tDesktopScreenRect.right: " & $tDesktopScreenRect.right & @CRLF) ConsoleWrite("$tDesktopScreenRect.bottom: " & $tDesktopScreenRect.bottom & @CRLF) ConsoleWrite(@CRLF) Local $iLeft = $tDesktopScreenRect.left Local $iTop = $tDesktopScreenRect.top Local $biWidth = $tDesktopScreenRect.right - $tDesktopScreenRect.left Local $biHeight = $tDesktopScreenRect.bottom - $tDesktopScreenRect.top Local $iChannels = 4 Local $hWnd = _WinAPI_GetDesktopWindow() Local $hDesktopDC = _WinAPI_GetDC($hWnd) Local $hMemoryDC = _WinAPI_CreateCompatibleDC($hDesktopDC) ;create compatible memory DC Local $tBIHDR = DllStructCreate($tagBITMAPINFO) $tBIHDR.biSize = DllStructGetSize($tBIHDR) $tBIHDR.biWidth = $biWidth $tBIHDR.biHeight = -$biHeight $tBIHDR.biPlanes = 1 $tBIHDR.biBitCount = $iChannels * 8 Local $aDIB = DllCall("gdi32.dll", "ptr", "CreateDIBSection", "hwnd", 0, "struct*", $tBIHDR, "uint", 0, "ptr*", 0, "ptr", 0, "dword", 0) _WinAPI_SelectObject($hMemoryDC, $aDIB[0]) Local $tBits = DllStructCreate('byte[' & $biWidth * $biHeight * $iChannels & ']', $aDIB[4]) _WinAPI_BitBlt($hMemoryDC, 0, 0, $biWidth, $biHeight, $hDesktopDC, $iLeft, $iTop, $SRCCOPY) _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $matImg = _cveMatCreateWithData($biHeight, $biWidth, $CV_8UC4, $tBits, 0) _cveImshowMat("Screen capture", $matImg) _cveWaitKey(0) _WinAPI_DeleteObject($aDIB[0]) $tBIHDR = 0 _WinAPI_DeleteDC($hMemoryDC) _WinAPI_ReleaseDC($hWnd, $hDesktopDC) _cveMatRelease($matImg) _cveDestroyWindow("Screen capture") _Opencv_DLLClose() Func _WinAPI_GetDesktopScreenRect() Local $iRight, $iBottom, $aRetrun Local $tRect = DllStructCreate($tagRECT) $tRect.Left = 0 $tRect.Top = 0 $tRect.Right = -1 $tRect.Bottom = -1 Local Const $tagDISPLAY_DEVICE = "dword Size;wchar Name[32];wchar String[128];dword Flags;wchar ID[128];wchar Key[128]" Local $tDisplayDevice = DllStructCreate($tagDISPLAY_DEVICE) $tDisplayDevice.Size = DllStructGetSize($tDisplayDevice) Local $tDisplaySettings = DllStructCreate($tagDEVMODE_DISPLAY) $tDisplaySettings.Size = DllStructGetSize($tDisplaySettings) Local $iDevNum = 0 While 1 ; _WinAPI_EnumDisplayDevices("", $iDevNum) $aRetrun = DllCall("user32.dll", "int", "EnumDisplayDevicesW", "ptr", 0, "dword", $iDevNum, "struct*", $tDisplayDevice, "dword", 1) If Not $aRetrun[0] Then ExitLoop $iDevNum += 1 If BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MIRRORING_DRIVER) Or Not BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_ATTACHED_TO_DESKTOP) Then ContinueLoop EndIf If BitAND($_cve_debug, 1) Then ConsoleWrite($tDisplayDevice.Name & @TAB & "Attached to desktop: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_ATTACHED_TO_DESKTOP) <> 0) & @CRLF) ConsoleWrite($tDisplayDevice.Name & @TAB & "Primary: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_PRIMARY_DEVICE) <> 0) & @CRLF) ConsoleWrite($tDisplayDevice.Name & @TAB & "Mirroring driver: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MIRRORING_DRIVER) <> 0) & @CRLF) ConsoleWrite($tDisplayDevice.Name & @TAB & "VGA compatible: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_VGA_COMPATIBLE) <> 0) & @CRLF) ConsoleWrite($tDisplayDevice.Name & @TAB & "Removable: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_REMOVABLE) <> 0) & @CRLF) ConsoleWrite($tDisplayDevice.Name & @TAB & "More display modes: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MODESPRUNED) <> 0) & @CRLF) ConsoleWrite(@CRLF) Endif ; _WinAPI_EnumDisplaySettings($tDisplayDevice.Name, $ENUM_CURRENT_SETTINGS) Local $sDevice = $tDisplayDevice.Name Local $sTypeOfDevice = 'wstr' If Not StringStripWS($sDevice, $STR_STRIPLEADING + $STR_STRIPTRAILING) Then $sTypeOfDevice = 'ptr' $sDevice = 0 EndIf $aRetrun = DllCall("user32.dll", "bool", "EnumDisplaySettingsW", $sTypeOfDevice, $sDevice, "dword", $ENUM_CURRENT_SETTINGS, "struct*", $tDisplaySettings) If Not $aRetrun[0] Then ContinueLoop If $tRect.Left > $tDisplaySettings.X Then $tRect.Left = $tDisplaySettings.X If $tRect.Top > $tDisplaySettings.Y Then $tRect.Left = $tDisplaySettings.Y $iRight = $tDisplaySettings.X + $tDisplaySettings.PelsWidth If $tRect.Right < $iRight Then $tRect.Right = $iRight $iBottom = $tDisplaySettings.Y + $tDisplaySettings.PelsHeight If $tRect.Bottom < $iBottom Then $tRect.Bottom = $iBottom WEnd $tDisplaySettings = 0 $tDisplayDevice = 0 Return $tRect EndFunc ;==>_WinAPI_GetDesktopScreenRect Edited July 15, 2021 by smbape Lion66 1
Lion66 Posted July 16, 2021 Posted July 16, 2021 Thank you. With your example, I was able to make a script to find icon by template on the desktop and click by mouse. I am sure that you offered the best option. But I am curious, can I use the _ScreenCapture_Capture function for these purposes ? In such style: _cveImreadAndCheck(_ScreenCapture_Capture("", 0, 0, -1, -1, false))
smbape Posted July 16, 2021 Posted July 16, 2021 There may be a way to use screen capture, but I don't know how. _ScreenCapture_Capture("", 0, 0, -1, -1, false) returns a bitmap as in Local $hBMP = _WinAPI_CreateCompatibleBitmap($hDDC, $iW, $iH) If you cat get from the bitmap something equivalent to what you get from Local $tBits = DllStructCreate('byte[' & $biWidth * $biHeight * $iChannels & ']', $aDIB[4]) _WinAPI_BitBlt($hMemoryDC, 0, 0, $biWidth, $biHeight, $hDesktopDC, $iLeft, $iTop, $SRCCOPY) then you can use it. A less efficient alternative will be: save the screen capture into a file, the load it with _cveImread
Lion66 Posted July 17, 2021 Posted July 17, 2021 On 11.07.2021 at 22:40, smbape said: Keep in mind that cv::matchTemplate is reliable only if you know that the template is present in the bigger image. If not, it has to be combined with something that tells you how good the match is, for example cv::compareHist Look at _cveFindTemplate in cve_extra.au3 for a "work in progress" way to reliably find a template in a bigger image. Now I've come to this, and I don't know where and how to apply it. Please show me how it's done.
smbape Posted July 19, 2021 Posted July 19, 2021 I was busy finding a satisfiying way to do that. I ended up hidding all the complicated code in _cveFindTemplate With the new udf version below there is an example of how you can find matches of a template. For your own images, adjust parameters, usually threshold is enough, until you get a satisfying result. expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include "emgucv-autoit-bindings\cve_extra.au3" _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $tDesktopScreenRect = _WinAPI_GetDesktopScreenRect() ConsoleWrite("$tDesktopScreenRect.left: " & $tDesktopScreenRect.left & @CRLF) ConsoleWrite("$tDesktopScreenRect.top: " & $tDesktopScreenRect.top & @CRLF) ConsoleWrite("$tDesktopScreenRect.right: " & $tDesktopScreenRect.right & @CRLF) ConsoleWrite("$tDesktopScreenRect.bottom: " & $tDesktopScreenRect.bottom & @CRLF) ConsoleWrite(@CRLF) Local $tBits = _cveGetDesktopScreenBits($tDesktopScreenRect) Local $matImg = _cveMatCreateWithData($tDesktopScreenRect.bottom - $tDesktopScreenRect.top, $tDesktopScreenRect.right - $tDesktopScreenRect.left, $CV_8UC4, $tBits, 0) Local $matTempl = _cveImreadAndCheck("mario_coin.png") Local $w = _cveMatGetWidth($matTempl) Local $h = _cveMatGetHeight($matTempl) ; we compute the histogram from the B G R channels ;Local $aChannels[3] = [0, 1, 2] Local $aChannels = _cveDefaultBGRChannels() ; Quantize each color to 32 levels ; The higher the value, the closer to the template the matched color is ; Local $aHistSize[3] = [32, 32, 32] Local $aHistSize = _cveDefaultBGRHistSize() ; Each color varies from 0 to 255 ; Local $aRanges[6] = [0, 256, 0, 256, 0, 256] Local $aRanges = _cveDefaultBGRRanges() ; The higher the value, the higher the match is exact Local $fThreshold = 0.8 ; Ignore the alpha channel when matching the template => $CV_COLOR_BGRA2BGR ; Only consider gray scale colors when matching the template => $CV_COLOR_BGRA2GRAY ; using cveCanny and/or gray scale can significantly reduce the matching time but also reduce the matching accuracy Local $hTimer = TimerInit() Local $aMatches = _cveFindTemplate($matImg, $matTempl, $fThreshold, $CV_COLOR_BGRA2GRAY) Local $fDiff = TimerDiff($hTimer) ConsoleWrite("Finding matches took " & $fDiff & "ms" & @CRLF) Local $iMatches = UBound($aMatches) ConsoleWrite("Number of matches: " & $iMatches & @CRLF) Local $tRedColor = _cvRGB(255, 0, 0) Local $tMatchRect = _cvRect(0, 0, $w, $h) For $i = 0 To $iMatches - 1 $tMatchRect.x = $aMatches[$i][0] $tMatchRect.y = $aMatches[$i][1] ; Draw a red rectangle around the matched position _cveRectangleMat($matImg, $tMatchRect, $tRedColor, 1, $CV_LINE_8, 0) Next _cveImshowMat("Source Image", $matImg) _cveWaitKey(0) $tMatchRect = 0 $tRedColor = 0 _cveMatRelease($matTempl) _cveMatRelease($matImg) $tBits = 0 _cveDestroyAllWindows() _Opencv_DLLClose() emgucv-autoit-bindings.zip Lion66 1
Lion66 Posted July 20, 2021 Posted July 20, 2021 Fun picture. And the example pleased. Again, a different approach. I can't keep up with your creativity! Thank you.
Lion66 Posted July 27, 2021 Posted July 27, 2021 Hi @smbape Is it possible with you UDF make a detect the key points using SURF detector, or others: AKAZE, SIFT? As in this example. I found your respective functions, but I myself can not create the logic. Can you show an example? Thanks.
smbape Posted July 27, 2021 Posted July 27, 2021 Hi @Lion66 At the present time I can't give you an example. SURF is a patented algorithm. Try to replace SURF with ORB and FLANNBASED (1) with BRUTEFORCE_HAMMING (5) There should be an UDF with ORB, detectAndCompute, knnMatch. If not, I will try when I have more time
Lion66 Posted July 28, 2021 Posted July 28, 2021 Your sentence doesn't mention AKAZE, KAZE, SIFT. Does that mean they're patented, too?
smbape Posted July 29, 2021 Posted July 29, 2021 (edited) Hi @Lion66 I managed to make a functionnal example. It was hard. I had to look into the function signatures to know the expected parameters. I had to modify the UDF. AKAZE, KAZE, SIFT algorithms are free Here is an example with ORB working with the new UDF version expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** Opt("MustDeclareVars", 1) #include <ButtonConstants.au3> #include <ComboConstants.au3> #include <EditConstants.au3> #include <File.au3> #include <FileConstants.au3> #include <GDIPlus.au3> #include <GuiComboBox.au3> #include <GUIConstantsEx.au3> #include <Math.au3> #include <StaticConstants.au3> #include <WindowsConstants.au3> #include "emgucv-autoit-bindings\cve_extra.au3" ;~ Sources: ;~ https://docs.opencv.org/4.5.2/d7/dff/tutorial_feature_homography.html ;~ https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/feature_homography/SURF_FLANN_matching_homography_Demo.cpp Local Const $OPENCV_SAMPLES_DATA_PATH = @ScriptDir #Region ### START Koda GUI section ### Form= Local $FormGUI = GUICreate("Features2D + Homography to find a known object", 1000, 707, 192, 95) Local $InputObject = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box.png", 230, 16, 449, 21) Local $BtnObject = GUICtrlCreateButton("Object", 689, 14, 75, 25) Local $InputScene = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box_in_scene.png", 230, 52, 449, 21) Local $BtnScene = GUICtrlCreateButton("Scene", 689, 50, 75, 25) Local $LabelMatchType = GUICtrlCreateLabel("Match type", 414, 92, 79, 20) GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif") Local $ComboMatchType = GUICtrlCreateCombo("", 502, 92, 177, 25, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE)) GUICtrlSetData(-1, "BruteForce|BruteForce-L1|BruteForce-Hamming|BruteForce-HammingLUT|BruteForce-Hamming(2)|BruteForce-SL2") Local $BtnExec = GUICtrlCreateButton("Execute", 832, 48, 75, 25) Local $LabelMatches = GUICtrlCreateLabel("Good Matches & Object detection", 377, 144, 245, 20) GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif") Local $GroupMatches = GUICtrlCreateGroup("", 20, 166, 958, 532) Local $PicMatches = GUICtrlCreatePic("", 25, 177, 948, 516) GUICtrlCreateGroup("", -99, -99, 1, 1) GUISetState(@SW_SHOW) #EndRegion ### END Koda GUI section ### Local $aMatchTypes[6] = [ _ $CV_NORM_L2, _ $CV_NORM_L1, _ $CV_NORM_HAMMING, _ $CV_NORM_HAMMING, _ $CV_NORM_HAMMING2, _ $CV_NORM_L2SQR _ ] _GUICtrlComboBox_SetCurSel($ComboMatchType, 2) _GDIPlus_Startup() _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $img_object, $img_scene Local $nMsg Local $sObject, $sScene Local $tBackgroundColor = _cvRGB(0xF0, 0xF0, 0xF0) Main() While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $BtnObject $sObject = ControlGetText($FormGUI, "", $InputObject) $sObject = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sObject) If @error Then $sObject = "" Else ControlSetText($FormGUI, "", $InputObject, $sObject) EndIf Case $BtnScene $sScene = ControlGetText($FormGUI, "", $InputScene) $sScene = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sScene) If @error Then $sScene = "" Else ControlSetText($FormGUI, "", $InputScene, $sScene) EndIf Case $ComboMatchType Detect() Case $BtnExec Clean() Main() EndSwitch WEnd _Opencv_DLLClose() _GDIPlus_Shutdown() Func Main() ;;! [load_image] ;;/ Load object and scene $sObject = ControlGetText($FormGUI, "", $InputObject) $img_object = _cveImreadAndCheck($sObject, $CV_IMREAD_GRAYSCALE) If @error Then $sObject = "" Return EndIf $sScene = ControlGetText($FormGUI, "", $InputScene) $img_scene = _cveImreadAndCheck($sScene, $CV_IMREAD_GRAYSCALE) If @error Then _cveMatRelease($img_object) $sObject = "" $sScene = "" Return EndIf ;;! [load_image] Detect() EndFunc ;==>Main Func Clean() If $sObject == "" Then Return _cveMatRelease($img_object) _cveMatRelease($img_scene) $sObject = "" EndFunc ;==>Clean Func Detect() Local $match_type = $aMatchTypes[_GUICtrlComboBox_GetCurSel($ComboMatchType)] ;;-- Step 1: Detect the keypoints using ORB Detector, compute the descriptors Local $numberOfFeatures = 500 ; Local $tFeature2DPtr = DllStructCreate("ptr value") Local $tSharedPtr = DllStructCreate("ptr") _cveOrbCreate($numberOfFeatures, 1.2, 8, 31, 0, 2, $CV_ORB_HARRIS_SCORE, 31, 20, $tFeature2DPtr, $tSharedPtr) Local $detector = $tFeature2DPtr.value Local $keypoints_object = _VectorOfKeyPointCreate() Local $keypoints_scene = _VectorOfKeyPointCreate() Local $descriptors_object = _cveMatCreate() Local $descriptors_scene = _cveMatCreate() _CvFeature2DDetectAndComputeMat($detector, $img_object, _cveNoArrayMat(), $keypoints_object, $descriptors_object, False) ; _CvFeature2DDetectAndComputeMat($detector, $img_scene, _cveNoArrayMat(), $keypoints_scene, $descriptors_scene, False) ; ;;-- Step 2: Matching descriptor vectors with a BruteForce based matcher ;; Since ORB is a floating-point descriptor NORM_L2 is used Local $tMatcherPtr = DllStructCreate("ptr value") Local $bf_matcher = _cveBFMatcherCreate($match_type, False, $tMatcherPtr) ; Local $matcher = $tMatcherPtr.value Local $knn_matches = _VectorOfVectorOfDMatchCreate() ; _cveDescriptorMatcherKnnMatch1Mat($matcher, $descriptors_object, $descriptors_scene, $knn_matches, 2, _cveNoArrayMat(), False) ; ;;-- Filter matches using the Lowe's ratio test Local $ratio_thresh = 0.75 ; Local $good_matches = _VectorOfDMatchCreate() ; Local $tVectorDMatchPtr = DllStructCreate("ptr value") Local $tDMatchPtr0 = DllStructCreate("ptr value") Local $tDMatchPtr1 = DllStructCreate("ptr value") For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1 _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $i, $tVectorDMatchPtr) _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 0, $tDMatchPtr0) Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value) _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 1, $tDMatchPtr1) Local $tDMatch1 = DllStructCreate($tagDMatch, $tDMatchPtr1.value) If $tDMatch0.distance < $ratio_thresh * $tDMatch1.distance Then _VectorOfDMatchPush($good_matches, $tDMatch0) EndIf Next ;;-- Draw matches Local $img_matches = _cveMatCreate() ; Local $matchesMask = _VectorOfByteCreate() _drawMatchedFeatures1Mat($img_object, $keypoints_object, $img_scene, $keypoints_scene, $good_matches, $img_matches, _cvScalarAll(-1), _ _cvScalarAll(-1), $matchesMask, $CV_DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS) ; ;;-- Localize the object Local $obj = _VectorOfPointFCreate() ; Local $scene = _VectorOfPointFCreate() ; Local $tObjectKeyPointPtr = DllStructCreate("ptr value") Local $tSceneKeyPointPtr = DllStructCreate("ptr value") For $i = 0 To _VectorOfDMatchGetSize($good_matches) - 1 ;;-- Get the keypoints from the good matches _VectorOfDMatchGetItemPtr($good_matches, $i, $tDMatchPtr0) Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value) _VectorOfKeyPointGetItemPtr($keypoints_object, $tDMatch0.queryIdx, $tObjectKeyPointPtr) Local $tObjectKeyPoint = DllStructCreate($tagKeyPoint, $tObjectKeyPointPtr.value) _VectorOfPointFPush($obj, $tObjectKeyPoint) _VectorOfKeyPointGetItemPtr($keypoints_scene, $tDMatch0.trainIdx, $tSceneKeyPointPtr) Local $tSceneKeyPoint = DllStructCreate($tagKeyPoint, $tSceneKeyPointPtr.value) _VectorOfPointFPush($scene, $tSceneKeyPoint) Next Local $H = _cveMatCreate() Local $i_arr_H = _cveInputArrayFromMat($H) Local $o_arr_H = _cveOutputArrayFromMat($H) Local $i_arr_obj = _cveInputArrayFromVectorOfPointF($obj) Local $i_arr_scene = _cveInputArrayFromVectorOfPointF($scene) Local $resultMask = _cveMatCreate() Local $o_arr_resultMask = _cveOutputArrayFromMat($resultMask) _cveFindHomography($i_arr_obj, $i_arr_scene, $o_arr_H, $CV_RANSAC, 3, $o_arr_resultMask) ; _cveOutputArrayRelease($o_arr_resultMask) _cveMatRelease($resultMask) If Not _cveMatIsEmpty($H) Then ;;-- Get the corners from the image_1 ( the object to be "detected" ) Local $img_object_size = _cvSize() _cveMatGetSize($img_object, $img_object_size) Local $obj_corners = _VectorOfPointFCreate() _VectorOfPointFPush($obj_corners, _cvPoint2f(0, 0)) _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, 0)) _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, $img_object_size.height)) _VectorOfPointFPush($obj_corners, _cvPoint2f(0, $img_object_size.height)) Local $scene_corners = _VectorOfPointFCreateSize(4) Local $i_arr_obj_corners = _cveInputArrayFromVectorOfPointF($obj_corners) Local $o_arr_scene_corners = _cveOutputArrayFromVectorOfPointF($scene_corners) _cvePerspectiveTransform($i_arr_obj_corners, $o_arr_scene_corners, $i_arr_H) ; Local $tPointFPtr = DllStructCreate("ptr value") _VectorOfPointFGetItemPtr($scene_corners, 0, $tPointFPtr) Local $scene_corners_0 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 1, $tPointFPtr) Local $scene_corners_1 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 2, $tPointFPtr) Local $scene_corners_2 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 3, $tPointFPtr) Local $scene_corners_3 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) ;;-- Draw lines between the corners (the mapped object in the scene - image_2 ) _cveLineMat($img_matches, _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _ _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _ _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _ _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _ _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _cvScalar(0, 255, 0), 4) ; _cveOutputArrayRelease($o_arr_scene_corners) _cveInputArrayRelease($i_arr_obj_corners) _VectorOfPointFRelease($scene_corners) _VectorOfPointFRelease($obj_corners) EndIf _cveInputArrayRelease($i_arr_scene) _cveInputArrayRelease($i_arr_obj) _cveOutputArrayRelease($o_arr_H) _cveInputArrayRelease($i_arr_H) _cveMatRelease($H) _VectorOfPointFRelease($scene) _VectorOfPointFRelease($obj) ;-- Show detected matches ; _cveImshowMat("Good Matches & Object detection", $img_matches) ; _cveImshowControlPic($img_matches, $FormGUI, $PicMatches, $tBackgroundColor) _VectorOfByteRelease($matchesMask) _cveMatRelease($img_matches) _VectorOfDMatchRelease($good_matches) _VectorOfVectorOfDMatchRelease($knn_matches) _cveBFMatcherRelease($bf_matcher) _cveMatRelease($descriptors_scene) _cveMatRelease($descriptors_object) _VectorOfKeyPointRelease($keypoints_scene) _VectorOfKeyPointRelease($keypoints_object) _cveOrbRelease($tSharedPtr) EndFunc ;==>Detect emgucv-autoit-bindings.zip Edited July 29, 2021 by smbape
Lion66 Posted July 30, 2021 Posted July 30, 2021 Hi smbape I went and found a short working code on Python, from which I wanted to start: import cv2 from matplotlib import pyplot as plt img1 = cv2.imread('box.png', cv2.IMREAD_GRAYSCALE) img2 = cv2.imread('box_in_scene.png', cv2.IMREAD_GRAYSCALE) orb = cv2.ORB_create() kp1, des1 = orb.detectAndCompute(img1, None) kp2, des2 = orb.detectAndCompute(img2, None) bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True) matches = bf.match(des1, des2) matches = sorted(matches, key=lambda x: x.distance) img3 = cv2.drawMatches(img1, kp1, img2, kp2, matches[:20], img2, flags=2) cv2.imshow("Matches", img3) cv2.waitKey() During this time, you wrote a huge piece of code! I can not imagine how it is possible! I'm shocked and thanks for your work! I would like to explain the result I want to get: In the picture of the scene, find the center of the rectangle of the found object to click on the mouse. Now about your code: I like the concept of your example, but some pairs of pictures does not accept, the program closes. I will attach such files, and when you have the opportunity to test them. Thanks.
smbape Posted July 30, 2021 Posted July 30, 2021 (edited) 2 hours ago, Lion66 said: In the picture of the scene, find the center of the rectangle of the found object to click on the mouse. Matching homography to find a known object can be tedious and I am not well informed in the subject. If the object always has the same size and orientation in the scene, I would have used matchTemplate like a previous example. If the object always has the same orientation but different size in a known scale range, I would have used multiscale matching like a previous example. Otherwise, I have no clue. 2 hours ago, Lion66 said: I like the concept of your example, but some pairs of pictures does not accept, the program closes. There is an error at _cveFindHomography. That is because there is a need for at least 4 point correspondences to calculate Homography. Cases where _VectorOfDMatchGetSize($good_matches) < 4 will crash. Here is an updated version handling that case expandcollapse popup#Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** Opt("MustDeclareVars", 1) #include <ButtonConstants.au3> #include <ComboConstants.au3> #include <EditConstants.au3> #include <File.au3> #include <FileConstants.au3> #include <GDIPlus.au3> #include <GuiComboBox.au3> #include <GUIConstantsEx.au3> #include <Math.au3> #include <StaticConstants.au3> #include <WindowsConstants.au3> #include "emgucv-autoit-bindings\cve_extra.au3" ;~ Sources: ;~ https://docs.opencv.org/4.5.2/d7/dff/tutorial_feature_homography.html ;~ https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/feature_homography/SURF_FLANN_matching_homography_Demo.cpp Local Const $OPENCV_SAMPLES_DATA_PATH = @ScriptDir #Region ### START Koda GUI section ### Form= Local $FormGUI = GUICreate("Features2D + Homography to find a known object", 1000, 707, 192, 95) Local $InputObject = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box.png", 230, 16, 449, 21) Local $BtnObject = GUICtrlCreateButton("Object", 689, 14, 75, 25) Local $InputScene = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box_in_scene.png", 230, 52, 449, 21) Local $BtnScene = GUICtrlCreateButton("Scene", 689, 50, 75, 25) Local $LabelMatchType = GUICtrlCreateLabel("Match type", 414, 92, 79, 20) GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif") Local $ComboMatchType = GUICtrlCreateCombo("", 502, 92, 177, 25, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE)) GUICtrlSetData(-1, "BruteForce|BruteForce-L1|BruteForce-Hamming|BruteForce-HammingLUT|BruteForce-Hamming(2)|BruteForce-SL2") Local $BtnExec = GUICtrlCreateButton("Execute", 832, 48, 75, 25) Local $LabelMatches = GUICtrlCreateLabel("Good Matches & Object detection", 377, 144, 245, 20) GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif") Local $GroupMatches = GUICtrlCreateGroup("", 20, 166, 958, 532) Local $PicMatches = GUICtrlCreatePic("", 25, 177, 948, 516) GUICtrlCreateGroup("", -99, -99, 1, 1) GUISetState(@SW_SHOW) #EndRegion ### END Koda GUI section ### Local $aMatchTypes[6] = [ _ $CV_NORM_L2, _ $CV_NORM_L1, _ $CV_NORM_HAMMING, _ $CV_NORM_HAMMING, _ $CV_NORM_HAMMING2, _ $CV_NORM_L2SQR _ ] _GUICtrlComboBox_SetCurSel($ComboMatchType, 2) _GDIPlus_Startup() _OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll") Local $img_object, $img_scene Local $nMsg Local $sObject, $sScene Local $tBackgroundColor = _cvRGB(0xF0, 0xF0, 0xF0) Main() While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $BtnObject $sObject = ControlGetText($FormGUI, "", $InputObject) $sObject = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sObject) If @error Then $sObject = "" Else ControlSetText($FormGUI, "", $InputObject, $sObject) EndIf Case $BtnScene $sScene = ControlGetText($FormGUI, "", $InputScene) $sScene = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sScene) If @error Then $sScene = "" Else ControlSetText($FormGUI, "", $InputScene, $sScene) EndIf Case $ComboMatchType Detect() Case $BtnExec Clean() Main() EndSwitch WEnd _Opencv_DLLClose() _GDIPlus_Shutdown() Func Main() ;;! [load_image] ;;/ Load object and scene $sObject = ControlGetText($FormGUI, "", $InputObject) $img_object = _cveImreadAndCheck($sObject, $CV_IMREAD_GRAYSCALE) If @error Then $sObject = "" Return EndIf $sScene = ControlGetText($FormGUI, "", $InputScene) $img_scene = _cveImreadAndCheck($sScene, $CV_IMREAD_GRAYSCALE) If @error Then _cveMatRelease($img_object) $sObject = "" $sScene = "" Return EndIf ;;! [load_image] Detect() EndFunc ;==>Main Func Clean() If $sObject == "" Then Return _cveMatRelease($img_object) _cveMatRelease($img_scene) $sObject = "" EndFunc ;==>Clean Func Detect() Local $match_type = $aMatchTypes[_GUICtrlComboBox_GetCurSel($ComboMatchType)] ;;-- Step 1: Detect the keypoints using ORB Detector, compute the descriptors Local $numberOfFeatures = 500 ; Local $tFeature2DPtr = DllStructCreate("ptr value") Local $tSharedPtr = DllStructCreate("ptr") _cveOrbCreate($numberOfFeatures, 1.2, 8, 31, 0, 2, $CV_ORB_HARRIS_SCORE, 31, 20, $tFeature2DPtr, $tSharedPtr) Local $detector = $tFeature2DPtr.value Local $keypoints_object = _VectorOfKeyPointCreate() Local $keypoints_scene = _VectorOfKeyPointCreate() Local $descriptors_object = _cveMatCreate() Local $descriptors_scene = _cveMatCreate() _CvFeature2DDetectAndComputeMat($detector, $img_object, _cveNoArrayMat(), $keypoints_object, $descriptors_object, False) ; _CvFeature2DDetectAndComputeMat($detector, $img_scene, _cveNoArrayMat(), $keypoints_scene, $descriptors_scene, False) ; ;;-- Step 2: Matching descriptor vectors with a BruteForce based matcher ;; Since ORB is a floating-point descriptor NORM_L2 is used Local $tMatcherPtr = DllStructCreate("ptr value") Local $bf_matcher = _cveBFMatcherCreate($match_type, False, $tMatcherPtr) ; Local $matcher = $tMatcherPtr.value Local $knn_matches = _VectorOfVectorOfDMatchCreate() ; _cveDescriptorMatcherKnnMatch1Mat($matcher, $descriptors_object, $descriptors_scene, $knn_matches, 2, _cveNoArrayMat(), False) ; ;;-- Filter matches using the Lowe's ratio test Local $ratio_thresh = 0.75 ; Local $good_matches = _VectorOfDMatchCreate() ; Local $tVectorDMatchPtr = DllStructCreate("ptr value") Local $tDMatchPtr0 = DllStructCreate("ptr value") Local $tDMatchPtr1 = DllStructCreate("ptr value") For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1 _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $i, $tVectorDMatchPtr) _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 0, $tDMatchPtr0) Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value) _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 1, $tDMatchPtr1) Local $tDMatch1 = DllStructCreate($tagDMatch, $tDMatchPtr1.value) If $tDMatch0.distance < $ratio_thresh * $tDMatch1.distance Then _VectorOfDMatchPush($good_matches, $tDMatch0) EndIf Next ;;-- Draw matches Local $img_matches = _cveMatCreate() ; Local $matchesMask = _VectorOfByteCreate() _drawMatchedFeatures1Mat($img_object, $keypoints_object, $img_scene, $keypoints_scene, $good_matches, $img_matches, _cvScalarAll(-1), _ _cvScalarAll(-1), $matchesMask, $CV_DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS) ; ;;-- Need at least 4 point correspondences to calculate Homography If _VectorOfDMatchGetSize($good_matches) >= 4 Then ;;-- Localize the object Local $obj = _VectorOfPointFCreate() ; Local $scene = _VectorOfPointFCreate() ; Local $tObjectKeyPointPtr = DllStructCreate("ptr value") Local $tSceneKeyPointPtr = DllStructCreate("ptr value") For $i = 0 To _VectorOfDMatchGetSize($good_matches) - 1 ;;-- Get the keypoints from the good matches _VectorOfDMatchGetItemPtr($good_matches, $i, $tDMatchPtr0) Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value) _VectorOfKeyPointGetItemPtr($keypoints_object, $tDMatch0.queryIdx, $tObjectKeyPointPtr) Local $tObjectKeyPoint = DllStructCreate($tagKeyPoint, $tObjectKeyPointPtr.value) _VectorOfPointFPush($obj, $tObjectKeyPoint) _VectorOfKeyPointGetItemPtr($keypoints_scene, $tDMatch0.trainIdx, $tSceneKeyPointPtr) Local $tSceneKeyPoint = DllStructCreate($tagKeyPoint, $tSceneKeyPointPtr.value) _VectorOfPointFPush($scene, $tSceneKeyPoint) Next Local $H = _cveMatCreate() Local $i_arr_H = _cveInputArrayFromMat($H) Local $o_arr_H = _cveOutputArrayFromMat($H) Local $i_arr_obj = _cveInputArrayFromVectorOfPointF($obj) Local $i_arr_scene = _cveInputArrayFromVectorOfPointF($scene) Local $resultMask = _cveMatCreate() Local $o_arr_resultMask = _cveOutputArrayFromMat($resultMask) _cveFindHomography($i_arr_obj, $i_arr_scene, $o_arr_H, $CV_RANSAC, 3, $o_arr_resultMask) ; _cveOutputArrayRelease($o_arr_resultMask) _cveMatRelease($resultMask) If Not _cveMatIsEmpty($H) Then ;;-- Get the corners from the image_1 ( the object to be "detected" ) Local $img_object_size = _cvSize() _cveMatGetSize($img_object, $img_object_size) Local $obj_corners = _VectorOfPointFCreate() _VectorOfPointFPush($obj_corners, _cvPoint2f(0, 0)) _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, 0)) _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, $img_object_size.height)) _VectorOfPointFPush($obj_corners, _cvPoint2f(0, $img_object_size.height)) Local $scene_corners = _VectorOfPointFCreateSize(4) Local $i_arr_obj_corners = _cveInputArrayFromVectorOfPointF($obj_corners) Local $o_arr_scene_corners = _cveOutputArrayFromVectorOfPointF($scene_corners) _cvePerspectiveTransform($i_arr_obj_corners, $o_arr_scene_corners, $i_arr_H) ; Local $tPointFPtr = DllStructCreate("ptr value") _VectorOfPointFGetItemPtr($scene_corners, 0, $tPointFPtr) Local $scene_corners_0 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 1, $tPointFPtr) Local $scene_corners_1 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 2, $tPointFPtr) Local $scene_corners_2 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) _VectorOfPointFGetItemPtr($scene_corners, 3, $tPointFPtr) Local $scene_corners_3 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value) ;;-- Draw lines between the corners (the mapped object in the scene - image_2 ) _cveLineMat($img_matches, _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _ _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _ _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _ _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _cvScalar(0, 255, 0), 4) ; _cveLineMat($img_matches, _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _ _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _cvScalar(0, 255, 0), 4) ; _cveOutputArrayRelease($o_arr_scene_corners) _cveInputArrayRelease($i_arr_obj_corners) _VectorOfPointFRelease($scene_corners) _VectorOfPointFRelease($obj_corners) EndIf _cveInputArrayRelease($i_arr_scene) _cveInputArrayRelease($i_arr_obj) _cveOutputArrayRelease($o_arr_H) _cveInputArrayRelease($i_arr_H) _cveMatRelease($H) _VectorOfPointFRelease($scene) _VectorOfPointFRelease($obj) EndIf ;-- Show detected matches ; _cveImshowMat("Good Matches & Object detection", $img_matches) ; _cveImshowControlPic($img_matches, $FormGUI, $PicMatches, $tBackgroundColor) _VectorOfByteRelease($matchesMask) _cveMatRelease($img_matches) _VectorOfDMatchRelease($good_matches) _VectorOfVectorOfDMatchRelease($knn_matches) _cveBFMatcherRelease($bf_matcher) _cveMatRelease($descriptors_scene) _cveMatRelease($descriptors_object) _VectorOfKeyPointRelease($keypoints_scene) _VectorOfKeyPointRelease($keypoints_object) _cveOrbRelease($tSharedPtr) EndFunc ;==>Detect Edited July 30, 2021 by smbape Lion66 1
Lion66 Posted July 31, 2021 Posted July 31, 2021 (edited) Thank you, very good turned out an example! And after I checked my pairs of images on examples in Python, I realized that there was not enough match. But if I understand correctly, there are two problems here: 1. Not enough coincidences. 2. No 4 points for drawing a frame. I compared the search for ORB points with search in FAST. ORB: import cv2 as cv imageread = cv.imread('2-border.png') ORB_object = cv.ORB_create() keypoints = ORB_object.detect(imageread) keypoints, descriptors = ORB_object.compute(imageread, keypoints) imageresult = cv.drawKeypoints(imageread, keypoints, None, color=(255,0,0), flags=0) cv.imshow('ORB_image', imageresult) cv.waitKey() FAST: import cv2 img = cv2.imread('2-border.png',0) fast = cv2.FastFeatureDetector_create(threshold=25) kp = fast.detect(img,None) img2 = cv2.drawKeypoints(img, kp, None,color=(255,0,0)) cv2.imshow('fast_true',img2),cv2.waitKey() https://docs.opencv.org/4.5.3/df/d0c/tutorial_py_fast.html#gsc.tab=0 And I see that Fast finds much more points!Is it possible to use this point search method with any comparison method, and will it work for turning images or with a resizing? Regarding the second problem: I would instead of a frame of four corners, would draw a circle with the center in the accumulation of correspondence points. If there are tools for this. I would be happy to see such a example if you have an interest and there is time. Thanks. P.S. In new version of UDF missing $CV_BGR2GRAY=6 in file cv_constants.au3. Old examples do not work. This is done on purpose? Edited July 31, 2021 by Lion66
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now