OpenCV v4 UDF

[smbape]

Hi again @Lion66

The theory was correct.
Using cv::convexHull, it is possible to determine if a polygone is concave.

Local $hull = _VectorOfPointCreate()
Local $o_arr_hull = _cveOutputArrayFromVectorOfPoint($hull)
_cveConvexHull($i_arr_dst_int32, $o_arr_hull)
For $i = 0 To _VectorOfPointGetSize($dst_int32) - 1
    _VectorOfPointGetItemPtr($dst_int32, $i, $tPointFPtr)
    Local $dst_int32_i = DllStructCreate($tagCvPoint, $tPointFPtr.value)

    _VectorOfPointGetItemPtr($hull, $i, $tPointFPtr)
    Local $hull_i = DllStructCreate($tagCvPoint, $tPointFPtr.value)

    If $hull_i.x <> $dst_int32_i.x Or $hull_i.y <> $dst_int32_i.y Then
        ConsoleWrite("Concave contour" & @CRLF)
        ExitLoop
    EndIf
Next
_cveOutputArrayRelease($o_arr_hull)
_VectorOfPointRelease($hull)

See the attached code.

160732-opencv-udf.au3

Edited August 14, 2021 by smbape

[Lion66]

I do not cease to be surprised at your fantastic knowledge! Of course it works!

It will take me a few days to look and try to understand, I am completely confused in the code options. I'll have questions later.

Thank you.

[water]

I have added your UDF to the wiki

[smbape]

Thank you.

[Lion66]

Hello @ smbape

Congratulations! 👏 👏 👏 Now you are in a wiki (as I said!). It's worthy of the job you're doing!

I found this example. He draws the frame so beautifully.

I see that here the method of finding the contour is applied here and after that boundingRect.
I wanted to try, but I can't handle it myself.
Perhaps you will have the opportunity to show how to do this.

Attached is an example on the python.

Thank you so much.

 

card.py

 

 

 

 

 

Edited August 23, 2021 by Lion66

[smbape]

Hi @Lion66

Sorry for the late response, I was on another project.
I read the python code.

The detection is done in the following steps:

• Detect objects of a scene using cv2.findContours.
  To improve the countour detection, the image is first blured with cv2.GaussianBlur, thus removing noises,
  then thresholded with cv2.threshold to enhance seperation of areas.
• For each countour, pass it through a bank of images (images/cards/sample/)
• The type of image that matches the most will be choosed as the type of the countour.

This technique will work if the background and the object are very contrasted.
If not, threshold will either blacked everything or whited everything, thus making countour detection impossible.

On 8/18/2021 at 9:07 PM, Lion66 said:

I wanted to try, but I can't handle it myself.

What do you  want to try?

• Countour detection?
• GaussianBlur?
• Threshold?
• knnMatch ?
• A combination of those?

What have you tried that was not working?

Do you have a python/autoit example of your try?

[Lion66]

A very important part when using your UDF is to correctly declare the type of variables and structures before applying the function.
I have little understanding of the structure of the DLL, and I lack instructions for translating the syntax from docs.opencv.org to this UDF.
My plan is:

_cveGaussianBlurMat($gray_obj, $Gaus_obj, $ksize, 0, 0, $CV_BORDER_DEFAULT)
_cveThresholdMat($gray_obj, $gray_obj, $thresh, $maxval, $type)
_cveFindContoursMat($matImage, $matContours, $matHierarchy, $mode, $method, $offset = _cvPoint())
_cveBoundingRectangleMat($matPoints, $boundingRect)

And already the first function fails.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
Opt("MustDeclareVars", 1)
#include "emgucv-autoit-bindings\cve_extra.au3"
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.3.4721\libs\x64\cvextern.dll")

Local $img_obj = _cveImreadAndCheck("pic1\box.png", $CV_IMREAD_COLOR)   ; query image (small object)
Local $gray_obj = _cveMatCreate()
_cveCvtColorMat($img_obj, $gray_obj, $CV_COLOR_BGR2GRAY)

;_cveGaussianBlurMat($matSrc, $matDst, $ksize, $sigmaX, $sigmaY = 0, $borderType = $CV_BORDER_DEFAULT)
Local $Gaus_obj = _cveMatCreate()
Local $ksize = "(3,3)"
_cveGaussianBlurMat($gray_obj, $Gaus_obj, $ksize, 3, 0, $CV_BORDER_DEFAULT)

;_cveThresholdMat($gray_obj, $gray_obj, $thresh, $maxval, $type)
;_cveFindContoursMat($matImage, $matContours, $matHierarchy, $mode, $method, $offset = _cvPoint())
;_cveBoundingRectangleMat($matPoints, $boundingRect)

_cveImshowMat("", $Gaus_obj)
_cveWaitKey()
_cveDestroyAllWindows()
_Opencv_DLLClose()

And another question:
Can I process the error?
In the compiled script, if possible to show the message to the user instead of emergency completion?

Edited August 24, 2021 by Lion66

[smbape]

Hi @Lion66

3 hours ago, Lion66 said:

A very important part when using your UDF is to correctly declare the type of variables and structures before applying the function.

You are right. I am aware of the high level of knwoledge needed in order to use the UDF correctly.
I planned on making a documentation for rules of translation from c++ to the UDF, however is it hard teach it correctly.

This is what I do to convert to the UDF:

• Look at the udf definition without "Mat" at the end. For cv2.gaussianBlur it will be _cveGaussianBlur
• The first line of the function is the c++ signature of the function, which is

  CVAPI(void) cveGaussianBlur(cv::_InputArray* src, cv::_OutputArray* dst, CvSize* ksize, double sigmaX, double sigmaY, int borderType);

   

• The "Mat" version is a function where every cv::_InputArray, _OutputArray, _InputOutArray are derived from a matrix
• CvSize* is a _cvSize() element. In this case Local $ksize = _cvSize(3,3)
• When in the c++ exemple one of the cv::_*Array is a cv::Scalar, the "Mat" version cannot be used.
  You have to first create a cv::Scalar with _cveScalarCreate, then use _cve*ArrayFromScalar to convert it to the expected type before using it in the function
• There are cases where the function has extra parameters required. Those parameters take their default value if not specified.
  If the UDF is not aware of those default values, you have to put them manually.
  In that case, you have to look at the documentation at opencv.
  For exemple _cveKAZEDetectorCreate(False, False, 0.001, 4, 4, $CV_KAZE_DIFF_PM_G2, $tFeature2DPtr, $tSharedPtr) comes from cv::KAZE::create

With the previous instructions, can you try again?

Not that I do not want to help, but I want to see if my instructions are understandable and correct.

Edited August 24, 2021 by smbape

[smbape]

3 hours ago, Lion66 said:

Can I process the error?

When calling a dll, AutoIt doesn't provide a way for it. It just crashes.
The best things you can do are:

• Tools > "Trace: Add Trace Lines"
• Set $_cve_debug = 1
  This will print all successfully executed functions.
  With the last executed function, you can manage to find where it crashes.
  Crashes are usually due to incorrect types or Null pointers.

Edited August 24, 2021 by smbape

[Lion66]

Thank you for your answers.

The topic is very extensive. Sorry, if I to be importunate.

I advance with "$ksize = _cvSize(3,3)", but I did not understand why it was told about:

2 hours ago, smbape said:

You have to first create a cv::Scalar with _cveScalarCreate, then use _cve*ArrayFromScalar to convert it to the expected type before usin it in the function

Now I'm stuck on the "$matHierarchy" (third parameter in _cveFindContoursMat)

Local $img_scene = _cveImreadAndCheck("pic1\cards.JPG", $CV_IMREAD_COLOR)   ; query image - scene
Local $gray_scene = _cveMatCreate()
_cveCvtColorMat($img_scene, $gray_scene, $CV_COLOR_BGR2GRAY)

Local $Gaus_scene = _cveMatCreate()
_cveGaussianBlurMat($gray_scene, $Gaus_scene, _cvSize(3,3), 0, 0)
_cveThresholdMat($Gaus_scene, $Gaus_scene, 200, 255, $CV_THRESH_BINARY)


Local $H = _cveMatCreate()
Local $o_arr_H = _cveOutputArrayFromMat($H)
_cveFindContoursMat($Gaus_scene, $Gaus_scene, $o_arr_H, $CV_RETR_EXTERNAL, $CV_CHAIN_APPROX_SIMPLE)

 

Edited August 24, 2021 by Lion66

[smbape]

1 hour ago, Lion66 said:

The topic is very extensive. Sorry, if I to be importunate.

I don't feel that way. It is a pleasure to have someone interested in my project.
That is why I am answering every question you have.

 

1 hour ago, Lion66 said:

but I did not understand why it was told about

It doesn't apply to this case.
You missed the first line, "When in the c++ exemple one of the cv::_*Array is a cv::Scalar, the "Mat" version cannot be used.".
For example, in Threshold_inRange.cpp#L89 , the method inRange takes a Scalar as a parameter. I was talking about those kind of cases.
The are also case where there are vectors like in SURF_FLANN_matching_homography_Demo.cpp#L73 . Again, _cve*ArrayFrom* should be used like in matching_homography_Demo.au3#L301

 

1 hour ago, Lion66 said:

Now I'm stuck on the "$matHierarchy":

It is more than that.
I you look at the documentation of cv::findContours, it is written

Quote

contours	Detected contours. Each contour is stored as a vector of points (e.g. std::vector<std::vector<cv::Point> >).
hierarchy	Optional output vector (e.g. std::vector<cv::Vec4i>), containing information about the image topology. It has as many elements as the

For contours, it is not a matrix, but a vector of points (std::vector<std::vector<cv::Point> >), that means, the "Mat" version cannot be used.
For hierarchy, it is an output vector (std::vector<cv::Vec4i>). Vec4i is a matrix. Therefore, hierarchy is a cv::_OutputArray from a vector of matrix.
It will then be

$contoursVect = _VectorOfVectorOfPointCreate()
$contours = _cveOutputArrayFromVectorOfVectorOfPoint($contourVect)

$hierarchyVect = _VectorOfMatCreate()
$hierarchy = _cveOutputArrayFromVectorOfMat($hierarchyVect)

; rest of the code here

; Then loop through the VectorOfVectorOfPoint like in
; https://github.com/smbape/node-emgucv-autoit-generator/blob/master/samples/tutorial_code/features2D/feature_homography/matching_homography_Demo.au3#L232

It is true that it needs c++ understanding.
If my directions are not enough to help you have a working script, come back to me

Edited August 24, 2021 by smbape

[Lion66]

I do not have knowledge in C++.

For academic purposes, your UDF is very good.

For practical purposes, I would transfer the creation of all auxiliary variables (necessary for the function itself) into UDF.

And the user would be given simple functions with user arguments.

But this is another story!

I have to admit I don't understand your explanation at this level. I'm sorry.

I myself would like to understand how to get a ready-made code.

But this is not my level and we are unlikely to advance like that.

It doesn't work again.

Local $contoursVect = _VectorOfVectorOfPointCreate()
Local $contours = _cveOutputArrayFromVectorOfVectorOfPoint($contoursVect)

Local $hierarchyVect = _VectorOfMatCreate()
Local $hierarchy = _cveOutputArrayFromVectorOfMat($hierarchyVect)

_cveFindContoursMat($Gaus_scene, $contours, $hierarchy, $CV_RETR_EXTERNAL, $CV_CHAIN_APPROX_SIMPLE)

 

[smbape]

1 hour ago, Lion66 said:

For practical purposes, I would transfer the creation of all auxiliary variables (necessary for the function itself) into UDF.

And the user would be given simple functions with user arguments.

I agree with you.
However, OpenCV has more that 3000 functions.
It is impossible, without spending months, to translate everything.
And supposed I am done. Tracking updates from the point I started translation will be tedious.
Like every big c/c++ library out there, people just give up on using them with AutoIt, or write dlls, which requires c/c++ knowledge.

 

1 hour ago, Lion66 said:

But this is not my level and we are unlikely to advance like that.

It doesn't work again.

I understand your frustration. I will stop here.
Can you post your current file and image.
I will fix it for you.

Edited August 24, 2021 by smbape

[Lion66]

You are very kind!

SearchContureDraw-v1.au3

[smbape]

@Lion66 you almost had it.

hierarchy should be an empty array. I didn't understand the word "optional" in the documentation
_cveFindContours should have been used instead of _cveFindContoursMat.

I also add a loop through the contours.

SearchContureDraw-v1.au3

Edited August 24, 2021 by smbape

[Lion66]

I'm stuck in the next step: _cveBoundingRectangleMat.
In docs.opencv.org I see one parameter:

"Rect cv::boundingRect

(

InputArray

array

)

"

In this UDF two parameters:

"_cveBoundingRectangle($points, $boundingRect)"

Here's my failed attempt.

 

SearchContureDraw-v2.au3

Edited August 25, 2021 by Lion66

[smbape]

Like in cards.py, boundingRect has to be called on each countour.
The documentation of boundingRect only has one parameter and returns a cv::Rect.
Usually the UDF has no return value and the last parameter will receive the returned value.

Changes have been made in SearchContureDraw-v2.au3

Following cards.py, to crop an image based on a cv::Rect do:

$boundingRect.x -= 15
$boundingRect.y -= 15
$boundingRect.width += 15
$boundingRect.height += 15
$cropped = _cveMatCreateFromRect($img_scene, $boundingRect)

SearchContureDraw-v2.au3

Edited August 25, 2021 by smbape

[Lion66]

Hi @smbape

First thanks for your help and for your patience!
In the script, I want to recognize only one image, and changing the size of the frame is not important.
I don't know how to properly link the result after Filter matches and the frame.
I came up with an option, but for some reason the loop on line 71 makes only one turn.

 

SearchContureDraw-v5.au3

[smbape]

You used the same variable in two different loops

; #71
For $i = 0 To _VectorOfVectorOfPointGetSize($contoursVect) - 1

; #134
For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1

; change it to
For $j = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1
        _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $j, $tVectorDMatchPtr)

 

Edited August 26, 2021 by smbape

[smbape]

You should also add the following condition after _cveBoundingRectangle

If $boundingRect.width < 2 Or $boundingRect.height < 2 Then
        ContinueLoop
    EndIf

_CvFeature2DDetectAndComputeMat does not work on images of size 1