Jump to content

Finding images inside others.

jebus495

By jebus495
in AutoIt General Help and Support

 Share

Recommended Posts

jebus495 Prodigy

jebus495

Posted
  • Author
Posted (edited)

WOW!

Thank you soo much.

Here's what I've got so far. :)

;; Here's an example array of what we want to search for.
Dim $aSearchArray[8][5] = [[1,1,1,1,0],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,0],[1,0,0,0,0],[1,0,0,0,0]]


Local $NumofPixelsHigh = 80;you will need to use the window info tool to establish size of image you want to use.
Local $NumofPixelsWide = 125;Same as above
Local $Coord[2] = [225, 349] ;Top left corner of image
Local $Target[$NumofPixelsHigh][$NumofPixelsWide], $Current[$NumofPixelsHigh][$NumofPixelsWide]
Local $Y, $X, $B = 0

For $Y = 0 to $NumofPixelsHigh -1
For $X = 0 to $NumofPixelsWide -1
$Target[$Y][$X] = Pixelgetcolor($Coord[0]+$X, $Coord[1]+$Y)
If $Target[$Y][$X] = "0x000000" then
$Target[$Y][$X] = 1
Else
$Target[$Y][$X] = 0
EndIf;This gets the color value, then checks to see if it is black, if it is we assign a 1 value, if not we assign a 0 value.
Next
Next
;; In this case, there is only one instance of what we want to search for.
;; Its position is 4, 3

For $x = 0 to UBound($Target,1)-1
    For $y = 0 to UBound($Target,2)-1
        If ( $Target[$x][$y] == $aSearchArray[0][0] ) Then ;; Just finds a starting point for the correct match, then later checks if this is correct
            If ( _IsCorrect($x,$y) == 1 ) Then
                MsgBox(0x40,"Au3 Test", "Correct match found at position [" & $x & "," & $y & "].")
                Exit
            EndIf
        EndIf
    Next
Next

Func _IsCorrect($n,$m) ; This is helper function which goes through the global array to see if this is the correct match we are looking for. Returns 1 for succes.
    $k = $n+UBound($aSearchArray,1)-1
    $l = $m+UBound($aSearchArray,2)-1
    If ( $k > UBound($Target,1) ) Then Return 0 ; goes outside the boundaries, so we can be sure this isn't correct
    If ( $l > UBound($Target,2) ) Then Return 0
   
    For $x = $n to $k
        For $y = $m to $l
            If ( $Target[$x][$y] <> $aSearchArray[$x-$n][$y-$m] ) Then
                Return 0
            EndIf
        Next
    Next
   
    Return 1
EndFuncoÝ÷ Ø­×hzÇâØ­ò0®+^ÙbëaÆ®¶­sdvÆö&Âb33c¶6V&6'&³uÕ³uÒÒ²ð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÒÂð¥³ÃÃÃÃÃÃÕ

I hope I have made some silly novice mistake.

Edited by jebus495
jvanegmond Universalist

jvanegmond

Posted
Posted (edited)

I hope I have made some silly novice mistake.

You're creating a new 2 dimensional array, and filling one element of it with a 2 dimensional array. I don't know what you end up with if you do that, but I know it's not right.

You should be doing something more like this.. :

Global $aSearchArray = $_plussign

I'm going to have another look at all of the code later.

Edited by Manadar

github.com/jvanegmond

UTA Wayfarer

UTA

Posted
Posted

Ok, so now we've got an array for each character. We've trimmed this array down so that we can easily compare it to other arrays that hold. Let's compare the arrays. That's exactly what the Func _CompareArray($pPixel1, $pPixel2, $pAccuracy = 0.95) does. It compares arrays based on a percentage of accuracy. If the arrays match is greater than the required accuracy it returns a value that is true, otherwise it returns a value that is false.

While playing around with your CompareArray-Function (Thanks for that) I stumbled across one little mistake:

I would replace 

Local $iTotal = ($minX-1) * ($minY-1)

with 

Local $iTotal = ($minX) * ($minY)

instead.

Otherwise I got a difference of more than 100% if everything is different :) .

UTA

jvanegmond Universalist

jvanegmond

Posted
Posted

While playing around with your CompareArray-Function (Thanks for that) I stumbled across one little mistake:

I would replace 

Local $iTotal = ($minX-1) * ($minY-1)

with 

Local $iTotal = ($minX) * ($minY)

instead.

Otherwise I got a difference of more than 100% if everything is different :) .

UTA

Thank you for spotting the mistakes. Updated in code and in above post.

github.com/jvanegmond

  • 2 months later...
EchaniDrgn Seeker

EchaniDrgn

Posted
Posted

I'm curious why you don't turn the definition into say a 32 or less bit integer. I haven't read this entire thread to see, but this would lower your number of comparisons to one per array rather than one per element of the array. I have some home-brew OCR in a bot of my own that searches a given area for a "signature" of pixels of a certain color. Creates a definition on the fly and compares that definition against a flat database (2d array). As each signature is checked it only continues the comparison if it continues to get a match. This isn't bulletproof OCR but it does allow me to consistently read the characters I'm looking for.

Here's a code snippet from my bot. I get the definition and turn it into an array of 16 integers only using the lowest 16 bits.

For $index = 0 to 6
        $currentOffset[0] = $OCR_LETTER_OFFSET_ARRAYS[$game][$index][0] + $windowPosition[0];
        $currentOffset[1] = $OCR_LETTER_OFFSET_ARRAYS[$game][$index][1] + $windowPosition[1] + $BROWSER_OFFSET[$target];
        $currentOffset = PixelSearch($currentOffset[0], $currentOffset[1], $currentOffset[0] + 30, _
                                     $currentOffset[1] + 30, $OCR_TEXT_COLOR[$game]);
        If @error = 1 Then
            MsgBox(0, "OCR reading Error", "Letter " & String($index + 1) & " not present or" & @LF & _
                                           "broswer offset not correct, attempting to fix.")
            If $error = 0 Then
                ControlSend($TARGET_STRINGS[$target], "", $CONTROL_ID_STRINGS[$target], "{BACKSPACE 7}");
                Sleep(500); Wait for the letters to re-populate
                Return GetOcrLetters(1);
            ElseIf $error = 1 Then
                $BROWSER_OFFSET[$target] = InputBox("Possible Incorrect Browser Offset", _
                                                    "Enter a new Browser Offset" & @LF & _
                                                    "Just Title Bar (0)"  & @LF & _
                                                    "Title Bar and Address Bar (30)"  & @LF & _
                                                    "Other (?)", "0", "");
                Return GetOcrLetters(2);
            Else
                MsgBox(0, "OCR Failed", "Could not determine OCR Letters");
            EndIf
        EndIf
        For $i = 0 to 15
            $tempLetter[$i] = 0;
        Next
        For $j = 0 to 15
            For $i = 0 to 15
                $color = PixelGetColor($currentOffset[0] + $i, $currentOffset[1] + $j);
                If $color = $OCR_TEXT_COLOR[$game] Then
                    $tempLetter[$j] = BitOR($tempLetter[$j], 2 ^ $i);
                EndIf
            Next
        Next
        $i = 0;
        While $i < $letterDefinitionLength[$game] AND NOT $found
            $j = 0;
            While $j < 16 AND NOT $found
                If $tempLetter[$j] <> $letterDefinitionArray[$game][$i][$j] Then
                    ExitLoop
                ElseIf $j = 15 Then
                    $found = True
                EndIf
                $j += 1;
            WEnd
            $i += 1;
        WEnd
        If $found Then
            $tempString = $tempString & $letterDefinitionArray[$game][$i - 1][16]; 16th index is the letter itself
            $found = False;
        Else
            If GUICtrlRead($guiArray[$SOUND_CHECKBOX][$GUI_HANDLE]) = $GUI_CHECKED Then
                SoundPlay(@WindowsDir & "\media\ding.wav")
            EndIf
            $tempChar = InputBox("Identify Letter", "What is the " & String($index + 1) & " (st/nd/rd/th) letter?", _
                                 "", " 1", 125, 95, $windowPosition[0], $windowPosition[1])
            $tempString = $tempString & $tempChar;
            $letterDefinitionArray[$game][$letterDefinitionLength[$game]][16] = $tempChar;
            For $i = 0 to 15
                $letterDefinitionArray[$game][$letterDefinitionLength[$game]][$i] = $tempLetter[$i];
            Next
            $letterDefinitionLength[$game] += 1;
        EndIf
    Next

To simplify the process of comparing an array, I put every character (seperated by 3 non-black columns of pixels) into its own array. Then I trim the excessive whitespace (0 values ) of that array, and then I can compare it to a stored value much more easily. I'll show you how first, step by step, and then with complete code. Note that the code is a little bit buggy, since I just randomly threw it together, but it does work.

Splitting up each character into its own array is quite tricky. I think that was one of the parts I had a lot of problem with. Anyway, what I do is the following: I constantly read pixels into an array. I go through each column of pixels (already simplified into black (1) and non-black (0)) and count the number of black pixels. When it finds no black pixels at three times after eachother, then it stops putting pixels into the current array, and creates a new array for it to read into. This process is repeated until you end up with a lot of arrays, one array for each character. For actual code look in the total example.

Now, you go through each character and you trim off the excess non-black. That's what the Func _PixelArrayTrimEmptyField($pPixels) does. It just trims off all the 0 around the 1 in a 2 dimensional array, and returns a new array that has black on the outsides.

Ok, so now we've got an array for each character. We've trimmed this array down so that we can easily compare it to other arrays that hold. Let's compare the arrays. That's exactly what the Func _CompareArray($pPixel1, $pPixel2, $pAccuracy = 0.95) does. It compares arrays based on a percentage of accuracy. If the arrays match is greater than the required accuracy it returns a value that is true, otherwise it returns a value that is false.

And if you're wondering how to store the characters in your script, this is how I've done it. It is quite efficient and AutoIt interprets it fast, so there's no need for long "Loading character defitions..." messages later on in your project. This for example is the capital letter M:

Dim $_Char_M_Capital[8][10] = [[1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,1,1],[0,0,1,1,0,0,0,0,0,0], [0,0,0,1,1,0,0,0,0,0],[0,0,0,1,1,0,0,0,0,0], [0,0,1,1,0,0,0,0,0,0],[1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,1,1]]

There's actually a function that turns 2 dimensional arrays into strings like that, which I had to write ... I've called it Func _ToDefinition($pPixels) and it returns a string just like the one above.

Anyway, sorry for the BIG BIG BIG read. Here's the code, enjoy!

#include <Color.au3>
#include <Array.au3>
#include <Math.au3>

#include-once

AutoItSetOption("WinTitleMatchMode", 4)

;#include "RuneOCR.au3"

Dim $_Char_M_Capital[8][10] = [[1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,1,1],[0,0,1,1,0,0,0,0,0,0],[0,0,0,1,1,0,0,0,0,0],[0,0,0,1,1,0,0,0,0,0],[0,0,1,1,0,0,0,0,0,0],[1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,1,1]]
Dim $_Char_i_Lower[2][7] = [[1,0,1,1,1,1,1],[1,0,1,1,1,1,1]]
Dim $_Char_n_Lower[6][6] = [[1,1,1,1,1,1],[1,1,1,1,1,1],[1,1,0,0,0,0],[1,0,0,0,0,0],[1,1,1,1,1,1],[0,1,1,1,1,1]]
Dim $_Char_e_Lower[6][6] = [[0,1,1,1,1,0],[1,1,1,1,1,1],[1,0,0,1,0,1],[1,0,0,1,0,1],[1,1,1,1,0,1],[0,1,1,0,0,1]]

Dim $Characters[4][2] = [ [$_Char_M_Capital, "M"] , _
[$_Char_i_Lower, "i"] , _
[$_Char_n_Lower, "n"] , _
[$_Char_e_Lower, "e"] ]

Sleep(3000)
_Main()

Func _Main()
    Local $nWidth = 77
    Local $nHeight = 19
    Local $aStorage[$nWidth][$nHeight]
    
    Dim $m[2] = [262, 213]
    
    For $x = 0 to $nWidth-1
        For $y = 0 to $nHeight-1
            $aStorage[$x][$y] = PixelGetColor($m[0] + $x+1,$m[1] + $y+1)
        Next
    Next

    Local $nResult = _PixelArrayToText($aStorage)
    
    MsgBox(0, "", $nResult)
EndFunc

Func _PixelArrayToText($pStorage)
    Local $lReturn = ""
    
    Local $lPixels = _PixelSearchInArray($pStorage, 30)
    
    ;Local $lTrimPixels = _PixelArrayTrimEmptyField($lPixels)
    Local $lTrimPixels = $lPixels
    
    ;_ArrayDisplay($lTrimPixels,Default,Default,1)
    
    Local $i = 0
    While 1
        Local $lThisChar = _PixelArraySplit($lTrimPixels,$i)
        If @error Then ExitLoop
        
        $lThisChar = _PixelArrayTrimEmptyField($lThisChar)
        
        For $n = 0 to UBound($Characters)-1
            If ( _CompareArray($Characters[$n][0], $lThisChar) ) Then
                $lReturn &= $Characters[$n][1]
            EndIf
        Next
        
        ;InputBox( "", "", _ToDefinition( $lThisChar ) )
        
        ;MsgBox(0, $i, _CalculateChecksum($lThisChar) )
        
        $i += 1
    WEnd
    
    Return $lReturn
EndFunc

Func _CompareArray($pPixel1, $pPixel2, $pAccuracy = 0.95)
    Local $minX = _Min(UBound($pPixel1,1), UBound($pPixel2,1))
    Local $minY = _Min(UBound($pPixel1,2), UBound($pPixel2,2))
    
    Local $iDiff = 0
    Local $iTotal = ($minX) * ($minY)
    
    For $x = 0 to $minX-1
        For $y = 0 to $minY-1
            If ( $pPixel1[$x][$y] <> $pPixel2[$x][$y] ) Then
                $iDiff += 1
            EndIf
        Next
    Next
    
    $a = ( $iDiff / $iTotal )
    If ( $a < (1-$pAccuracy) ) Then ; Returns true based on a 90% match
        Return 1
    Else
        Return 0
    EndIf
EndFunc

Func _ToDefinition($pPixels)
    
    ;_ArrayDisplay($pPixels,Default,Default,1)
    
    Local $lReturn = "Dim $char[" & UBound($pPixels,1) & "][" & UBound($pPixels,2) & "] = ["
    
    For $x = 0 to UBound($pPixels,1)-1
        $lReturn &= "["
        For $y = 0 to UBound($pPixels,2)-1
            If ($pPixels[$x][$y]) Then
                $lReturn &= "1,"
            Else
                $lReturn &= "0,"
            EndIf
        Next
        $lReturn = StringTrimRight($lReturn,1)
        $lReturn &= "],"
    Next
    
    $lReturn = StringTrimRight($lReturn,1)
    $lReturn &= "]"
    
    Return $lReturn
    
EndFunc

Func _PixelArraySplit($pPixels,$pSplit)
    Local $thisChar = 0
    Local $prevEmpty = True
    Local $startX = -1
    Local $stopX = -1
    Local $a = 0
    
    For $x = 0 to UBound($pPixels,1)-1
        If ( _PixelColIsEmpty($pPixels,$x) ) Then
            If ( $prevEmpty == False ) Then
                $stopX = $x
                If ( $thisChar == $pSplit ) Then
                    $a = 1
                    ExitLoop
                Else
                    $thisChar += 1
                EndIf
            EndIf
            $prevEmpty = True
        Else
            If ( $prevEmpty == True ) Then
                $startX = $x
            EndIf
            $prevEmpty = False
        EndIf
    Next
    
    If ( $a == 0 ) Then
        Return SetError(1,0,0)
    EndIf
    
    $stopX -= 1
    Local $lReturn[Abs($startX-$stopX)][UBound($pPixels,2)]
    
    For $x = $startX to $stopX
        ReDim $lReturn[$x-$startX+1][UBound($pPixels,2)]
        
        For $y = 0 to UBound($pPixels,2)-1
            $lReturn[$x-$startX][$y] = $pPixels[$x][$y]
        Next
    Next
    
    Return $lReturn
EndFunc

Func _PixelColIsEmpty($pPixels,$x)
    For $y = 0 to UBound($pPixels,2)-1
        If ( $pPixels[$x][$y] ) Then
            Return 0
        EndIf
    Next
    Return 1
EndFunc

Func _PixelArrayTrimEmptyField($pPixels)
    Local $lowX = 100000
    Local $highX = 0
    Local $lowY = 100000
    Local $highY = 0
    
    For $x = 0 to UBound($pPixels,1)-1
        For $y = 0 to UBound($pPixels,2)-1
            If ( $pPixels[$x][$y] == 1 ) Then
                
                If ( $lowX > $x ) Then
                    $lowX = $x
                EndIf
                If ( $highX < $x ) Then
                    $highX = $x
                EndIf
                
                If ( $lowY > $y ) Then
                    $lowY = $y
                EndIf
                If ( $highY < $y ) Then
                    $highY = $y
                EndIf
                
            EndIf
        Next
    Next
    
    Local $lDiffX = Abs( $lowX - $highX )
    Local $lDiffY = Abs( $lowY - $highY )
    
    Local $lReturn[$lDiffX+1][$lDiffY+1]
    
    For $x = $lowX to $highX
        For $y = $lowY to $highY
            $lReturn[$x-$lowX][$y-$lowY] = $pPixels[$x][$y]
        Next
    Next
    
    Return $lReturn
EndFunc

Func _PixelSearchInArray($pPixels, $pVariation)
    Dim $lColor[2] = [0xe2e0d9, 0xd5d5d3]
    
    $lPixels = _FilterPixelsByColorArray($pPixels, $lColor, 30)
    
    Return $lPixels
EndFunc

Func _FilterPixelsByColorArray($pPixels, $pColors, $pVariation)
    If UBound($pPixels,0) <> 2 Then Return SetError(1,0,0)
    
    Local $pReturn[UBound($pPixels,1)][UBound($pPixels,2)]
    
    For $x = 1 to UBound($pPixels,1)-1
        For $y = 1 to UBound($pPixels,2)-1
            For $i = 0 to UBound($pColors)-1
                If ( _ColorInBounds( $pPixels[$x][$y], $pColors[$i] , 30) ) Then
                    $pReturn[$x][$y] = 1
                Else
                    $pReturn[$x][$y] = 0
                EndIf
            Next
        Next
    Next
    
    Return $pReturn
EndFunc

Func _ColorInBounds($pMColor, $pTColor, $pVariation)
    $lMCBlue = _ColorGetBlue($pMColor)
    $lMCGreen = _ColorGetGreen($pMColor)
    $lMCRed = _ColorGetRed($pMColor)
    
    $lTCBlue = _ColorGetBlue($pTColor)
    $lTCGreen = _ColorGetGreen($pTColor)
    $lTCRed = _ColorGetRed($pTColor)
    
    $a = Abs($lMCBlue - $lTCBlue)
    $b = Abs($lMCGreen - $lTCGreen)
    $c = Abs($lMCRed - $lTCRed)
    
    If ( ( $a < $pVariation ) AND ( $b < $pVariation ) AND ( $c < $pVariation ) ) Then
        Return 1
    Else
        Return 0
    EndIf
EndFunc
jvanegmond Universalist

jvanegmond

Posted
Posted

Hi EchaniDrgn, firstly thank you for your interest. The reason for not doing the method you described has everything to do with allowing a partial OCR match. In my situation, I could not predict the background colours nor the exact format, size and colour of the text I was reading and thus had to rely on a partial character. In my case I have chosen for the closest match to any definition character.

I could not predict where any inaccuracies in the process of comparing the read character to the definition might occur, and so I keep going until I know for sure that this character either is a close match or not.

After using both of these techniques, still not every character was read reliably and so I had to rely on a custom string compare which also worked with a margin of matching the original string. Quite the project.

github.com/jvanegmond

  • 1 month later...
EchaniDrgn Seeker

EchaniDrgn

Posted
Posted

Hi EchaniDrgn, firstly thank you for your interest. The reason for not doing the method you described has everything to do with allowing a partial OCR match. In my situation, I could not predict the background colours nor the exact format, size and colour of the text I was reading and thus had to rely on a partial character. In my case I have chosen for the closest match to any definition character.

I could not predict where any inaccuracies in the process of comparing the read character to the definition might occur, and so I keep going until I know for sure that this character either is a close match or not.

After using both of these techniques, still not every character was read reliably and so I had to rely on a custom string compare which also worked with a margin of matching the original string. Quite the project.

That's cool, I hadn't noticed the level of error you had in there before. I guess my OCR could be made to resemble your definition comparison by doing a bitwise and with 2^i numbers through each row and figuring out how many matches you get. I'm fortunate that the flash game I'm using my OCR with has the characters in the same place and the color I've chosen to look for is unique to the letters I'm comparing to.

Unfortunately the CAPTCHA images aren't so predictable. Would be great if it was. :-)

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...