Jump to content

Find BMP in another BMP (imagesearch)

junkew

By junkew,
in AutoIt Example Scripts

 Share

Recommended Posts

MasonMill Polymath

MasonMill

Posted
Posted

I was getting an error so i put all the functions in the main script so it looks like this:

pt('MustDeclareVars', 1)
;#include <GUIConstants.au3>
#include <GDIPlus.au3>
#Include <ScreenCapture.au3>

;~ Constants for type of picture matching
const $c24RGBFullMatch=1    ;Load as 24 bits and full match
const $c24RGBPartialMatch=2 ;Load as 24 bits and partial match
const $c16RGBFullMatch=3    ;Load as 16 bits and full match
const $c16RGBPartialMatch=4 ;Load as 16 bits and partial match

; ** Example start **
; Screen samples
Global Const $Bitmap1Filename = @TempDir & "\FULLSCREEN.bmp"
Global Const $Bitmap2Filename = @TempDir & "\CALCULATOR.bmp"
Global Const $Bitmap3Filename = @TempDir & "\BACKSPACE.bmp"

;BMP samples
Global Const $Bitmap4Filename = @TempDir & "\7WDS_BW.bmp"
Global Const $Bitmap5Filename = @TempDir & "\SEVEN_BW.bmp"
Global Const $Bitmap6Filename = @TempDir & "\CAT_BW.bmp"
Global Const $Bitmap7Filename = @TempDir & "\AUTOIT3.bmp";Make sure the homepage of autoit with 3 is visible
local $calcHWND, $begin, $pos, $aWinPos, $aWinCSize, $start
local $pBitmap, $BitmapData, $pixelFormat
Opt("WinTitleMatchMode", 4);Matching windows by advanced options

; Initialize GDI+ library
_GDIPlus_Startup ()

;Calculator, make sure its started
$calcHWND = WinGetHandle("[CLASS:SciCalc]")
if $calchwnd = "" Then
    run("calc.exe")
    sleep(2000)
    $calcHWND = WinGetHandle("[CLASS:SciCalc]")
EndIf

;Capture the calculator screen
$begin = TimerInit()
winactivate($calcHWND)
_ScreenCapture_CaptureWnd($Bitmap2Filename, $calcHWND,0,0,-1,-1,False)
ConsoleWrite("Saving calculator window " & TimerDiff($begin) & " milliseconds " & @LF)
winmove("[ACTIVE]","",223,341)

;Capture an area within the calculator area backspace
$begin = TimerInit()
$pos=controlgetpos("","","[CLASS:Button; INSTANCE:22]")
$aWinPos = WinGetPos($calchWnd)
$aWinCSize = WinGetClientSize($calchWnd)
;Q&D calculation of offsets to capture
_ScreenCapture_Capture($Bitmap3Filename, ($aWinPos[0]+$pos[0]) + ($aWinPos[2] - $aWinCSize[0])-3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])-3, $aWinPos[0]+$pos[0]+$pos[2]+3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])+$pos[3]-3,false)
ConsoleWrite("Saving backspacebutton " & TimerDiff($begin) & " milliseconds " & @LF)

; Capture full screen
$begin = TimerInit()
_ScreenCapture_Capture($Bitmap1Filename,0,0,-1,-1,False)
ConsoleWrite("Saving full screen took " & TimerDiff($begin) & " milliseconds " & @LF)


; Not very usefull to find full screen and tricky as screen is most likely changed (clock, output from application etc.)
; findTester("SCREEN",$Bitmap1Filename,TRUE);Find the full screen itself
; findTester($Bitmap1Filename,$Bitmap1Filename,$c24RGBFullMatch);Find the full screen itself 31 seconds
findTester($Bitmap1Filename,$Bitmap1Filename,$c24RGBPartialMatch);Find the full screen itself 2.8529088255245  seconds
; findTester($Bitmap1Filename,$Bitmap1Filename,$c16RGBFullMatch);Find the full screen itself 21 seconds
findTester($Bitmap1Filename,$Bitmap1Filename,$c16RGBPartialMatch);Find the full screen itself 1 seconds

; Be aware that overlapping windows, moving things on screen can make it difficult to find on full screen
findTester("SCREEN",$Bitmap2Filename,$c24RGBFullMatch);Find the full calculatorscreen
findTester("SCREEN",$Bitmap2Filename,$c24RGBPArtialMatch);Find the full calculatorscreen with partial match
findTester("SCREEN",$Bitmap2Filename,$c16RGBFullMatch);Find the full calculatorscreen
findTester("SCREEN",$Bitmap2Filename,$c16RGBPArtialMatch);Find the full calculatorscreen with partial match

findTester($Bitmap1Filename,$Bitmap2Filename,$c16RGBFullMatch);Find the full calculatorscreen
findTester($Bitmap1Filename,$Bitmap2Filename,$c16RGBPartialMatch);Find the full calculatorscreen with partial match

findTester($Bitmap2Filename,$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
findTester($Bitmap2Filename,$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
findTester($Bitmap2Filename,$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button
findTester($Bitmap2Filename,$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

winactivate($calcHWND);Make sure calculator is active on the screen
findTester("[ACTIVE]",$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
findTester("[ACTIVE]",$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
findTester("[ACTIVE]",$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button on active screen
findTester("[ACTIVE]",$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

findTester("SCREEN",$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
findTester("SCREEN",$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
findTester("SCREEN",$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button
findTester("SCREEN",$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

findTester($Bitmap4Filename,$Bitmap5Filename,$c16RGBFullMatch);Find the seven
findTester($Bitmap4Filename,$Bitmap5Filename,$c16RGBPartialMatch);Find the seven with partial match

findTester($Bitmap4Filename,$Bitmap6Filename,$c16RGBFullMatch);Find the cat
findTester($Bitmap4Filename,$Bitmap6Filename,$c16RGBPartialMatch);Find the cat with partial match

findTester($Bitmap1Filename,$Bitmap7Filename,$c24RGBFullMatch);Find the 3 of Autoit Homepage
findTester($Bitmap1Filename,$Bitmap7Filename,$c24RGBPartialMatch);Find the 3 of Autoit Homepage
findTester($Bitmap1Filename,$Bitmap7Filename,$c16RGBFullMatch);Find the 3 of Autoit Homepage
findTester($Bitmap1Filename,$Bitmap7Filename,$c16RGBPartialMatch);Find the 3 of Autoit Homepage



_GDIPlus_Shutdown()



func filegetname($f)
    local $i
    $i=stringinstr($f,"\",false,-1)
    if $i > 0 Then
        return stringmid($f,$i+1)
    Else
        return $f
    EndIf
EndFunc

;Do the actual find
Func FindTester($BMP1, $BMP2, $Bool)
    local $tResult
    $start = TimerInit()
    $tResult = findBMP($BMP1,$BMP2, $Bool)
    ConsoleWrite($tResult &  " " & FileGetName($BMP2) & " in " & FileGetName($BMP1) & " ** matchtype " & $Bool & " time elapsed: " & TimerDiff($start) & "  milliseconds" & @LF)
EndFunc

Func findBMP($BMP1, $BMP2, $MatchType=$c24RGBFullMatch)
    Dim $fLine[1];Line number of found line(s), redimmed when second picture size is known
    Dim $BMP1Data="", $BMP1Width=0, $BMP1Height=0, $BMP1LineWidth=0;
    Dim $BMP2Data="", $BMP2Width=0, $BMP2Height=0, $BMP2LineWidth=0
    Dim $foundAt = "", $matchPossible=FALSE, $matchedLines=0, $foundAtLeft=-1, $foundAtTop=-1
    Dim $bestMatchLine=-1, $HighestMatchingLines=-1; For knowing when no match is found where best area is
    Dim $iPos=1;
    dim $imgBytes;
    
    local $iFuzzyDist, $searchFor, $iAbove, $bMatchPossible, $aboveLine
    local $j, $imgBits

    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c24RGBPartialMatch) then
        $imgBytes=3
    Else    
        $imgBytes=2
    endif
    
; Load the bitmap to search in
    getImage($BMP1, $BMP1Data, $BMP1Width, $BMP1Height, $BMP1LineWidth, $imgBytes)
    $BMP1Data = BinaryToString($BMP1Data)

; Load the bitmap to find
    getImage($BMP2, $BMP2Data, $BMP2Width, $BMP2Height, $BMP2LineWidth, $imgBytes)
;Make it strings to be able to use string functions for searching
    $BMP2Data = BinaryToString($BMP2Data)
    
;For reference of line where in BMP2FindIn a line of BMP2Find was found
    If $BMP2Height = 0 Then
        SetError(1,0,0)
        Return  False
    EndIf

    ReDim $fline[$BMP2Height]
    
;If exact match check every 1 line else do it more fuzzy (as most likely other lines are unique)
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c16RGBFullMatch) Then
        $iFuzzyDist = 1
    Else
;Check fuzzy every 10% of lines
        $iFuzzyDist = ceiling(($bmp2height * 0.1))
    endIf

    $begin = TimerInit()
;Look for each line of the bitmap if it exists in the bitmap to find in
    For $i = 0 To $BMP2Height - 1
;Minus imgbytes as last bits are padded with unpredictable bytes (24 bits image assumption) or 2 when 16 bits
        $searchFor = StringMid($BMP2Data, 1 + ($i * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
        $iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos)
;       $iPos = StringInStr($BMP1Data, $searchFor)


;Look for all lines above if there is also a match
;Not doing it for the lines below as speed is more important and risk of mismatch on lines below is small
        $iAbove=1
        if $iPos > 0 then 
            $bMatchPossible=True
            $matchedLines=1;As first found line is matched we start counting
;Location of the match
                                                $foundAtTop = Int($iPos / $BMP1lineWidth) -$i
            $foundAtLeft =  int(mod($iPos,$bmp1linewidth) / $imgBytes)
        Else
            $bMatchPossible=false
            exitloop
        endif

        while (($i+$iAbove) <= ($BMP2Height -1)) and ($bMatchPossible=True)
            $searchFor = StringMid($BMP2Data, 1 + (($i + $iAbove) * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
            $aboveLine = stringmid($BMP1Data,$iPos + ($iAbove * $BMP1LineWidth), ($BMP2LineWidth - $imgBytes)) 
        
            if $aboveLine <> $searchFor Then 
                $bMatchPossible=False
;To remember the area with the best match
                if $matchedLines >= $HighestMatchingLines Then
                    $HighestMatchingLines = $matchedLines
                    
    ;Best guess of location
;~                  $foundAtTop = $fline[$i] + $i - $BMP2Height  
                    $foundAtTop = Int($iPos / $BMP1lineWidth);+ $i - $BMP2Height          
                    $bestMatchLine = Int($iPos / $BMP1lineWidth) 
                EndIf
                ExitLoop            
            EndIf
            $matchedLines=$matchedLines + 1
            $iAbove=$iAbove+$iFuzzyDist
        WEnd
        
;If bMatchPossible is still true most likely we have found the bitmap       
        if $bmatchPossible = True then
;~          ConsoleWrite("Could match top: " & $foundAtTop & " left: " & $foundAtLeft & " in " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;           MouseMove($foundatleft,$foundatTop)
            exitloop
        else 
;~          consolewrite("i not matched " & $ipos & " " & $matchedlines & @crlf )
        EndIf
    
    Next

;For some debugging of time
;   if $bMatchPossible = True Then
;       ConsoleWrite("Searching took " & TimerDiff($begin) / 1000 & "  seconds " & @LF)
;   Else
;       ConsoleWrite("NOT FOUND Searching took " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;   endif

;Return an error if not found else return an array with all information
    if $bMatchPossible = False Then
        SetError(1, 0, 0)
    endif
;   return stringsplit($bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine,";")
    return $bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine
EndFunc;==>findBMP




;    getImage(  $BMP1,         $BMP1Data,       $BMP1Width,   $BMP1Height,  $BMP1LineWidth, $imgBytes)

Func GetImage($BMPFile, byref $BMPDataStart, byref $Width, byRef $Height, byref $Stride, $imgBytes=3)
local $Scan0, $pixelData, $hbScreen, $pBitmap, $pBitmapCap, $handle

; Load the bitmap to search in
    If $BMPFile="SCREEN" Then
        $hbScreen=_ScreenCapture_Capture("",0,0,-1,-1,False)
        $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
    Else
;try to get a handle
        $handle = WinGetHandle($BMPFile)
        If @error Then
;Assume its an unknown handle so correct filename should be given
            $pBitmap = _GDIPlus_BitmapCreateFromFile($BMPFile)
        Else
            $hbScreen=_ScreenCapture_CaptureWnd("",$handle,0,0,-1,-1,False)
            $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap           
        EndIf
    EndIf
    
;Get $tagGDIPBITMAPDATA structure
;~  ConsoleWrite("Bitmap Width:    " & _GDIPlus_ImageGetWidth($pBitmap) & @CRLF )
;~  ConsoleWrite("Bitmap Height:      " & _GDIPlus_ImageGetHeight($pBitmap) & @CRLF)

;~  24 bits (3 bytes) or 16 bits (2 bytes) comparison
    if ($imgBytes=2) then
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF16RGB555)
    Else
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF24RGB)
    endIf
        
    If @ERROR Then MsgBox(0,"","Error locking region " & @error)
    
    $Stride = DllStructGetData($BitmapData, "Stride");Stride - Offset, in bytes, between consecutive scan lines of the bitmap. If the stride is positive, the bitmap is top-down. If the stride is negative, the bitmap is bottom-up.
    $Width = DllStructGetData($BitmapData, "Width");Image width - Number of pixels in one scan line of the bitmap.
    $Height = DllStructGetData($BitmapData, "Height");Image height - Number of scan lines in the bitmap.
    $PixelFormat = DllStructGetData($BitmapData, "PixelFormat");Pixel format - Integer that specifies the pixel format of the bitmap
    $Scan0 = DllStructGetData($BitmapData, "Scan0");Scan0 - Pointer to the first (index 0) scan line of the bitmap.
    
    $pixelData = DllStructCreate("ubyte lData[" & (abs($Stride) * $Height-1) & "]", $Scan0) 
    $BMPDataStart = $BMPDataStart & DllStructGetData($pixeldata,"lData")
    
    _GDIPlus_BitmapUnlockBits($pBitmap, $BitmapData)
    _GDIPlus_ImageDispose ($pBitmap)
    _WinAPI_DeleteObject ($pBitmap)

EndFunc;==>GetImage
;** Example end **
  • Replies 103
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

junkew
junkew

Code below searches a certain bitmap (all supported formats of GDI+) picture in another picture or on screen (after saved as bitmap) not depending on 3rd party dll's Pentium 4 of 3 GHz takes

junkew
junkew

updated 1st post with all working examples under W7 - 64 bits. Added more debugging lines so investigating based on hexfiles can be done if no match is found or why its unclear that there is no match

soadmania Seeker

soadmania

Posted
Posted (edited)

Damn i cant make it work.

Im trying to search this: Posted Image

in this: Posted Image

They are bmp actually. not jpg. And my code to search: _FindBMP( "capt.bmp", "1.bmp")

It always gives False;0;-1;-1;8;10;-1;-1 .

Help pl0x.

Edit: If i search this Posted Image in "SCREEN" it finds it. But if i search this Posted Image in "SCREEN" it cant find it. I ve tried in Partial mode too but its same.

Edited by soadmania
MasonMill Polymath

MasonMill

Posted
Posted

Junkew,

i looked in the DLL and its 2 = Element value out of range.

What does that mean? Other than the element value is out of range lol. Im not super pro programmer but i do understand a lot if you can explain it. Thanks!

MasonMill Polymath

MasonMill

Posted
Posted

Junkew,

Ok, sorry for all my posts. Sometimes when somethings acting fishy i do it on my laptop because it seems to work better. So when i pasted your original scripts the calculator get handle of the backspace didnt work because on mine there was no backspace text so i changed it to instance 2. So now THAT works but i get nothing after that. It doesn't consolewrite and it doesnt screen capture, i dont think, because i open a paint instance and push paste but it doesn't paste anything. Is that wrong for thinking that? After it drags the calculator in place it just waits about 10 seconds and the script terminates. Any ideas on whats going on? Again, i just used your first script posting. Thanks man! This script is very intuitive and would love to make a version of my own!

-Mason

MasonMill Polymath

MasonMill

Posted
Posted

Ok so i got it all working, yay! lol. But! Can you make it use a threshold of some sort, so it doesn't have to be an exact match? If im understanding it right it creates a string of 1's and 0's whether or not it finds a pixel match. Can it give a range of numbers as a threshold or would that severely slow it down? Makes me wonder if you could use MATLAB to create large arrays to make this more efficient and sped it up? And then you could give each value a threshold. Thanks!

  • 4 weeks later...
junkew Universalist

junkew

Posted
  • Author
Posted

It works a little differently. It doesn't create a string of 1 and 0.

It works on a line by line string basis of 3 combined bytes (RGB colors) which depends on if you do save as 16 or 24 bits pictures.

You can do manipulation on the whole picturestring by regexp replace to remove colors (replace every 3th byte with a 0). This is however tricky (as regexp function stops on byte with value 0) and timeconsuming so speedwise its not fast enough to find it within one second (at least thats normally my requirement for my purpose).

FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets

  • 5 months later...
n0manarmy Seeker

n0manarmy

Posted
Posted

Well I've got the application working for exact matches, unfortunately what I need to search for is sometimes a pixel larger or a pixel smaller all around as well as the middle of the number can be a different color. For example, if I'm searching for 9 or 8, they will be randomly one pixel larger or one pixel smaller as well as the middle of the 9 or the 8 might be gray, or yellow, with the 9 or 8 varying a few minute shades of green. I've tried converting to black and white which would work perfectly but I don't know of a function within AutoIT to convert. I tried imagemagick but its not as clean as I would like and is therefore worse. I can save the screenshot as mono with paint in Windows 7 and it gets rid of all the color variations and the colors in the middle. If I could find a way to replicate this then I would only need to worry about the pixel size and I'm sure the partial match would suffice.

  • 3 weeks later...
Kissen Seeker

Kissen

Posted
Posted

hello,

if I start the examples I get this errors:

Sample code from the frist post:

>"D:\Programme\Auto-IT\SciTe\..\autoit3.exe" /ErrorStdOut "D:\Programme\Auto-IT\Tests\findBMP\test1.au3" 
Saving calculator window 515.889309954198 milliseconds 
D:\Programme\Auto-IT\meine Scripte\Tests\imagesearch\imagesearch.au3 (51) : ==> Subscript used with non-Array variable.:
_ScreenCapture_Capture($Bitmap3Filename, ($aWinPos[0]+$pos[0]) + ($aWinPos[2] - $aWinCSize[0])-3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])-3, $aWinPos[0]+$pos[0]+$pos[2]+3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])+$pos[3]-3,false)
_ScreenCapture_Capture($Bitmap3Filename, ($aWinPos[0]+$pos^ ERROR
>Exit code: 1    Time: 3.161

Test code from this post: 

>"D:\Programme\Auto-IT\SciTe\..\autoit3.exe" /ErrorStdOut "D:\Programme\Auto-IT\Tests\findBMP\test2.au3"    
Saving full screen took 186.547172894879 milliseconds 
False** Full screen totalmatch time elapsed: 0.0231831140549986  seconds
>Exit code: 0    Time: 0.754

can you help me to run this function, please?

it sounds so great.

Kissen Seeker

Kissen

Posted
Posted

Hmm... I can't edit my first post.

I found my mistake... I replaced the bmp-image but I forgot to change this line:

$pos=controlgetpos("","","[CLASS:Button; TEXT:BACKSPACE]")

Now its working.

Great funktion! thank you for it!

  • 2 months later...
Arm Seeker

Arm

Posted
Posted

Can you upload your working scrip Kissen?

------------------------------------------------------"You are never a loser,until you quit trying"------------------------------------------------------

  • 11 months later...
  • 2 weeks later...
indecisive Seeker

indecisive

Posted
Posted 

Opt('MustDeclareVars', 1)
;#include <GUIConstants.au3>
#include <GDIPlus.au3>
const $c24RGBFullMatch=1    ;Load as 24 bits and full match
const $c24RGBPartialMatch=2 ;Load as 24 bits and partial match
const $c16RGBFullMatch=3    ;Load as 16 bits and full match
const $c16RGBPartialMatch=4 ;Load as 16 bits and partial match
MsgBox(0,"",findBMP("1.bmp", "2.bmp", $c24RGBFullMatch))

Func findBMP($BMP1, $BMP2, $MatchType=$c24RGBFullMatch)
    Dim $fLine[1];Line number of found line(s), redimmed when second picture size is known
    Dim $BMP1Data="", $BMP1Width=0, $BMP1Height=0, $BMP1LineWidth=0;
    Dim $BMP2Data="", $BMP2Width=0, $BMP2Height=0, $BMP2LineWidth=0
    Dim $foundAt = "", $matchPossible=FALSE, $matchedLines=0, $foundAtLeft=-1, $foundAtTop=-1
    Dim $bestMatchLine=-1, $HighestMatchingLines=-1; For knowing when no match is found where best area is
    Dim $iPos=1;
    dim $imgBytes;
    local $iFuzzyDist, $searchFor, $iAbove, $bMatchPossible, $aboveLine
    local $j, $imgBits
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c24RGBPartialMatch) then
        $imgBytes=3
    Else
        $imgBytes=2
    endif
; Load the bitmap to search in
    getImage($BMP1, $BMP1Data, $BMP1Width, $BMP1Height, $BMP1LineWidth, $imgBytes)
    $BMP1Data = BinaryToString($BMP1Data)
; Load the bitmap to find
    getImage($BMP2, $BMP2Data, $BMP2Width, $BMP2Height, $BMP2LineWidth, $imgBytes)
;Make it strings to be able to use string functions for searching
    $BMP2Data = BinaryToString($BMP2Data)
;For reference of line where in BMP2FindIn a line of BMP2Find was found
    If $BMP2Height = 0 Then
        SetError(1,0,0)
        Return  False
    EndIf
    ReDim $fline[$BMP2Height]
;If exact match check every 1 line else do it more fuzzy (as most likely other lines are unique)
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c16RGBFullMatch) Then
        $iFuzzyDist = 1
    Else
;Check fuzzy every 10% of lines
        $iFuzzyDist = ceiling(($bmp2height * 0.1))
    endIf
    $begin = TimerInit()
;Look for each line of the bitmap if it exists in the bitmap to find in
    For $i = 0 To $BMP2Height - 1
;Minus imgbytes as last bits are padded with unpredictable bytes (24 bits image assumption) or 2 when 16 bits
        $searchFor = StringMid($BMP2Data, 1 + ($i * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
        $iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos)
;      $iPos = StringInStr($BMP1Data, $searchFor)

;Look for all lines above if there is also a match
;Not doing it for the lines below as speed is more important and risk of mismatch on lines below is small
        $iAbove=1
        if $iPos > 0 then
            $bMatchPossible=True
            $matchedLines=1;As first found line is matched we start counting
;Location of the match
                                                $foundAtTop = Int($iPos / $BMP1lineWidth) -$i
            $foundAtLeft =  int(mod($iPos,$bmp1linewidth) / $imgBytes)
        Else
            $bMatchPossible=false
            exitloop
        endif
        while (($i+$iAbove) <= ($BMP2Height -1)) and ($bMatchPossible=True)
            $searchFor = StringMid($BMP2Data, 1 + (($i + $iAbove) * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
            $aboveLine = stringmid($BMP1Data,$iPos + ($iAbove * $BMP1LineWidth), ($BMP2LineWidth - $imgBytes))
            if $aboveLine <> $searchFor Then
                $bMatchPossible=False
;To remember the area with the best match
                if $matchedLines >= $HighestMatchingLines Then
                    $HighestMatchingLines = $matchedLines
    ;Best guess of location
;~                $foundAtTop = $fline[$i] + $i - $BMP2Height
                    $foundAtTop = Int($iPos / $BMP1lineWidth);+ $i - $BMP2Height
                    $bestMatchLine = Int($iPos / $BMP1lineWidth)
                EndIf
                ExitLoop
            EndIf
            $matchedLines=$matchedLines + 1
            $iAbove=$iAbove+$iFuzzyDist
        WEnd
;If bMatchPossible is still true most likely we have found the bitmap
        if $bmatchPossible = True then
;~        ConsoleWrite("Could match top: " & $foundAtTop & " left: " & $foundAtLeft & " in " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;          MouseMove($foundatleft,$foundatTop)
            exitloop
        else
;~        consolewrite("i not matched " & $ipos & " " & $matchedlines & @crlf )
        EndIf
    Next
;For some debugging of time
;   if $bMatchPossible = True Then
;      ConsoleWrite("Searching took " & TimerDiff($begin) / 1000 & "  seconds " & @LF)
;   Else
;      ConsoleWrite("NOT FOUND Searching took " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;   endif
;Return an error if not found else return an array with all information
    if $bMatchPossible = False Then
        SetError(1, 0, 0)
    endif
;   return stringsplit($bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine,";")
    return $bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine
EndFunc;==>findBMP
Func GetImage($BMPFile, byref $BMPDataStart, byref $Width, byRef $Height, byref $Stride, $imgBytes=3)
local $Scan0, $pixelData, $hbScreen, $pBitmap, $pBitmapCap, $handle
; Load the bitmap to search in
    If $BMPFile="SCREEN" Then
        $hbScreen=_ScreenCapture_Capture("",0,0,-1,-1,False)
        $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
    Else
;try to get a handle
        $handle = WinGetHandle($BMPFile)
        If @error Then
;Assume its an unknown handle so correct filename should be given
            $pBitmap = _GDIPlus_BitmapCreateFromFile($BMPFile)
        Else
            $hbScreen=_ScreenCapture_CaptureWnd("",$handle,0,0,-1,-1,False)
            $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
        EndIf
    EndIf
;Get $tagGDIPBITMAPDATA structure
;~  ConsoleWrite("Bitmap Width:    " & _GDIPlus_ImageGetWidth($pBitmap) & @CRLF )
;~  ConsoleWrite("Bitmap Height:      " & _GDIPlus_ImageGetHeight($pBitmap) & @CRLF)
;~  24 bits (3 bytes) or 16 bits (2 bytes) comparison
    if ($imgBytes=2) then
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF16RGB555)
    Else
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF24RGB)
    endIf
    If @ERROR Then MsgBox(0,"","Error locking region " & @error)
    $Stride = DllStructGetData($BitmapData, "Stride");Stride - Offset, in bytes, between consecutive scan lines of the bitmap. If the stride is positive, the bitmap is top-down. If the stride is negative, the bitmap is bottom-up.
    $Width = DllStructGetData($BitmapData, "Width");Image width - Number of pixels in one scan line of the bitmap.
    $Height = DllStructGetData($BitmapData, "Height");Image height - Number of scan lines in the bitmap.
    $PixelFormat = DllStructGetData($BitmapData, "PixelFormat");Pixel format - Integer that specifies the pixel format of the bitmap
    $Scan0 = DllStructGetData($BitmapData, "Scan0");Scan0 - Pointer to the first (index 0) scan line of the bitmap.
    $pixelData = DllStructCreate("ubyte lData[" & (abs($Stride) * $Height-1) & "]", $Scan0)
    $BMPDataStart = $BMPDataStart & DllStructGetData($pixeldata,"lData")
    _GDIPlus_BitmapUnlockBits($pBitmap, $BitmapData)
    _GDIPlus_ImageDispose ($pBitmap)
    _WinAPI_DeleteObject ($pBitmap)
EndFunc;==>GetImage

-can somebody tell me what's wrong with this code? I just pasted the functions posted on the first page of the thread, the necessary declarations, and the statement that calls _findBMP. Please help me with this. Thanks! ;)

indecisive Seeker

indecisive

Posted
Posted 

Opt('MustDeclareVars', 1)
;#include <GUIConstants.au3>
#include <GDIPlus.au3>
const $c24RGBFullMatch=1    ;Load as 24 bits and full match
const $c24RGBPartialMatch=2 ;Load as 24 bits and partial match
const $c16RGBFullMatch=3    ;Load as 16 bits and full match
const $c16RGBPartialMatch=4 ;Load as 16 bits and partial match
MsgBox(0,"",findBMP("1.bmp", "2.bmp", $c24RGBFullMatch))

Func findBMP($BMP1, $BMP2, $MatchType=$c24RGBFullMatch)
    Dim $fLine[1];Line number of found line(s), redimmed when second picture size is known
    Dim $BMP1Data="", $BMP1Width=0, $BMP1Height=0, $BMP1LineWidth=0;
    Dim $BMP2Data="", $BMP2Width=0, $BMP2Height=0, $BMP2LineWidth=0
    Dim $foundAt = "", $matchPossible=FALSE, $matchedLines=0, $foundAtLeft=-1, $foundAtTop=-1
    Dim $bestMatchLine=-1, $HighestMatchingLines=-1; For knowing when no match is found where best area is
    Dim $iPos=1;
    dim $imgBytes;
    local $iFuzzyDist, $searchFor, $iAbove, $bMatchPossible, $aboveLine
    local $j, $imgBits
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c24RGBPartialMatch) then
        $imgBytes=3
    Else
        $imgBytes=2
    endif
; Load the bitmap to search in
    getImage($BMP1, $BMP1Data, $BMP1Width, $BMP1Height, $BMP1LineWidth, $imgBytes)
    $BMP1Data = BinaryToString($BMP1Data)
; Load the bitmap to find
    getImage($BMP2, $BMP2Data, $BMP2Width, $BMP2Height, $BMP2LineWidth, $imgBytes)
;Make it strings to be able to use string functions for searching
    $BMP2Data = BinaryToString($BMP2Data)
;For reference of line where in BMP2FindIn a line of BMP2Find was found
    If $BMP2Height = 0 Then
        SetError(1,0,0)
        Return  False
    EndIf
    ReDim $fline[$BMP2Height]
;If exact match check every 1 line else do it more fuzzy (as most likely other lines are unique)
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c16RGBFullMatch) Then
        $iFuzzyDist = 1
    Else
;Check fuzzy every 10% of lines
        $iFuzzyDist = ceiling(($bmp2height * 0.1))
    endIf
    $begin = TimerInit()
;Look for each line of the bitmap if it exists in the bitmap to find in
    For $i = 0 To $BMP2Height - 1
;Minus imgbytes as last bits are padded with unpredictable bytes (24 bits image assumption) or 2 when 16 bits
        $searchFor = StringMid($BMP2Data, 1 + ($i * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
        $iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos)
;      $iPos = StringInStr($BMP1Data, $searchFor)

;Look for all lines above if there is also a match
;Not doing it for the lines below as speed is more important and risk of mismatch on lines below is small
        $iAbove=1
        if $iPos > 0 then
            $bMatchPossible=True
            $matchedLines=1;As first found line is matched we start counting
;Location of the match
                                                $foundAtTop = Int($iPos / $BMP1lineWidth) -$i
            $foundAtLeft =  int(mod($iPos,$bmp1linewidth) / $imgBytes)
        Else
            $bMatchPossible=false
            exitloop
        endif
        while (($i+$iAbove) <= ($BMP2Height -1)) and ($bMatchPossible=True)
            $searchFor = StringMid($BMP2Data, 1 + (($i + $iAbove) * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
            $aboveLine = stringmid($BMP1Data,$iPos + ($iAbove * $BMP1LineWidth), ($BMP2LineWidth - $imgBytes))
            if $aboveLine <> $searchFor Then
                $bMatchPossible=False
;To remember the area with the best match
                if $matchedLines >= $HighestMatchingLines Then
                    $HighestMatchingLines = $matchedLines
    ;Best guess of location
;~                $foundAtTop = $fline[$i] + $i - $BMP2Height
                    $foundAtTop = Int($iPos / $BMP1lineWidth);+ $i - $BMP2Height
                    $bestMatchLine = Int($iPos / $BMP1lineWidth)
                EndIf
                ExitLoop
            EndIf
            $matchedLines=$matchedLines + 1
            $iAbove=$iAbove+$iFuzzyDist
        WEnd
;If bMatchPossible is still true most likely we have found the bitmap
        if $bmatchPossible = True then
;~        ConsoleWrite("Could match top: " & $foundAtTop & " left: " & $foundAtLeft & " in " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;          MouseMove($foundatleft,$foundatTop)
            exitloop
        else
;~        consolewrite("i not matched " & $ipos & " " & $matchedlines & @crlf )
        EndIf
    Next
;For some debugging of time
;   if $bMatchPossible = True Then
;      ConsoleWrite("Searching took " & TimerDiff($begin) / 1000 & "  seconds " & @LF)
;   Else
;      ConsoleWrite("NOT FOUND Searching took " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;   endif
;Return an error if not found else return an array with all information
    if $bMatchPossible = False Then
        SetError(1, 0, 0)
    endif
;   return stringsplit($bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine,";")
    return $bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine
EndFunc;==>findBMP
Func GetImage($BMPFile, byref $BMPDataStart, byref $Width, byRef $Height, byref $Stride, $imgBytes=3)
local $Scan0, $pixelData, $hbScreen, $pBitmap, $pBitmapCap, $handle
; Load the bitmap to search in
    If $BMPFile="SCREEN" Then
        $hbScreen=_ScreenCapture_Capture("",0,0,-1,-1,False)
        $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
    Else
;try to get a handle
        $handle = WinGetHandle($BMPFile)
        If @error Then
;Assume its an unknown handle so correct filename should be given
            $pBitmap = _GDIPlus_BitmapCreateFromFile($BMPFile)
        Else
            $hbScreen=_ScreenCapture_CaptureWnd("",$handle,0,0,-1,-1,False)
            $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
        EndIf
    EndIf
;Get $tagGDIPBITMAPDATA structure
;~  ConsoleWrite("Bitmap Width:    " & _GDIPlus_ImageGetWidth($pBitmap) & @CRLF )
;~  ConsoleWrite("Bitmap Height:      " & _GDIPlus_ImageGetHeight($pBitmap) & @CRLF)
;~  24 bits (3 bytes) or 16 bits (2 bytes) comparison
    if ($imgBytes=2) then
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF16RGB555)
    Else
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF24RGB)
    endIf
    If @ERROR Then MsgBox(0,"","Error locking region " & @error)
    $Stride = DllStructGetData($BitmapData, "Stride");Stride - Offset, in bytes, between consecutive scan lines of the bitmap. If the stride is positive, the bitmap is top-down. If the stride is negative, the bitmap is bottom-up.
    $Width = DllStructGetData($BitmapData, "Width");Image width - Number of pixels in one scan line of the bitmap.
    $Height = DllStructGetData($BitmapData, "Height");Image height - Number of scan lines in the bitmap.
    $PixelFormat = DllStructGetData($BitmapData, "PixelFormat");Pixel format - Integer that specifies the pixel format of the bitmap
    $Scan0 = DllStructGetData($BitmapData, "Scan0");Scan0 - Pointer to the first (index 0) scan line of the bitmap.
    $pixelData = DllStructCreate("ubyte lData[" & (abs($Stride) * $Height-1) & "]", $Scan0)
    $BMPDataStart = $BMPDataStart & DllStructGetData($pixeldata,"lData")
    _GDIPlus_BitmapUnlockBits($pBitmap, $BitmapData)
    _GDIPlus_ImageDispose ($pBitmap)
    _WinAPI_DeleteObject ($pBitmap)
EndFunc;==>GetImage

-can somebody tell me what's wrong with this code? I just pasted the functions posted on the first page of the thread, the necessary declarations, and the statement that calls _findBMP. Please help me with this. Thanks! ;)

  • 1 month later...
junkew Universalist

junkew

Posted
  • Author
Posted (edited)

First try to get the initial post working and work from there

I revised the initial post under W7 / 64 bits only needing those 2 "fixes"

replace

$calcHWND = WinGetHandle("[CLASS:SciCalc]")

with

$calcHWND = WinGetHandle("[REGEXPCLASS:.*Calc.*]")

and

replace

$pos=controlgetpos("","","[CLASS:Button; TEXT:BACKSPACE]")

with

$pos=controlgetpos("","","[CLASS:Button; INSTANCE:2]")

Somehow on the calculator I am not able to find by TEXT: BACKSPACE

Is TEXT broken in getControlPos even AU3Info x86 and 64 seems not to be identifying with text on controls. AU3Info 64 seems to be behaving best.

I have some doubt on finding in full screen as it seems to be misaligning bytes and as such no match on fullscreen finds. It works however on default window bitmaps

** There seems to be a difference when AERO W7 theme is used compared to Windows classic theme **

To me its a little unclear why the BMP data in itself is then different. I would expect a BMP to be a BMP but somehow its not. So for testing you should use windows classic theme

Edited by junkew

FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets

ienthach Seeker

ienthach

Posted
Posted

Great job.

What about shade variation image?

I capture a window simulator of android then find a subimg (captured before) in it. But the first time it's successful, after click on subimg (subimg is a button) then comeback previous screen, findbmp was unsuccessful.

  • 7 months later...
junkew Universalist

junkew

Posted
  • Author
Posted

updated 1st post with all working examples under W7 - 64 bits.

Added more debugging lines so investigating based on hexfiles can be done if no match is found or why its unclear that there is no match.

Added a small threshold to have a match of 97% of the bits/bytes on a line. (small risk of false positives) which seems to be a workaround for weird situation that calculator finding gives a false.

Investigating the false on calculator.bmp showed in the debugfiles that actually windows/GDI is indeed making calculator.bmp bits different compared to the same bits in a full screen where calculator is shown.

FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets

junkew Universalist

junkew

Posted
  • Author
Posted

Great job.

What about shade variation image?

I capture a window simulator of android then find a subimg (captured before) in it. But the first time it's successful, after click on subimg (subimg is a button) then comeback previous screen, findbmp was unsuccessful.

Please try again. Should work now based on changed code and should be easier to investigate why it fails matching.

FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...