Find BMP in another BMP (imagesearch)

[junkew]

Code below searches a certain bitmap (all supported formats of GDI+) picture in another picture or on screen (after saved as bitmap)

not depending on 3rd party dll's

Pentium 4 of 3 GHz takes

* about 4.5 seconds on a 1280*1024 desktop to find a 260*260 bitmap. Partial match 0.5 seconds

* about 1 seconds on a 1280*1024 desktop to find a 20*17 bitmap. Partial match 0.5 seconds

Some references I used to come up with this solution

* code to get a smaller bitmap: http://www.autoitscript.com/forum/index.ph...=pixelchecksum)

* Pixelchecksum seems not to do the full job but in combination with below it can become very fast: http://www.autoitscript.com/forum/index.ph...l=pixelchecksum

* A pixel by pixel searcher http://www.autoitscript.com/forum/index.ph...&hl=compare

* UDF Get or Read Pixel from Memory

* GetDIBits

Not fully finished but all comments/enhancements are appreciated

* Save to file or directly on special window description SCREEN, [ACTIVE] or name of picture file

* Searching is done on a line by line basis instead of pixel by pixel

* Bitmap to search in example program should be fully visible to work for first 3 examples as it are screenshots taken. Same applies for 3 in AUTOIT homepage

* Speed depends on the size of the bitmap you are searching for but is about linear and < 5 seconds

* Some doubts on left and top returned (sometimes seems to be off by 1 or several pixels)

* Transparency (when search letters on a different background) http://www.winprog.org/tutorial/transparency.html

* Match quicker by doing a bitblt with srcinvert when first line is found (instead of checking line by line)

* $BMP1 and $BMP2 to be HBITMAP handle as input instead of filenames (will make searching within partial screen

Sample to work

* Install pics.zip into %temp% / Save BMP attachments for example to @tempdir (%temp%) for example to work

Sample code

expandcollapse popup

#Region includes
Opt('MustDeclareVars', 1)
;#include <GUIConstants.au3>
#include <GDIPlus.au3>
#Include <ScreenCapture.au3>
#include <string.au3>
#include <WinAPI.au3>
#include <WindowsConstants.au3>
#EndRegion

;~ If debugging output is wanted to understand and analyse the algorithm put this on true/false
;~ Const $bDebug=True
Const $bDebug=False

#region example
;~ Const $cMatchLinePercentage=0.97 ;Minimum matching, put on 1 if exact matching is needed on all lines
Const $cMatchLinePercentage=0.99    ;Minimum matching, put on 1 if exact matching is needed on all lines

;~ Constants for type of picture matching
const $c24RGBFullMatch=1    ;Load as 24 bits and full match
const $c24RGBPartialMatch=2 ;Load as 24 bits and partial match
const $c16RGBFullMatch=3    ;Load as 16 bits and full match
const $c16RGBPartialMatch=4 ;Load as 16 bits and partial match

; ** Example start **
; Screen samples
Global Const $Bitmap1Filename = @scriptDir & "\FULLSCREEN.BMP"
Global Const $Bitmap2Filename = @scriptDir & "\CALCULATOR.BMP"
Global Const $Bitmap3Filename = @scriptDir & "\BACKSPACE.BMP"

;BMP samples
Global Const $Bitmap4Filename = @scriptDir & "\7WDS_BW.BMP"
Global Const $Bitmap5Filename = @scriptDir & "\SEVEN_BW.BMP"
Global Const $Bitmap6Filename = @scriptDir & "\CAT_BW.BMP"
Global Const $Bitmap7Filename = @scriptDir & "\AUTOIT3.bmp";Make sure the homepage of autoit with 3 is visible
local $calcHWND, $begin, $pos, $aWinPos, $aWinCSize, $start
local $pBitmap, $BitmapData, $pixelFormat
Opt("WinTitleMatchMode", 4);Matching windows by advanced options

; Initialize GDI+ library
_GDIPlus_Startup ()

;Calculator, make sure its started
$calcHWND = WinGetHandle("[REGEXPCLASS:.*Calc.*]")
If $calcHWND = "" Then
Run("calc.exe")
Sleep(2000)
$calcHWND = WinGetHandle("[REGEXPCLASS:.*Calc.*]")
EndIf

If $calcHWND = "" Then
    consolewrite("Using active window which is 99% sure equal to the calculator window" & @LF )
    $calcHWND = WinGetHandle("[ACTIVE]")
EndIf

consolewrite($calcHWND)

;Capture the calculator screen
$begin = TimerInit()
winactivate($calcHWND)
Sleep(1000)
_ScreenCapture_CaptureWnd($Bitmap2Filename, $calcHWND,0,0,-1,-1,False)
ConsoleWrite("Saving calculator window " & TimerDiff($begin) & " milliseconds " & @LF)
winmove("[ACTIVE]","",223,341)

;~ Get the base information of the calc window
$aWinPos = WinGetPos($calchWnd)
$aWinCSize = WinGetClientSize($calchWnd)
Sleep(500)

;Capture an area within the calculator area backspace
$begin = TimerInit()
$pos=controlgetpos("","","[CLASS:Button; TEXT:BACKSPACE]")
;~ $pos=controlgetpos("","","[CLASS:Button; INSTANCE:2]")
if @error=1 Then
    consolewrite("Not recognizing so we are on a windows 10 environment most likely, fix by providing manual controls" & @CRLF)
    consolewrite("Read frequently asked questions in Wiki number 31 and use simplespy" & @CRLF)
    local $pos[4]

;~  Backspace button <394;693;75;74>
;~  Button 2         <315;927;75;74>
    $pos[0]=394
    $pos[1]=693
    $pos[2]=75
    $pos[3]=74
;~  Func _UIA_DrawRect($tLeft, $tRight, $tTop, $tBottom, $color = 0xFF, $PenWidth = 4)
    _UIA_DrawRect($pos[0],$pos[0]+$pos[2],$pos[1],$pos[1]+$pos[2])

;~  Fix quick and dirty as simplespy gives screencoordinates not relative to window
    $pos[0]=394 - $aWinpos[0]
    $pos[1]=693 - $aWinpos[1]
EndIf


;Q&D calculation of offsets to capture
_ScreenCapture_Capture($Bitmap3Filename, ($aWinPos[0]+$pos[0]) + ($aWinPos[2] - $aWinCSize[0])-3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])-3, $aWinPos[0]+$pos[0]+$pos[2]+3, ($awinpos[1]+$pos[1])+ ($aWinPos[3]-$aWinCSize[1])+$pos[3]-3,false)
ConsoleWrite("Saving backspacebutton " & TimerDiff($begin) & " milliseconds " & @LF)

; Capture full screen
$begin = TimerInit()
_ScreenCapture_Capture($Bitmap1Filename,0,0,-1,-1,False)
ConsoleWrite("Saving full screen took " & TimerDiff($begin) & " milliseconds " & @LF)
Sleep(500)

func filegetname($f)
    local $i
    $i=stringinstr($f,"\",false,-1)
    if $i > 0 Then
        return stringmid($f,$i+1)
    Else
        return $f
    EndIf
EndFunc

;Do the actual find
Func FindTester($BMP1, $BMP2, $Bool)
    local $tResult
    $start = TimerInit()
    $tResult=findBMP($BMP1,$BMP2, $Bool)
    ConsoleWrite($tResult &  " " & FileGetName($BMP2) & " in " & FileGetName($BMP1) & " ** matchtype " & $Bool & " time elapsed: " & TimerDiff($start) & "  milliseconds" & @LF)
EndFunc

if not $bDebug=True Then
    ; Not very usefull to find full screen and tricky as screen is most likely changed (clock, output from application etc.)
    ; findTester("SCREEN",$Bitmap1Filename,TRUE);Find the full screen itself
    ; findTester($Bitmap1Filename,$Bitmap1Filename,$c24RGBFullMatch);Find the full screen itself 31 seconds
    findTester($Bitmap1Filename,$Bitmap1Filename,$c24RGBPartialMatch);Find the full screen itself 2.8529088255245  seconds
    ; findTester($Bitmap1Filename,$Bitmap1Filename,$c16RGBFullMatch);Find the full screen itself 21 seconds
    findTester($Bitmap1Filename,$Bitmap1Filename,$c16RGBPartialMatch);Find the full screen itself 1 seconds

    ; Be aware that overlapping windows, moving things on screen can make it difficult to find on full screen
    findTester("SCREEN",$Bitmap2Filename,$c24RGBFullMatch);Find the full calculatorscreen
    findTester("SCREEN",$Bitmap2Filename,$c24RGBPArtialMatch);Find the full calculatorscreen with partial match
    findTester("SCREEN",$Bitmap2Filename,$c16RGBFullMatch);Find the full calculatorscreen
    findTester("SCREEN",$Bitmap2Filename,$c16RGBPArtialMatch);Find the full calculatorscreen with partial match

    findTester($Bitmap1Filename,$Bitmap2Filename,$c16RGBFullMatch);Find the full calculatorscreen
    findTester($Bitmap1Filename,$Bitmap2Filename,$c16RGBPartialMatch);Find the full calculatorscreen with partial match

    findTester($Bitmap2Filename,$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
    findTester($Bitmap2Filename,$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
    findTester($Bitmap2Filename,$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button
    findTester($Bitmap2Filename,$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

    winactivate($calcHWND);Make sure calculator is active on the screen
    findTester("[ACTIVE]",$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
    findTester("[ACTIVE]",$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
    findTester("[ACTIVE]",$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button on active screen
    findTester("[ACTIVE]",$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

    findTester("SCREEN",$Bitmap3Filename,$c24RGBFullMatch);Find the backspace button
    findTester("SCREEN",$Bitmap3Filename,$c24RGBPartialMatch);Find the backspace button with partial match
    findTester("SCREEN",$Bitmap3Filename,$c16RGBFullMatch);Find the backspace button
    findTester("SCREEN",$Bitmap3Filename,$c16RGBPartialMatch);Find the backspace button with partial match

    findTester($Bitmap4Filename,$Bitmap5Filename,$c16RGBFullMatch);Find the seven
    findTester($Bitmap4Filename,$Bitmap5Filename,$c16RGBPartialMatch);Find the seven with partial match

    findTester($Bitmap4Filename,$Bitmap6Filename,$c16RGBFullMatch);Find the cat
    findTester($Bitmap4Filename,$Bitmap6Filename,$c16RGBPartialMatch);Find the cat with partial match

    findTester($Bitmap1Filename,$Bitmap7Filename,$c24RGBFullMatch);Find the 3 of Autoit Homepage
    findTester($Bitmap1Filename,$Bitmap7Filename,$c24RGBPartialMatch);Find the 3 of Autoit Homepage
    findTester($Bitmap1Filename,$Bitmap7Filename,$c16RGBFullMatch);Find the 3 of Autoit Homepage
    findTester($Bitmap1Filename,$Bitmap7Filename,$c16RGBPartialMatch);Find the 3 of Autoit Homepage
Else
;~  findTester($Bitmap4Filename,$Bitmap6Filename,$c24RGBFullMatch);Find the cat
;~  findTester($Bitmap1Filename,$Bitmap2Filename,$c24RGBFullMatch);Find the full calculatorscreen
    findTester("SCREEN",$Bitmap2Filename,$c24RGBFullMatch);Find the full calculatorscreen
EndIf

_GDIPlus_Shutdown()

;** Example end **
#EndRegion

#Region actual functions
;===============================================================================
; Function Name:    findBMP
; Description:    Finds a bitmap (.BMP) in another BMP file (other formats PNG and TIFF work also other formats less usefull GIF, JPEG, Exif, WMF, and EMF should work)
; Syntax:          findBMP($BMP1, $BMP2, $MatchType=TRUE)
;
; Parameter(s):  $BMP1           = Filename of bitmap to search in
;                  $BMP2             = Filename of bitmap to search for
;                  $MatchType       = c24RGBFullMatch, c24RGBPartialMatch, c16RGBFullMatch, c16RGBPartialMatch
;
; Return Value(s):  On Success:   = Returns Array List
;                  On Failure:   = @error 1 (Control was found but there was an error with the DLLCall)
;
; Author(s):        JunkEW
;
; Note(s):
;               * Its never an exact match even with TRUE as last few bits are disposed in algorithm and lines below
;                are not checked under assumption that those are 99.99% of the time correct
;              * locking bits overview http://www.bobpowell.net/lockingbits.htm
; ToDo:
;               * Transparency (when search letters on a different background) http://www.winprog.org/tutorial/transparency.html
;               * Match quicker by doing a bitblt with srcinvert when first line is found (instead of checking line by line)
;               * $BMP1 and $BMP2 to be HBITMAP handle as input instead of filenames (will make searching within partial screen easier)
; Example(s):
;
;===============================================================================

Func findBMP($BMP1, $BMP2, $MatchType=$c24RGBFullMatch)
    Dim $fLine[1];Line number of found line(s), redimmed when second picture size is known
    Dim $BMP1Data="", $BMP1Width=0, $BMP1Height=0, $BMP1LineWidth=0;
    Dim $BMP2Data="", $BMP2Width=0, $BMP2Height=0, $BMP2LineWidth=0
    Dim $foundAt = "", $matchPossible=FALSE, $matchedLines=0, $foundAtLeft=-1, $foundAtTop=-1
    Dim $bestMatchLine=-1, $HighestMatchingLines=-1; For knowing when no match is found where best area is
    Dim $iPos=1;
    dim $imgBytes, $avData;

    local $iFuzzyDist, $searchFor, $iAbove, $bMatchPossible, $aboveLine
    local $j, $imgBits
    local $tDebug

    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c24RGBPartialMatch) then
        $imgBytes=3
    Else
        $imgBytes=2
    endif

; Load the bitmap to search in
    getImage($BMP1, $BMP1Data, $BMP1Width, $BMP1Height, $BMP1LineWidth, $imgBytes)
    $BMP1Data = BinaryToString($BMP1Data)

; Load the bitmap to find
    getImage($BMP2, $BMP2Data, $BMP2Width, $BMP2Height, $BMP2LineWidth, $imgBytes)
;Make it strings to be able to use string functions for searching
    $BMP2Data = BinaryToString($BMP2Data)

if $bDebug=True Then
;~ START debugging
;Get some debugging information
    FileDelete(@ScriptDir & "\BMP1DATA.TXT")
    FileDelete(@ScriptDir & "\BMP2DATA.TXT")
    FileDelete(@ScriptDir & "\COMPAREDLINES.TXT")

    consolewrite($BMP1Width & @crlf)
    consolewrite($BMP1Height & @crlf)
    consolewrite($BMP1LineWidth & @crlf)
    consolewrite(stringlen($BMP1Data) & @crlf)

    consolewrite(".{" & $BMP1LineWidth & "}"& @CRLF)

    consolewrite(".{" & $BMP2LineWidth & "}"& @CRLF)
    $tDebug=StringRegExpReplace(_stringtohex($BMP1Data),"(.{" & $BMP1LineWidth*2 & "})","$1" & @CRLF)
    consolewrite(@error & @extended & @CRLF)
    FileWrite (@ScriptDir & "\BMP1DATA.TXT" , $tdebug)

    $tDebug=StringRegExpReplace(_stringtohex($BMP2Data),"(.{" & $BMP2LineWidth*2 & "})","$1" & @CRLF)
    consolewrite(@error & @extended & @CRLF)
    FileWrite (@ScriptDir & "\BMP2DATA.TXT" , $tdebug)
;~ END debugging
EndIf

;For reference of line where in BMP2FindIn a line of BMP2Find was found
    If $BMP2Height = 0 Then
        SetError(1,0,0)
        Return  False
    EndIf

    ReDim $fline[$BMP2Height]

;If exact match check every 1 line else do it more fuzzy (as most likely other lines are unique)
    if ($MatchType=$c24RGBFullMatch) or ($matchtype=$c16RGBFullMatch) Then
        $iFuzzyDist = 1
    Else
;Check fuzzy every 10% of lines
        $iFuzzyDist = ceiling(($bmp2height * 0.1))
    endIf

    $begin = TimerInit()
;Look for each line of the bitmap if it exists in the bitmap to find in
;~ Split bitmap to search in lines
;~  $avData=stringregexp($BMP2Data, ".{1," & $BMP2lineWidth & "}+", 3)
dim $searchForRegEx
$searchForRegEx = StringMid($BMP2Data, 1 + (0 * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
$searchForRegEx = $searchForRegEx & ".*" & StringMid($BMP2Data, 1 + ($BMP2Height * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
Local $aArray=stringregexp($BMP1Data,$searchForRegEx,3)
if @error=0 Then
    consolewrite("Regex found " & TimerDiff($begin) & " ms " & @crlf)
;~ For $i = 0 To UBound($aArray) - 1
;~     consolewrite("RegExp Test with Option 3 - " & $i & $aArray[$i])
;~ Next
endif


    For $i = 0 To $BMP2Height - 1
;Minus imgbytes as last bits are padded with unpredictable bytes (24 bits image assumption) or 2 when 16 bits
        $searchFor = StringMid($BMP2Data, 1 + ($i * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
;~      $searchfor=stringleft($avData[$i],$BMP2lineWidth - $imgBytes)

        $iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos)
;       $iPos = StringInStr($BMP1Data, $searchFor)


;Look for all lines above if there is also a match
;Not doing it for the lines below as speed is more important and risk of mismatch on lines below is small
        $iAbove=1
        if $iPos > 0 then
            $bMatchPossible=True
            $matchedLines=1;As first found line is matched we start counting
;Location of the match
            $foundAtTop = Int($iPos / $BMP1lineWidth) -$i
            $foundAtLeft =  int(mod($iPos,$bmp1linewidth) / $imgBytes)
        Else
;~          No 1st line found so far nothing to start matching
            $bMatchPossible=false
            exitloop
        endif

        while (($i+$iAbove) <= ($BMP2Height -1)) and ($bMatchPossible=True)
            $searchFor = StringMid($BMP2Data, 1 + (($i + $iAbove) * $BMP2lineWidth), ($BMP2lineWidth - $imgBytes))
;~          $searchfor = stringleft($avData[$i+$iAbove],$BMP2lineWidth - $imgBytes)
            $aboveLine = stringmid($BMP1Data,$iPos + ($iAbove * $BMP1LineWidth), ($BMP2LineWidth - $imgBytes))

            if $bDebug=True Then
            ;~ START debugging
                $tDebug=StringRegExpReplace(_stringtohex($searchfor),"(.{8})","$1_")
                FileWrite (@ScriptDir & "\COMPAREDLINES.TXT" , $tDebug & @crlf)
                $tDebug=StringRegExpReplace(_stringtohex($aboveline),"(.{8})","$1_")
                FileWrite (@ScriptDir & "\COMPAREDLINES.TXT" , $tDebug & @crlf)
                If $aboveLine <> $searchFor Then
                    FileWrite (@ScriptDir & "\COMPAREDLINES.TXT" , "** MISMATCH ** of line " & (1+$iAbove) & " in 2nd bitmap at line " & ($foundAtTop + $i) & @crlf)
                endIf
            ;~ END debugging
            EndIf

            if comparePicLine($aboveline,$searchfor) = false then
                $bMatchPossible=False
;To remember the area with the best match
                if $matchedLines >= $HighestMatchingLines Then
                    $HighestMatchingLines = $matchedLines

    ;Best guess of location
;~                  $foundAtTop = $fline[$i] + $i - $BMP2Height
                    $foundAtTop = Int($iPos / $BMP1lineWidth);+ $i - $BMP2Height
                    $bestMatchLine = Int($iPos / $BMP1lineWidth)
                EndIf
                ExitLoop
            EndIf
            $matchedLines=$matchedLines + 1
            $iAbove=$iAbove+$iFuzzyDist
        WEnd

;If bMatchPossible is still true most likely we have found the bitmap
        if $bmatchPossible = True then
;~          ConsoleWrite("Could match top: " & $foundAtTop & " left: " & $foundAtLeft & " in " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;           MouseMove($foundatleft,$foundatTop)
            exitloop
        else
;~          consolewrite("i not matched " & $ipos & " " & $matchedlines & @crlf )
        EndIf

    Next

;For some debugging of time
;   if $bMatchPossible = True Then
;       ConsoleWrite("Searching took " & TimerDiff($begin) / 1000 & "  seconds " & @LF)
;   Else
;       ConsoleWrite("NOT FOUND Searching took " & TimerDiff($begin) / 1000 & "  seconds" & @LF)
;   endif

;Return an error if not found else return an array with all information
    if $bMatchPossible = False Then
        SetError(1, 0, 0)
    endif
;   return stringsplit($bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine,";")
    return $bMatchPossible & ";" & $matchedLines & ";" & $foundAtLeft & ";" & $foundAtTop & ";" & $bmp2width & ";" & $BMP2Height & ";" & $HighestMatchingLines & ";" & $bestMatchLine
EndFunc;==>findBMP

func comparePicLine($s1,$s2)
    local $sLen, $iMatched
    if $s1=$s2 Then
        return True
    Else
        $iMatched=0
        $sLen=stringlen($s1)
        for $tJ=1 to $slen
            if stringmid($s1,$tJ,1)=stringmid($s2,$tJ,1) Then
                $iMatched=$imatched+1
            Else
                if $bDebug=True Then
                ;~ START debugging
                    FileWrite (@ScriptDir & "\COMPAREDLINES.TXT" , "Debug mismatch pixel " & $tj & "=" & int($tj/4) & @crlf)
                endif
            endif
        Next
        if  ($iMatched / $sLen ) > $cMatchLinePercentage then
            return true
        Else
            return false
        endif
    EndIf
endfunc

Func GetImage($BMPFile, byref $BMPDataStart, byref $Width, byRef $Height, byref $Stride, $imgBytes=3)
local $Scan0, $pixelData, $hbScreen, $pBitmap, $pBitmapCap, $handle

; Load the bitmap to search in
    If $BMPFile="SCREEN" Then
        $hbScreen=_ScreenCapture_Capture("",0,0,-1,-1,False)
        $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
    Else
;try to get a handle
        $handle = WinGetHandle($BMPFile)
        If @error Then
;Assume its an unknown handle so correct filename should be given
            $pBitmap = _GDIPlus_BitmapCreateFromFile($BMPFile)
        Else
            $hbScreen=_ScreenCapture_CaptureWnd("",$handle,0,0,-1,-1,False)
            $pBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hbScreen); returns memory bitmap
        EndIf
    EndIf

;Get $tagGDIPBITMAPDATA structure
;~  ConsoleWrite("Bitmap Width:    " & _GDIPlus_ImageGetWidth($pBitmap) & @CRLF )
;~  ConsoleWrite("Bitmap Height:      " & _GDIPlus_ImageGetHeight($pBitmap) & @CRLF)

;~  24 bits (3 bytes) or 16 bits (2 bytes) comparison
    if ($imgBytes=2) then
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF16RGB555)
;~      $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF32ARGB)
    Else
        $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF24RGB)
;~      $BitmapData= _GDIPlus_BitmapLockBits($pBitmap, 0, 0, _GDIPlus_ImageGetWidth($pBitmap), _GDIPlus_ImageGetHeight($pBitmap), $GDIP_ILMREAD, $GDIP_PXF32ARGB)
    endIf

    If @ERROR Then MsgBox(0,"","Error locking region " & @error)

    $Stride = DllStructGetData($BitmapData, "Stride");Stride - Offset, in bytes, between consecutive scan lines of the bitmap. If the stride is positive, the bitmap is top-down. If the stride is negative, the bitmap is bottom-up.
    $Width = DllStructGetData($BitmapData, "Width");Image width - Number of pixels in one scan line of the bitmap.
    $Height = DllStructGetData($BitmapData, "Height");Image height - Number of scan lines in the bitmap.
    $PixelFormat = DllStructGetData($BitmapData, "PixelFormat");Pixel format - Integer that specifies the pixel format of the bitmap
    $Scan0 = DllStructGetData($BitmapData, "Scan0");Scan0 - Pointer to the first (index 0) scan line of the bitmap.

    $pixelData = DllStructCreate("ubyte lData[" & (abs($Stride) * $Height-1) & "]", $Scan0)
    $BMPDataStart = $BMPDataStart & DllStructGetData($pixeldata,"lData")

    _GDIPlus_BitmapUnlockBits($pBitmap, $BitmapData)
    _GDIPlus_ImageDispose ($pBitmap)
    _WinAPI_DeleteObject ($pBitmap)

EndFunc;==>GetImage
; Draw rectangle on screen.
Func _UIA_DrawRect($tLeft, $tRight, $tTop, $tBottom, $color = 0xFF, $PenWidth = 4)
    Local $hDC, $hPen, $obj_orig, $x1, $x2, $y1, $y2
    $x1 = $tLeft
    $x2 = $tRight
    $y1 = $tTop
    $y2 = $tBottom
    $hDC = _WinAPI_GetWindowDC(0) ; DC of entire screen (desktop)
    $hPen = _WinAPI_CreatePen($PS_SOLID, $PenWidth, $color)
    $obj_orig = _WinAPI_SelectObject($hDC, $hPen)

    _WinAPI_DrawLine($hDC, $x1, $y1, $x2, $y1) ; horizontal to right
    _WinAPI_DrawLine($hDC, $x2, $y1, $x2, $y2) ; vertical down on right
    _WinAPI_DrawLine($hDC, $x2, $y2, $x1, $y2) ; horizontal to left right
    _WinAPI_DrawLine($hDC, $x1, $y2, $x1, $y1) ; vertical up on left

    ; clear resources
    _WinAPI_SelectObject($hDC, $obj_orig)
    _WinAPI_DeleteObject($hPen)
    _WinAPI_ReleaseDC(0, $hDC)
EndFunc   ;==>_UIA_DrawRect
#EndRegion

 

Edit: Speed and partial match enhancements. Search is within a split second especially when partial match is turned on

Edit2: Cleaned up code in a nicer way, returns best match results even if find is false

Edit 24 march 2008: Becomes dependent on beta v3.2.11.5 (beta)

;$iPos = StringInStr($BMP1Data, $searchFor)

$iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos+1)

Edit 13th june 2008 updated for new autoit version

Edit 31st august 2008 added some reference links

Edit 04 sept 2008 completely rewritten using GDI plus functions and speedenhancements by using different 24 bits and 16 bits pictures

Edit 23 nov 2008 small additional check on bmp2height beeing 0

Edit 1st dec 2008 fixed top/left, added [active] find on active screen, work with window special description

Edit 20st feb 2009 fixed Stringinstr to use additional parameters, now matchin b/w pictures better

$iPos = StringInStr($BMP1Data, $searchFor, 2, 1, $iPos)

Edit jan 5th 2013 full updated all working under Windows 7 / 64 bits (not tested under windows XP but no reason to be broken)

edit jan 20st 2018 checked with windows 10, quickfix as calculator is not recognized by AU3Inf, basic algorithm still works fine

see also

works with BMP, PNG and TIFF

added threshold for small bits off on a single line (unclear why GDI writes those pixels/shade variation)

speed I assume is no issue for nowadays CPU power

pics.zip

Edited January 20, 2018 by junkew
check with W10 and 3.3.14.2

[sbgwb0769]

very very thank you!hehe~~~~~~~~~~~

[Armand]

ummzz....

not quit accurate working.... try finding pic A inside pic B in these:

A.

B.

Could be very useful if it could work on such pictures...

[junkew]

Searching the picture as suggested just works on my machine. Remove the 2 right vertical whitespaces of your picture to search.

It seems that the 2 pictures are slightly different.

Finding the other characters I would suggest to only match on the bottom 8 lines as it looks like the cats are not there and its unique enough to recognize what you want.

Other suggestion I can give take a look at the hexadecimal values of the picture and make the line matching a little more advanced.

As your pictures are black and white it should not be very hard. Study the code and you can make it work to not only match a rectangle but search for boundary.

Offcourse its hard to get the cats out of it and if it becomes coloured it will become even harder to match.

Edit:

* Based on the seven sample I modified the UDF in such a way that if result is false you still get the best top/left location on a best guess basis.

* Works only with BMP files not with PNG to see it working you should save the PNG as seven.BMP

Something like

Global Const $Bitmap4Filename = @TempDir & "\SEVEN.bmp"
$start = TimerInit()
$tResult=findBMP($Bitmap1Filename,$Bitmap4Filename,TRUE)
ConsoleWrite($tResult &  "** Full seven totalmatch time elapsed: " & TimerDiff($start) / 1000 & "  seconds" & @LF)

Edited March 21, 2008 by junkew

[Armand]

@junkew

how could there be any difference if picture A is in fact a slice of B ?!

i am almost definite that there is a mistake in the comparing algorithm...

anyhow, just trying to help you improving you UDF

thanks in advance!

[Impulse08]

@junkew

how could there be any difference if picture A is in fact a slice of B ?!

i am almost definite that there is a mistake in the comparing algorithm...

anyhow, just trying to help you improving you UDF

thanks in advance!

I can see what he's saying. Just by do a copy image and pasting it into MSPaint, I see that your first 7 cuts off part of the cat on the right side. So the algorithm to find the whole image A is failing because there is some extra whitespace in image A on the right versus image B.

Try it out for yourself from your post. Right-click on the image and copy image. Then paste into MSPaint. Then do the same for image A. You'll see that image A is slightly larger than the 7 in B

[Oldschool]

Well done!! This is awesome work!!

I've been trying to come up with a way, but you beat me to it.

[Armand]

does anyone has anyone have any example with pre'made .bmp files ?!

[junkew]

does anyone has anyone have any example with pre'made .bmp files ?!

What do you mean?

* http://www.autoitscript.com/forum/index.ph...=pixelchecksum) gives an idea how to store an BMP.

If you remove all code between ; ** Example start ** and ; ** Example end ** and put in something like

$tResult=findBMP("C:\TEMP\BITMAP1.BMP", "C:\TEMP\BITMAP2.BMP",TRUE)

It should work although not thoroughly tested with BMP not created by AutoIt.

Known Issues

* Rounding rectangles on window can cause mismatch as backgroundcolor on corner can be different compared to moment when screenshot was taken.

[Armand]

well.. i meant that you'll try and do something more like what i've done...

post pic A.

[PICTURE]

post pic B.

[PICTURE]

and the script that handles both of these.

Cause i'd really really really like to use this UDF for non-autoit pictures...

[junkew]

I tried and actually the algorithm seems to be misbehaving on black/white pictures. (fixed)

The less color in the picture(s) the more risk it will not properly recognize (Even if saved as 24 bits BMP)

I will rewrite the algoritm as soon as I have new beta installed as to fix/enhance the algorithm I am in need of stringinstr function where I can give startposition or I have to put the bitmaps in an array for each line of data which will slow down performance (Not sure how long it takes to load a desktop of 1280*1024 into an array of 1280 lines and with an array I need to do more comparisons as done in current algorithm).

* http://www.autoitscript.com/forum/index.ph...mp;#entry496554

Edit 24 march 2008: Modified first posting and added attachments and samples on how to handle

Edited March 24, 2008 by junkew

[junkone]

i get error AVector: [] Out of bounds. in C:\Program Files\AutoIt3\Include\GDIPlus.au3 (2688) : ==> Subscript used with non-Array variable.:

Return $aResult[0] <> 0

Return $aResult^ ERROR

>Exit code: 1 Time: 275.393

I am not sure what this means. appreiate any help. I am running it on windows 2000 prof

[junkew]

First try if a script like below works

#include <ScreenCapture.au3>
_ScreenCapture_Capture(@tempdir & "\screen.bmp",0,0,-1,-1,false)

If this gives the same problem you most likely have not GDIPLUS.DLL on the right path location

[junkone]

bingo. gdiplus is not installed on win2k by default. i installed it and it works beautifully. however i havea smaill issue. i used the app to take a screenshot of my app. then i am using the same to take subsequent screenshots to compare and see if it is the same. however this is what i find.

$tResult=findBMP($BitmapOriginalFile,$BitmapCurrentFile,False)

it always returns "False;1;0;0;808;627;2;22"

i would have imagined that it would return true as both the images are supposed to be taken the same way without any change. strange is the way it works.

[Toady]

this is neat, good work!

[junkew]

bingo. gdiplus is not installed on win2k by default. i installed it and it works beautifully. however i havea smaill issue. i used the app to take a screenshot of my app. then i am using the same to take subsequent screenshots to compare and see if it is the same. however this is what i find.

$tResult=findBMP($BitmapOriginalFile,$BitmapCurrentFile,False)

it always returns "False;1;0;0;808;627;2;22"

i would have imagined that it would return true as both the images are supposed to be taken the same way without any change. strange is the way it works.

Try code like below. What happens if you compare the same screen bitmap to each other. On my system it works and returns True (takes about 30 seconds to match a full screen in all details).

Remember that when anything changes on the screen you have a mismatch. Moving a mouse can already change highlighted pixels of a button,

taskbar at the bottom, animated gifs, characters typed somewhere.

Global Const $Bitmap1Filename = @TempDir & "\FULLSCREEN.bmp"
; Capture full screen
$begin = TimerInit()
_ScreenCapture_Capture($Bitmap1Filename,0,0,-1,-1,false)
ConsoleWrite("Saving full screen took " & TimerDiff($begin) & " milliseconds " & @LF)

;Do the actual find
;Find the full screen itself
$start = TimerInit()
$tResult=findBMP($Bitmap1Filename,$Bitmap1Filename,TRUE)
ConsoleWrite($tResult &  "** Full screen totalmatch time elapsed: " & TimerDiff($start) / 1000 & "  seconds" & @LF)

[junkone]

i do not capture the full screen. i onlyh capture the specific window. obviously there maybe changes if i did fullscreen because of the clock at the bottom of my window.

$targetHWND = WinGetHandle("Presentation Server")

$begin = TimerInit()

winactivate($targetHWND)

_ScreenCapture_CaptureWnd($BitmapCurrentFile, $targetHWND)

ConsoleWrite("Saving NR window " & TimerDiff($begin) & " milliseconds " & @LF)

$WshShell.LogEvent ($INFORMATION, "Saved NR L window ")

then i compare it with a copy of the window captured using teh same method previously

$tResult=findBMP($BitmapOriginalFile,$BitmapCurrentFile,False)

it alwys returns false with the same difference False;1;0;0;808;627;2;22"

[junkew]

i do not capture the full screen

...

it alwys returns false with the same difference False;1;0;0;808;627;2;22"

1. Try to do the same for calculator screen take twice a calculator snapshot. On my system it returns correct result

2. Attach your 2 bitmapfiles to the post I can check then if the algorithm has an issue.

Actually the result you get back means that there actually is only 1 matching line. To me it suggest(s) that your picture is off by one line or one pixel column and the second screenshot is not fully equal to the previous one taken.

Is the title on your window exactly the same (no spaces added etc.)?

[Armand]

@junkew

any luck there with the external monochrome bitmaps algorithm ?

[junkew]

@junkew

any luck there with the external monochrome bitmaps algorithm ?

I modified the first post and included the samples you gave to work with.

You can work with black and white pictures as long as you store them as 24 bits bitmap.

With monochrome bitmaps the algorithm will only work if you align on 8 bits (which then becomes a byte and then it will work as algorithm is based on byte boundaries) but still its tricky as the picture you are searching in should also be on an 8 bits boundary.

The "problem" with monochrome is that each pixel is bit encoded and as such 8 pixels are in a byte whereas in a 24 color bitmap you have 3 bytes for 1 pixel.

If you want to find a monochrome bmp on a 24 bmp colored screen I would suggest to first convert it to a 24 bits color bitmap.

To be able to do 2 color bitmap searching I have to shift each string each time 8 times and compare it 8 times with a shifted string of the picture to search in.

Imagine that you want to find 00010001 on a screen of 32 pixels width which is 4 bytes like 00000001 00010000 00000000 10010000

I then would have to shift like below

00010001

shift 1: 00000010 00100000 00000001 00100000 would not match as the matching string is over the first 2 bytes

shift 2: 00000100 01000000 00000010 01000000 would not match as the matching string is over the first 2 bytes

shift 3: 00001000 10000000 00000100 10000000 would not match as the matching string is over the first 2 bytes

shift 4: 00010001 00000000 00001001 00000000 would match

Incorporating that kind of logic is making it much more complicated then to switch to a getPixel algorithm

Edited April 1, 2008 by junkew