ImageSearch and MouseDrag in background window

[AutoBert]

Imagesearch insm't needed. The targecontrol is moved to new position. Mousedrag gives same result and in a inactive window, the mouse couldn't moved.

[SaeidN]

I saw this topic, but I think the codes are old and doesn't work anymore. Can anyone make it work please?

[SaeidN]

Why this doesn't work ?

 

#include <SendMessage.au3>
#Include <WinAPI.au3>

Global $wHandle = "Untitled - Paint"

ControlClickDrag($wHandle,"left",300,500,100,80)

Func ControlClickDrag($wHandle, $Button="left", $X1="", $Y1="", $X2="", $Y2="")
   Local $MK_LBUTTON  = 0x0001
   Local $WM_LBUTTONDOWN  = 0x0201
   Local $WM_LBUTTONUP  = 0x0202
   Local $MK_RBUTTON  = 0x0002
   Local $WM_RBUTTONDOWN  = 0x0204
   Local $WM_RBUTTONUP  = 0x0205
   Local $WM_MOUSEMOVE  = 0x0200
   Local $i = 0

   Select
 Case $Button = "left"
$Button  = $MK_LBUTTON
$ButtonDown  = $WM_LBUTTONDOWN
$ButtonUp  = $WM_LBUTTONUP
 Case $Button = "right"
$Button  = $MK_RBUTTON
$ButtonDown  = $WM_RBUTTONDOWN
$ButtonUp  = $WM_RBUTTONUP
   EndSelect

   DllCall("user32.dll", "int", "SendMessage", "hwnd", $wHandle, "int", $ButtonDown, "int", $Button, "long", _MakeLong($X1, $Y1))
   DllCall("user32.dll", "int", "SendMessage", "hwnd", $wHandle, "int", $ButtonUp, "int", $Button, "long", _MakeLong($X2, $Y2))
EndFunc

Func _MakeLong($LoWord,$HiWord)
Return BitOR($HiWord * 0x10000, BitAND($LoWord, 0xFFFF))
EndFunc

 

[SaeidN]

On 10/26/2016 at 11:55 PM, InunoTaishou said:

If I remember correctly there's an alternate ImageSearch function (in the dll) that takes a handle to a bitmap, the bitmap you can get by using CaptureScreen. Then you can use SendMessage to send

1. $WM_LBUTTONDOWN,
2. $WM_MOUSEMOVE
3. $WM_LBUTTONUP

#RequireAdmin
Func CaptureWindow($iLeft = 0, $iTop = 0, $iWidth = -1, $iHeight = -1, Const $bSaveImage = False, Const $hWnd = WinGetHandle("[Active]"))
    Local $rect_window = WinGetPos($hWnd)
    Local $hDC = _WinAPI_GetWindowDC($hWnd)
    Local $hDestDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBitmap = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hDestSv = _WinAPI_SelectObject($hDestDC, $hBitmap)
    Local $hSrcDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBmp = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hSrcSv = _WinAPI_SelectObject($hSrcDC, $hBmp)

    _WinAPI_PrintWindow($hWnd, $hSrcDC, True)

    ScreenToClient($hWnd, $iLeft, $iTop)

    If ($iWidth > 0 and $iHeight > 0) Then
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $iWidth, $iHeight, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    Else
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $rect_window[2] - 16 - $iLeft, $rect_window[3] - 38 - $iTop, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    EndIf

    _WinAPI_SelectObject($hDestDC, $hDestSv)
    _WinAPI_SelectObject($hSrcDC, $hSrcSv)
    _WinAPI_ReleaseDC($hWnd, $hDC)
    _WinAPI_DeleteDC($hDestDC)
    _WinAPI_DeleteDC($hSrcDC)
    _WinAPI_DeleteObject($hBmp)

    If ($image_search_tolerance < 50) Then $image_search_tolerance += 50
    If ($image_search_tolerance > 80) Then $image_search_tolerance = 80

    If ($bSaveImage) Then _ScreenCapture_SaveImage(@ScriptDir & "\CaptureWindow.jpg", $hBitmap, False)

    Return $hBitmap
EndFunc   ;==>CaptureWindow

DllCall("ImageSearchDll.dll", "str", "ImageSearchExt", "int", $iLeft, "int", $iTop, "int", $iRight, "int", $iBottom, "int", $iTolerance, "ptr", $hHbitmapImageToFind, "ptr", $hHbitmapImageToSearch)

Should get you started. Just put the DLLCall in your ImageSearch function. $hHBitmapImageToSearch is gonna be your CaptureWindow and $hHbitmapImage is a handle to a bitmap of the image to look for

What is $mergecopy here? because it's not declared.

I have one screenshot jpg file, one small image to find it in that screenshot jpg file, and I have this script. Why it's not working?

Local $hHbitmapImageToFind  = @ScriptDir & "/findthisimage.JPG"
Local $hHBitmapImageToSearch = @ScriptDir & "/screenshot.JPG"

$res = DllCall("ImageSearchDll.dll", "str", "ImageSearchExt", "int", 0, "int", 0, "int",844, "int", 352, "int", 100, "ptr", $hHbitmapImageToFind, "ptr", $hHbitmapImageToSearch)

If $res = 1 Then MsgBox(0,0,$res)

 

[Bert]

AutoBert asked this and I'm asking again - what are the name of the applications? Telling us you already written hundreds of lines of code - sorry - don't care. The names of the applications please. Image searching is a HORRIBLE way to automate. Time and time again I see people take this approach for they think it is easy when in fact it is NOT.

When you automate an application by hooking into the controls - the window can be in the background or even minimized. You can also depending on the application do things by command line and that makes it even more stable. I've been here working with AutoIt and automating things since 2005 so I know what I'm talking about. Just list out the names of the applications and we can help you. Most likely we have the code already written and we simply need to put the pieces together. You will be surprised how easy this can be for you.

If you insist on image searching - understand you won't get much help at all for the reasons I'm listing above.

Ball is in your court.

[Chillaz]

Hi @SaeidN,

I read this post, whilst in search of my own answers. And I have sympathy for you regarding the unpleasant responses/tone from some people.  Don't worry, it's not just you - I have read hundreds of forum posts on this forum, and many others have suffered the same.  It's a shame really, as this has prevented me (for many years) from creating an account here, and participating with this community. Let's hope I don't regret this decision!

As you've also encountered, many of the examples provided in responses to peoples questions, simply don't work "as is" (often they're bits and pieces of code that's copied from a script which has dependencies in other scripts). Since the person posting the question clearly wont already know how the solution works, it's tough road ahead to try and understand it, without having a working example.

Regarding your main question - Yes, It's possible to ImageSearch a background window and passively control it.

Regarding your question about understanding the functions in ImageSearch - Image search should come with 2 files: ImageSearchDLL.DLL, and ImageSearch.au3. Open/Edit the AU3 script, it provides some basic information about how it works, and you can see all the functions it has (which you can call), and what parameters they take. For your purpose, the main "_imageSearch" functional is all you want.  *Make sure you get the version of ImageSearch which actually supports searching in another image instead of on screen, this is not clear anywhere, but the original version I had does not have this additional variable in the AU3 script info, nor does it work when you try to search in another Image (image in memory or on file).

I am building a framework, which when finished will be utilised the same way you would utilise _imageSearch, except it is designed to fill all the obvious missing features, such as targeting a background window for search and passively clicking. In my frame work, I have been adding notes/code which will avoid confusing issues which many people struggle with, and are given the wrong answer most times (I know I try every wrong answer until I sorted it for myself).

So far all these features work, except for MouseDrag on background window (which I am currently working to resolve).  You've probable already noticed that ControlClick doesn't support individually; MouseDown, MouseMouse, or MouseUp, thus you cannot do a mouse drag (A simple click on the spot works fine).  I'm currently testing "_SendMessage" and directly calling the $User32 DLL itself (PostMessage / SendMessage), to send a standard LeftButton click to MSPaint but not a single one of the examples has worked for me yet.

Anyhow, shortly I will now post the scripts which I have built so far:

[Chillaz]

Firstly, this is my ImageSearch.au3

I have tried to keep this as original as I found it, to support future changes by the originator. 
(FYI: The DLL is a packaged version of the feature which is native in AutoHotKey - AHK, so that we can use it over here at AutoIT!).

*Ensure that your script, and the 2 ImageSearch files are all in the same folder, and not in any subfolders!

You're version should already have the line:

$HBMP - optional hbitmap to search in. sending 0 will search the desktop.

Note my changes near the top. The version of the ImageSearch I got (which has the extended search in another image capabilities) happens to be the 32bit (x86) version of the DLL, so ALL your scripts must run in the same architecture. This is how enforce it it to run in 32bit mode, and also a fail safe on the ImageSearch script incase called by a script which run/compiled in 64bit.

#RequireAdmin
#AutoIt3Wrapper_UseX64=n

If @AutoItX64 = 1 Then
    MsgBox(0, "", "Error: " & @CRLF & _
                  "You have Run/Compiled this as x64. " & @CRLF & _
                  "This uses a 32bit DLL. so you must Run/Compile the Script as x86" & @CRLF & _
                  "If this message still appears, try to re-install AutoIt.")
    Exit
EndIf

My full ImageSearch.au3 script:

expandcollapse popup

#include-once
; ------------------------------------------------------------------------------
;
; AutoIt Version: 3.0
; Language:       English
; Description:    Functions that assist with Image Search
;                 Require that the ImageSearchDLL.dll be loadable
;
; ------------------------------------------------------------------------------

;===============================================================================
;
; Description:      Find the position of an image on the desktop
; Syntax:           _ImageSearchArea, _ImageSearch
; Parameter(s):
;                   $findImage - the image file location or HBitmap to locate on the
;                                   desktop or in the Specified HBitmap
;                   $tolerance - 0 for no tolerance (0-255). Needed when colors of
;                                image differ from desktop. e.g GIF
;                   $resultPosition - Set where the returned x,y location of the image is.
;                                     1 for centre of image, 0 for top left of image
;                   $x $y - Return the x and y location of the image
;
;                   $HBMP - optional hbitmap to search in. sending 0 will search the desktop.
;
; Return Value(s):  On Success - Returns 1
;                   On Failure - Returns 0
;
; Note: Use _ImageSearch to search the entire desktop, _ImageSearchArea to specify
;       a desktop region to search
;
;===============================================================================


#RequireAdmin
#AutoIt3Wrapper_UseX64=n

If @AutoItX64 = 1 Then
    MsgBox(0, "", "Error: " & @CRLF & _
                  "You have Run/Compiled this as x64. " & @CRLF & _
                  "This uses a 32bit DLL. so you must Run/Compile the Script as x86" & @CRLF & _
                  "If this message still appears, try to re-install AutoIt.")
    Exit
EndIf




Func _ImageSearch($findImage, $resultPosition, ByRef $x, ByRef $y, $tolerance, $HBMP=0)
   return _ImageSearchArea($findImage,$resultPosition,0,0,@DesktopWidth,@DesktopHeight,$x,$y,$tolerance,$HBMP)
EndFunc

Func _ImageSearchArea($findImage, $resultPosition, $x1, $y1, $right, $bottom, ByRef $x, ByRef $y, $tolerance, $HBMP=0)
    ;MsgBox(0,"asd","" & $x1 & " " & $y1 & " " & $right & " " & $bottom)

    If IsString($findImage) Then
        if $tolerance>0 then $findImage = "*" & $tolerance & " " & $findImage
        If $HBMP = 0 Then
            $result = DllCall("ImageSearchDLL.dll","str","ImageSearch","int",$x1,"int",$y1,"int",$right,"int",$bottom,"str",$findImage)
        Else
            $result = DllCall("ImageSearchDLL.dll","str","ImageSearchEx","int",$x1,"int",$y1,"int",$right,"int",$bottom,"str",$findImage,"ptr",$HBMP)
        EndIf
    Else
        $result = DllCall("ImageSearchDLL.dll","str","ImageSearchExt","int",$x1,"int",$y1,"int",$right,"int",$bottom, "int",$tolerance, "ptr",$findImage,"ptr",$HBMP)
    EndIf

    ; If error exit
    if $result[0]="0" then return 0

    ; Otherwise get the x,y location of the match and the size of the image to
    ; compute the centre of search
    $array = StringSplit($result[0],"|")

   $x=Int(Number($array[2]))
   $y=Int(Number($array[3]))
   if $resultPosition=1 then
      $x=$x + Int(Number($array[4])/2)
      $y=$y + Int(Number($array[5])/2)
   endif
   return 1
EndFunc

;===============================================================================
;
; Description:      Wait for a specified number of seconds for an image to appear
;
; Syntax:           _WaitForImageSearch, _WaitForImagesSearch
; Parameter(s):
;                   $waitSecs  - seconds to try and find the image
;                   $findImage - the image to locate on the desktop
;                   $tolerance - 0 for no tolerance (0-255). Needed when colors of
;                                image differ from desktop. e.g GIF
;                   $resultPosition - Set where the returned x,y location of the image is.
;                                     1 for centre of image, 0 for top left of image
;                   $x $y - Return the x and y location of the image
;
; Return Value(s):  On Success - Returns 1
;                   On Failure - Returns 0
;
;
;===============================================================================
Func _WaitForImageSearch($findImage, $waitSecs, $resultPosition, ByRef $x, ByRef $y, $tolerance, $HBMP=0)
    $waitSecs = $waitSecs * 1000
    $startTime=TimerInit()
    While TimerDiff($startTime) < $waitSecs
        sleep(100)
        $result=_ImageSearch($findImage,$resultPosition,$x, $y,$tolerance,$HBMP)
        if $result > 0 Then
            return 1
        EndIf
    WEnd
    return 0
EndFunc

;===============================================================================
;
; Description:      Wait for a specified number of seconds for any of a set of
;                   images to appear
;
; Syntax:           _WaitForImagesSearch
; Parameter(s):
;                   $waitSecs  - seconds to try and find the image
;                   $findImage - the ARRAY of images to locate on the desktop
;                              - ARRAY[0] is set to the number of images to loop through
;                                ARRAY[1] is the first image
;                   $tolerance - 0 for no tolerance (0-255). Needed when colors of
;                                image differ from desktop. e.g GIF
;                   $resultPosition - Set where the returned x,y location of the image is.
;                                     1 for centre of image, 0 for top left of image
;                   $x $y - Return the x and y location of the image
;
; Return Value(s):  On Success - Returns the index of the successful find
;                   On Failure - Returns 0
;
;
;===============================================================================
Func _WaitForImagesSearch($findImage, $waitSecs, $resultPosition, ByRef $x, ByRef $y, $tolerance, $HBMP=0)
    $waitSecs = $waitSecs * 1000
    $startTime=TimerInit()
    While TimerDiff($startTime) < $waitSecs
        for $i = 1 to $findImage[0]
            sleep(100)
            $result=_ImageSearch($findImage[$i],$resultPosition,$x, $y,$tolerance,$HBMP)
            if $result > 0 Then
                return $i
            EndIf
        Next
    WEnd
    return 0
EndFunc

 

[Chillaz]

Here's the 2 ImageSearch files:

ImageSearch.au3

ImageSearchDLL.dll

[Chillaz]

Place holder #1

Going to sleep - I'll add the rest later

[Chillaz]

Place holder #2

Going to sleep - I'll add the rest later

[VandanaS]

On 11/8/2016 at 7:44 PM, Bert said:

AutoBert asked this and I'm asking again - what are the name of the applications? Telling us you already written hundreds of lines of code - sorry - don't care. The names of the applications please. Image searching is a HORRIBLE way to automate. Time and time again I see people take this approach for they think it is easy when in fact it is NOT.

When you automate an application by hooking into the controls - the window can be in the background or even minimized. You can also depending on the application do things by command line and that makes it even more stable. I've been here working with AutoIt and automating things since 2005 so I know what I'm talking about. Just list out the names of the applications and we can help you. Most likely we have the code already written and we simply need to put the pieces together. You will be surprised how easy this can be for you.

If you insist on image searching - understand you won't get much help at all for the reasons I'm listing above.

Ball is in your court.

HI Bert,

Working on imageSearch for more than a month, but imageSearch is behaving in very different way. sometimes it works and sometimes it doesn't. Working fine for large images for not for small images. I have read most of the post on the forums. Tried changing tolerance of image, resolution of the screen, tried with different screenshots of the same image. Tried Almost everything from my end. But still it is not working . Any suggestion on this.

 

 

 

 

[lenclstr746]

hi guys pls share a example  ı don't understand  (for background image search)

Edited May 26, 2017 by lenclstr746

[Jos]

Post 2 in our forums and again not making much sense. Be clear about what you want to accomplish and don't post in a thread unless it is directly related to it.

Jos

Edited May 26, 2017 by Jos

[lenclstr746]

pls share a work background image search script for we pls ı dont understand your comments :/

[Jos]

23 minutes ago, lenclstr746 said:

pls share a work background image search script for we pls ı dont understand your comments :/

You first tell us what you exactly want to do in stead of asking for code!

Jos 

[Bert]

On 3/2/2017 at 4:32 AM, VandanaS said:

HI Bert,

Working on imageSearch for more than a month, but imageSearch is behaving in very different way. sometimes it works and sometimes it doesn't. Working fine for large images for not for small images. I have read most of the post on the forums. Tried changing tolerance of image, resolution of the screen, tried with different screenshots of the same image. Tried Almost everything from my end. But still it is not working . Any suggestion on this.

Yes, one suggestion - The names of the applications you are wanting to work with please. This is the 4th time we have asked and will be the last time we asked. So far you are being quite evasive in your answer and that leads me to believe you are doing something game related that you KNOW is against forum rules. So, If you don't respond OR you do not list the applications - I will only believe you are trying to automate a game which a moderator will get involved. 

Ball is in your court.