Trying to make the mouse follow a black line/circle

[fixitrod]

I have a need to make the mouse push down the the left button and trace a shape that goes in all directions .  If it can follow the line in a full circle I'd be set. I'm not concerned with the left mouse down, I have that figured out for later. My problem is having the cursor follow a circle in paint. It'll follow some but I think it's going toward 0,0 I'm guessing but falls off once it gets to a certain point. 

I've searched, I've tried several methods. This code below has been the simplest that works close but won't follow a full, or half circle. It just falls off. 

More detail that you need but if you've ever seen contour lines on a map, that's what I'm ultimately wanting to follow. I'll be doing it on my own images that I create by scanning the water with sonar and making lake maps. I want to put my image on google earth and turn the contour lines into paths so they have latitude longitude data. Tracing them manually is very time consuming and accuracy suffers.

Thank you for any time you spend on this. It would be very helpful. 

HotKeySet("{ESC}", "Terminate")
Const $color = 0x000000
Local $start = MouseGetPos()

While 1
    $pos = MouseGetPos()
    
    If $pos = $start Then ContinueLoop
       
    $pix = PixelSearch($pos[0] - 5, $pos[1] - 5, $pos[0] + 5, $pos[1] + 5, $color, 5)
    
    If @error = 1 Then ContinueLoop
       
    $start = $pos
    
    MouseMove($pix[0], $pix[1])
Wend

Func Terminate()
    Exit
EndFunc

 

Edited February 21, 2016 by fixitrod
fixed a mistake in code

[fixitrod]

If there is another way to make the mouse follow a black circle in paint I'm all ears. That's all I need. I searched and tried 2 other scripts but they fall off the line as well. Thanks.

[Melba23]

fixitrod,

A fun little problem - this approach seems to work nicely:

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <MsgBoxConstants.au3>

HotKeySet("{SPACE}", "_Follow")
HotKeySet("{ESC}", "_Exit")

$hGUI = GUICreate("Test", 500, 500)

$cGraphic = GUICtrlCreateGraphic(0, 0, 500, 500)
GUICtrlSetGraphic($cGraphic, $GUI_GR_ELLIPSE, 50, 50, 400, 400)
GUICtrlSetGraphic($cGraphic, $GUI_GR_RECT, 150, 150, 200, 200)
GUICtrlSetGraphic($cGraphic, $GUI_GR_PIE, 200, 300, 100, 30, 70)


GUISetState()

While 1
    Switch GUIGetMsg()
        Case $GUI_EVENT_CLOSE
            Exit
    EndSwitch
WEnd

Func _Follow()

    ; Array holding next position to check for each direction [delta-X, Delta-Y]
    Local $aDirn[8][2] = [[0, -1], _
                          [1, -1], _
                          [1, 0], _
                          [1, 1], _
                          [0, 1], _
                          [-1, 1], _
                          [-1, 0], _
                          [-1, -1]]

    ; Array to hold search directions relative to current direction
    Local $aSearch[] = [0, 1, -1, 2, -2, 3, -3]

    ; See if on line
    $aStartPos = MouseGetPos()
    $iStart_X = $aStartPos[0]
    $iStart_Y = $aStartPos[1]
    If PixelGetColor($iStart_X, $iStart_Y) = 0x000000 Then
        ; Set start position
        $iCurr_X = $iStart_X
        $iCurr_Y = $iStart_Y
    Else
        ConsoleWrite("Searching for line" & @CRLF)
        ; Search for initial point in immediately surrounding pixels
        For $iFindDirn = 0 To UBound($aSearch) - 1
            $iFind_X = $iStart_X + $aDirn[$iFindDirn][0]
            $iFind_Y = $iStart_Y + $aDirn[$iFindDirn][1]
            If PixelGetColor($iFind_X, $iFind_Y) = 0x000000 Then
                ExitLoop
            EndIf
        Next
        ; Check if found
        If $iFindDirn > UBound($aSearch) - 1 Then
            ConsoleWrite("Cannot find line!" & @CRLF & @CRLF)
            Return
        EndIf
        ; Set start position
        $iCurr_X = $iFind_X
        $iCurr_Y = $iFind_Y
        $iStart_X = $iFind_X
        $iStart_Y = $iFind_Y
    EndIf

    ConsoleWrite("Ready to go!" & @CRLF)

    ; Check for possible directions
    $iCurrDirn = -1
    For $iDirnCheck = 0 To 7
        ; Search around for another pixel
        $iFirstMove_X = $iCurr_X + $aDirn[$iDirnCheck][0]
        $iFirstMove_Y = $iCurr_Y + $aDirn[$iDirnCheck][1]
        If PixelGetColor($iFirstMove_X, $iFirstMove_Y) = 0x000000 Then
            ; Set direction
            $iCurrDirn = $iDirnCheck
            ExitLoop
        EndIf
    Next
    ; Check direction is set
    If $iCurrDirn = -1 Then
        ConsoleWrite("Cannot follow!" & @CRLF & @CRLF)
        Return
    EndIf

    While 1

        ; Check likely directions for pixel
        For $iDirnChange = 0 To UBound($aSearch) - 1
            ; Determine direction to check
            $iCheckDirn = $iCurrDirn + $aSearch[$iDirnChange]

            If $iCheckDirn < 0 Then
                $iCheckDirn = $iCheckDirn + 8
            ElseIf $iCheckDirn > 7 Then
                $iCheckDirn = $iCheckDirn - 8
            EndIf

            ; Prevent backtracking
            If Abs($iCurrDirn - $iCheckDirn) = 4 Then ContinueLoop

            ; Look for pixel in that direction
            $iCheck_X = $iCurr_X + $aDirn[$iCheckDirn][0]
            $iCheck_Y = $iCurr_Y + $aDirn[$iCheckDirn][1]

            If PixelGetColor($iCheck_X, $iCheck_Y) = 0x000000 Then
                ; Move to this pixel
                ExitLoop
            EndIf
        Next

        If $iDirnChange > UBound($aSearch) - 1 Then
            ConsoleWrite("Lost it!" & @CRLF & @CRLF)
        Else
            ; Check if back to start
            If $iCheck_X = $iStart_X And $iCheck_Y = $iStart_Y Then
                ConsoleWrite("Back to the beginning!" & @CRLF & @CRLF)
                ExitLoop
            Else
                ; Set new coords
                $iCurr_X = $iCheck_X
                $iCurr_Y = $iCheck_Y
                ; Set new direction
                $iCurrDirn = $iCheckDirn

                ; Move mouse
                MouseMove($iCurr_X, $iCurr_Y)

                Sleep(10)
            EndIf
        EndIf
    WEnd

EndFunc

Func _Exit()
    Exit
EndFunc

You need to get the cursor with 1 pixel of the line and then press {SPACE} - any further away than that and the script sulks and announces that it cannot find the line. The "follow" algorithm will easily cope with right-angle turns (as you can see with the square) but will not follow anything tighter than that as it would seriously risk doubling back and can usually cope with more acute angles (as you can see with the triangle) - although that depends on exactly how the angle is drawn at the pixel level.

Any use? if not then please post a file showing the actual lines you want to follow so I can try to tweak the code.

M23

Edit: Now copes with more than 90 degree angles (most of the time!)

Edited February 22, 2016 by Melba23

[fixitrod]

This is unreal! I will have time tomorrow to try this out. I'm shocked at how much code it took! I can't thank you enough. I will try it out and post updates. I'll also post the type of thing I'm tracing so you know what I'm using it for. Thank you so much for your time and skills! 

-fixitrod-

[Melba23]

fixitrod,

Quote

I'm shocked at how much code it took!

Interesting - I was pleasantly surprised that it took so little!

M23

[water]

Most of the lines are used for documentation and to make the code more readable

[fixitrod]

Ok, I've gotten to try this out and it's awesome! I really appreciate the time you've put into it already. I've attached an example with non-black lines. As you can see I will need to stop and start the trace because their are depth numbers in the way but that's not a problem. But, I need to trace each depth all the way around with the mouse pushed down to create a "path". Stopping and manually clicking then starting script back wont be a problem. I will then save each path as a depth number manually. 

Since you offered to tweak the code... I'm actually going to try to do these things but not sure I can. I'm trying to learn from this. Autoit is brand new to me. I found it by searching for anything that can do this project. But, here's what i think I'll need. Thank you very much!!!!!

1. Is it possible to make it work anywhere on the screen? Basically, where every the mouse position is on any window? 

2. Follow whatever color the mouse is position on at the time of clicking the space bar with a slight shade variation? (maybe pixelsearch the current position with variation of 10 identified with a variable? Not sure, trying to think it through)

3. Use the spacebar to toggle start and pause? (I have no idea right now but will research it)

4. Hold the left mouse button down while tracing? (mousedown left feature I believe)

[fixitrod]

I've got the "trace any color your on" part figured out. But, not the rest yet. I added the color variable and changed all the colors 0x000000 to $color. If there is a better way please let me know. 

; See if on line
    $aStartPos = MouseGetPos()
    $iStart_X = $aStartPos[0]
    $iStart_Y = $aStartPos[1]
   $color = PixelGetColor($iStart_X, $iStart_Y) ; I just added this line and changed all 0x000000 to $color

 

[fixitrod]

I noticed its much slower going down but I don't understand why. Could you explain?

Also, I added MouseDown("left") right above MouseMove command. I think that'll take care of the mouse being pushed while tracing. Just need to figure out 1 and 3 on the list above

[InunoTaishou]

I'm curious, why are you trying to make the mouse follow the line?

[water]

Has been described in post #1.

[fixitrod]

Of the 4 things I needed to figure out I think I got three of them figured out. Please verify my code if you don't mind. It's attached.

But, I'm not sure if this will work at all in google earth for some reason. I tested and it work on other programs by commenting out the the GUI stuff at the beginning except the while loop. This may  be the wrong way, just let me know please. 

But, I'm not sure why, but in google earth when the mouse is on the map, it doesn't seem to want to follow even if the path tool isn't turned on yet. This is where I'm going to get lost. I'm going to dig in some more tomorrow but I don't know really know what to do next. I did make a message box show me the color to make sure it was the right color... it is.

When in paint with that image of a the lake it works pretty good but I need to allow slight shade differentiation somehow. But for the most part it works great.

When in google earth the cursor is a hand all the time. If I put it on any color and press spacebar the program says "lost it" right away or "cannot follow". I don't know why.  Just to make sure it was seeing the right color I made a message box pop up with the color code. Since I was laying the lake image onto the map I wanted to make sure it was pulling the color under the image. It is not. It's pulling the right color. I even zoomed way in on the image so there were several pixels of the same shade. I've attached a sample of a section it could not follow in google earth but worked fine in paint. But, it works on other screens, just not in google earth. Any ideas? Thank you!!!!!!!!

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <MsgBoxConstants.au3>

HotKeySet("{SPACE}", "_Follow")
HotKeySet("{ESC}", "_Exit")

;GUI commented so program will work on the full screen... I think this is how to do it.
;$hGUI = GUICreate("Test", 500, 500)

;$cGraphic = GUICtrlCreateGraphic(0, 0, 500, 500)
;GUICtrlSetGraphic($cGraphic, $GUI_GR_ELLIPSE, 50, 50, 400, 400)
;GUICtrlSetGraphic($cGraphic, $GUI_GR_RECT, 150, 150, 200, 200)
;GUICtrlSetGraphic($cGraphic, $GUI_GR_PIE, 200, 300, 100, 30, 70)


;GUISetState()
;I tried to comment the this while statement but the program just exits right away. 
;I'm still learning
While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
      Exit
   EndSwitch
WEnd

Func _Follow()

    ; Array holding next position to check for each direction [delta-X, Delta-Y]
    Local $aDirn[8][2] = [[0, -1], _
                          [1, -1], _
                          [1, 0], _
                          [1, 1], _
                          [0, 1], _
                          [-1, 1], _
                          [-1, 0], _
                          [-1, -1]]

    ; Array to hold search directions relative to current direction
    Local $aSearch[] = [0, 1, -1, 2, -2, 3, -3]

    ; See if on line
    $aStartPos = MouseGetPos()
    $iStart_X = $aStartPos[0]
    $iStart_Y = $aStartPos[1]
   $color = PixelGetColor($iStart_X, $iStart_Y)
    If PixelGetColor($iStart_X, $iStart_Y) = $color Then
        ; Set start position
        $iCurr_X = $iStart_X
        $iCurr_Y = $iStart_Y
    Else
        ConsoleWrite("Searching for line" & @CRLF)
        ; Search for initial point in immediately surrounding pixels
        For $iFindDirn = 0 To UBound($aSearch) - 1
            $iFind_X = $iStart_X + $aDirn[$iFindDirn][0]
            $iFind_Y = $iStart_Y + $aDirn[$iFindDirn][1]
            If PixelGetColor($iFind_X, $iFind_Y) = $color Then
                ExitLoop
            EndIf
        Next
        ; Check if found
        If $iFindDirn > UBound($aSearch) - 1 Then
            ConsoleWrite("Cannot find line!" & @CRLF & @CRLF)
            Return
        EndIf
        ; Set start position
        $iCurr_X = $iFind_X
        $iCurr_Y = $iFind_Y
        $iStart_X = $iFind_X
        $iStart_Y = $iFind_Y
    EndIf

    ConsoleWrite("Ready to go!" & @CRLF)

    ; Check for possible directions
    $iCurrDirn = -1
    For $iDirnCheck = 0 To 7
        ; Search around for another pixel
        $iFirstMove_X = $iCurr_X + $aDirn[$iDirnCheck][0]
        $iFirstMove_Y = $iCurr_Y + $aDirn[$iDirnCheck][1]
        If PixelGetColor($iFirstMove_X, $iFirstMove_Y) = $color Then
            ; Set direction
            $iCurrDirn = $iDirnCheck
            ExitLoop
        EndIf
    Next
    ; Check direction is set
    If $iCurrDirn = -1 Then
        ConsoleWrite("Cannot follow!" & @CRLF & @CRLF)
        Return
    EndIf

    While 1

        ; Check likely directions for pixel
        For $iDirnChange = 0 To UBound($aSearch) - 1
            ; Determine direction to check
            $iCheckDirn = $iCurrDirn + $aSearch[$iDirnChange]

            If $iCheckDirn < 0 Then
                $iCheckDirn = $iCheckDirn + 8
            ElseIf $iCheckDirn > 7 Then
                $iCheckDirn = $iCheckDirn - 8
            EndIf

            ; Prevent backtracking
            If Abs($iCurrDirn - $iCheckDirn) = 4 Then ContinueLoop

            ; Look for pixel in that direction
            $iCheck_X = $iCurr_X + $aDirn[$iCheckDirn][0]
            $iCheck_Y = $iCurr_Y + $aDirn[$iCheckDirn][1]

            If PixelGetColor($iCheck_X, $iCheck_Y) = $color Then
                ; Move to this pixel
                ExitLoop
            EndIf
        Next

        If $iDirnChange > UBound($aSearch) - 1 Then
            ConsoleWrite("Lost it!" & @CRLF & @CRLF)
        Else
            ; Check if back to start
            If $iCheck_X = $iStart_X And $iCheck_Y = $iStart_Y Then
                ConsoleWrite("Back to the beginning!" & @CRLF & @CRLF)
                ExitLoop
            Else
                ; Set new coords
                $iCurr_X = $iCheck_X
                $iCurr_Y = $iCheck_Y
                ; Set new direction
                $iCurrDirn = $iCheckDirn

                ; Click left mouse button and Move mouse
                MouseDown("Left")
                MouseMove($iCurr_X, $iCurr_Y)

                Sleep(10)
            EndIf
        EndIf
    WEnd

EndFunc

Func _Exit()
    Exit
    MouseUP("Left") ;added this because I noticed the mouse was staying down after exit.
EndFunc

 

[Melba23]

fixitrod,

Quote

it works on other screens, just not in google earth. Any ideas?

The fact it works in Paint shows that the code does indeed function - so it must be something in Google Earth that prevents it and my "starter for ten" is this:

If you do not have some form of pointer as a mouse cursor (you say you have a "hand") I wonder if the point from which you are trying to start is not sufficiently close (1 pixel) to the line for the code to detect it?

M23

[Melba23]

fixitrod,

Another thought. How about getting the mouse to follow the path in Paint with the image at the correct scale and saving the coordinates relative to a fixed point into a file. Then you could read those coordinates from the file and move the mouse along the same path inside Google Earth. That would separate the 2 events and could provide a solution.

If you think this is worth pursuing, could you please post a copy of the image file you are using - then I can try and refine the code and see about how to store the coordinates so you can try to get the code to follow the same path in Google Earth.

M23

[mikell]

Melba,
I played a little with your code - pretty nice even if I'm a bit lost on the purpose of your $aSearch array 

fixitrod,
For shade variation you could use

PixelSearch($iX, $iY, $iX, $iY, $color, 10)
If not @error Then ...

; instead of

If PixelGetColor($iX, $iY) = $color Then ...

and change the HotKeySet("{SPACE}",...) to use the spacebar to pause and resume the script
My 2 cents :

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <MsgBoxConstants.au3>

HotKeySet("{SPACE}", "_Follow")
HotKeySet("{ESC}", "_Exit")

$hGUI = GUICreate("Test", 500, 500)
$cGraphic = GUICtrlCreateGraphic(0, 0, 500, 500)
GUICtrlSetGraphic($cGraphic, $GUI_GR_ELLIPSE, 50, 50, 400, 400)
GUICtrlSetGraphic($cGraphic, $GUI_GR_RECT, 150, 150, 200, 200)
GUICtrlSetGraphic($cGraphic, $GUI_GR_PIE, 200, 300, 100, 30, 70)
GUISetState()

Global $pause

While 1
    Switch GUIGetMsg()
        Case $GUI_EVENT_CLOSE
            Exit
    EndSwitch
WEnd



Func _Follow()
    Local $color = 0x000000

    ; Array holding next position to check for each direction [delta-X, Delta-Y]
    Local $aDir[8][2] = [[0, -1], _
                          [1, -1], _
                          [1, 0], _
                          [1, 1], _
                          [0, 1], _
                          [-1, 1], _
                          [-1, 0], _
                          [-1, -1]]

    ; Array to hold search directions relative to current direction
    Local $aSearch[] = [0, 1, -1, 2, -2, 3, -3]

;=====================================
; determine start point

    ; See if on line
    $aStartPos = MouseGetPos()
    $iStart_X = $aStartPos[0]
    $iStart_Y = $aStartPos[1]
    PixelSearch($iStart_X, $iStart_Y, $iStart_X, $iStart_Y, $color, 10, 1)
    If not @error Then
  ;  If PixelGetColor($iStart_X, $iStart_Y) = $color Then
        ; Set start position
        $iCurr_X = $iStart_X
        $iCurr_Y = $iStart_Y
    Else
        ConsoleWrite("Searching for line" & @CRLF)
        ; Search for initial point in immediately surrounding pixels
        For $iFindDir = 0 To UBound($aSearch) - 1
            $iFind_X = $iStart_X + $aDir[$iFindDir][0]
            $iFind_Y = $iStart_Y + $aDir[$iFindDir][1]
            PixelSearch($iFind_X, $iFind_Y, $iFind_X, $iFind_Y, $color, 10, 1)
            If not @error Then
          ;  If PixelGetColor($iFind_X, $iFind_Y) = $color Then
                ExitLoop
            EndIf
        Next
        ; Check if found
        If $iFindDir > UBound($aSearch) - 1 Then
            ConsoleWrite("Cannot find line!" & @CRLF & @CRLF)
            Return
        EndIf
        ; Set start position
        $iCurr_X = $iFind_X
        $iCurr_Y = $iFind_Y
        $iStart_X = $iFind_X
        $iStart_Y = $iFind_Y
    EndIf

    ConsoleWrite("Ready to go!" & @CRLF)
;======================================
; determine direction 

    ; Check for possible directions
    $iCurrDir = -1
    For $iDirCheck = 0 To 7
        ; Search around for another pixel
        $iFirstMove_X = $iCurr_X + $aDir[$iDirCheck][0]
        $iFirstMove_Y = $iCurr_Y + $aDir[$iDirCheck][1]
        PixelSearch($iFirstMove_X, $iFirstMove_Y, $iFirstMove_X, $iFirstMove_Y, $color, 10, 1)
        If not @error Then
   ;     If PixelGetColor($iFirstMove_X, $iFirstMove_Y) = 0x000000 Then
            ; Set direction
            $iCurrDir = $iDirCheck
            ExitLoop
        EndIf
    Next
    ; Check direction is set
    If $iCurrDir = -1 Then
        ConsoleWrite("Cannot follow!" & @CRLF & @CRLF)
        Return
    EndIf

;===================================

HotKeySet("{SPACE}", "_Pause")

   While 1

        ; Check likely directions for pixel
        For $iDirChange = 0 To UBound($aSearch) - 1

            ; Determine direction to check
            $iCheckDir = $iCurrDir + $aSearch[$iDirChange]

            If $iCheckDir < 0 Then
                $iCheckDir = $iCheckDir + 8
            ElseIf $iCheckDir > 7 Then
                $iCheckDir = $iCheckDir - 8
            EndIf

            ; Prevent backtracking
            If Abs($iCurrDir - $iCheckDir) = 4 Then ContinueLoop

            ; Look for pixel in that direction
            $iCheck_X = $iCurr_X + $aDir[$iCheckDir][0]
            $iCheck_Y = $iCurr_Y + $aDir[$iCheckDir][1]
       ;;     If PixelGetColor($iCheck_X, $iCheck_Y) = 0x000000 Then
            PixelSearch($iCheck_X, $iCheck_Y, $iCheck_X, $iCheck_Y, $color, 10, 2)
            If not @error Then
                ; Move to this pixel
                ExitLoop
            EndIf

        Next

        If $iDirChange > UBound($aSearch) - 1 Then
            ConsoleWrite("Lost it!" & @CRLF & @CRLF)
        Else
            ; Check if back to start
            If $iCheck_X = $iStart_X And $iCheck_Y = $iStart_Y Then
                ConsoleWrite("Back to the beginning!" & @CRLF & @CRLF)
                ExitLoop
            Else
                ; Set new coords
                $iCurr_X = $iCheck_X
                $iCurr_Y = $iCheck_Y
                ; Set new direction
                $iCurrDir = $iCheckDir

                ; Move mouse
                MouseMove($iCurr_X, $iCurr_Y)

              While $pause
                  Sleep(10)
              WEnd

             ;   Sleep(10)
            EndIf
        EndIf
    WEnd

HotKeySet("{SPACE}", "_Follow")
EndFunc


;=============================================
Func _Pause()
    $pause = not $pause
EndFunc

Func _Exit()
    Exit
EndFunc

 

[Melba23]

mikell,

Quote

I'm a bit lost on the purpose of your $aSearch array 

It prioritizes the direction in which to look for a new pixel on the line by giving the required offset from the current direction. $iCurrDirn is the current direction and is the index of the $aDirn array used to determine the required change in X/Y coordinates:

• Let us assume the current direction ($iCurrDirn) is 3 - that means that we last adjusted the X/Y coordinates by (+1, +1).
• The first check for the next pixel on the line uses $iCurrDirn + $aSearch[0] (3 + 0 = 3), so the direction remains 3 and we look at the pixel at (+1, +1)
• If a suitable pixel is not found, then the next check is at $iCurrDirn + $aSearch[1] (3 + 1 = 4) so the direction becomes 4 and we look at pixel (0, +1)
• And so on.

In essence we try to keep going in a straight line and then look further and further to each side if no suitable pixel is found. Clearer?

M23

 

 

[mikell]

Ahhhh ok... it's for optimization
I didn't see at once the correlation with the position of the elements in the $aDirn array
I assume that such an approach is obvious for a pilot but it is a little out of my skills 

Anyway after reading three times, yes it's clearer now, thank you 

[Melba23]

mikell,

Exactly. And if you can come up with a more elegant way to do the same sort of thing I would be delighted to hear it.

M23

[EmilyLove]

@Melba23 Wow man this is impressive code. Very very nice.

[mikell]

Melba,
Now I understand it I think that your way is pretty clever, I really don't see how to do this better so I will not even try