MouseClick doesn't seem to always be in the right place

[mitchelwb]

I've got a simple little script that is trying to step through data in a datagrid.  I have written and rewritten this thing to try and keep it running, but it just keeps coming up with values I just don't think it can get.  I've played with the timing and it seems to run a little better when slowed down, but setting my $delay higher and higher gets me less and less improvement, but starts to really drag the process to a crawl.  Removing the sleeps entirely isn't even that bad. 

Every time I send a ^C to copy a value, I verify it, and then when I'm done with it, I reset the Clipboard with a known value that will never show in my data and reverify the clipboard.  So either my reset/verification is not doing it's job or my mouse double click is not getting the coordinates updated.  Neither of these options are great because I'm running out of ideas how to make sure I get what I want.  About all I can do at this point is attempt my copy, then validate that the clipboard isn't my reset string, isn't zero length, and THEN match it to a date REGEX to see if it's the data from the correct column.  That would be triple checking something that shouldn't need to be checked once.  Scrip and output below:

 

expandcollapse popup

$desiredDate = "2/9/2015"
$yCoord = 290
$yStep = 15
$recDisplay = 9 ;number of rows to read before starting to scroll
$totalRec = 0
$PONList = ""
$delay = 250
$dcDelay = 12
$count = 0
$clipReset = "chicken"


$oCoastal = WinActivate("Find FOC")
WinWaitActive($oCoastal)

if @error Then
    msgBox(1, "winactivate error", @error)
    Exit
endif

AutoItSetOption("MouseCoordMode", 0)
AutoItSetOption("SendKeyDownDelay", 100)

ClipPut("")

While 1
    ;get date
    ;MouseMove(284, $yCoord, 1)
    MouseClick("left", 284, $yCoord, 2, $dcDelay)
    Sleep($delay)

    while 1
        MouseClick("left", 284, $yCoord, 2, $dcDelay)
        ;Sleep($delay)
        Send("^c")
        ;Sleep($delay)
        $thisDate = clipGet()

        if StringCompare($thisDate, $clipReset) <> 0 and StringLen($thisDate) > 0 Then
            resetClip($clipReset)
            ExitLoop
        EndIf
    WEnd

    ConsoleWrite($count & ": Date: " & $thisDate)

    ;check date matches what we want
    ;if not StringCompare($desiredDate, $thisDate) = 0 Then  ;with some date functions, this could be improved to a date range
    if StringRegExp($thisDate, $desiredDate) = 0 Then ;just check to see if the desired date exists inside what was copied
        ;we have reached the end of our records
        MsgBox(1, "Found " & $totalRec & " total records", $PONList)
        Exit ;or exitloop
    endif

    ;get pon
    ;MouseMove(81, $yCoord, 1)
    MouseClick("left", 81, $yCoord, 2, $dcDelay)
    Sleep($delay)

    While 1
        MouseClick("left", 81, $yCoord, 2, $dcDelay)
        ;Sleep($delay)
        Send("^c")
        ;Sleep($delay)
        $thisPON = clipGet()

        if StringCompare($thisPON, $clipReset) <> 0 and StringLen($thisPON) > 0 Then
            resetClip($clipReset)
            ExitLoop
        EndIf
    WEnd

    ConsoleWrite(" PON:" & $thisPON & @CRLF)

    ;Check if we want this record
    if StringCompare(StringMid($thisPON, 10, 1), "D") = 0 Then
        ;We want this record!
        $totalRec += 1
        $PONList = $PONList & ", " & $thisPon
    EndIf

    if $recDisplay >= 1 Then
        ;still in the first 10 records and need to move the mouseclick position down by a record
        $yCoord += $yStep
        $recDisplay -= 1
    Else
        ;reached the bottom of the data grid and need to click the down arrow in the bottom right of the grid to advance one record
        MouseClick("left", 650, 438,1)
        ;Sleep($delay)
    endIf
    ;consolewrite($recDisplay & @CRLF)
    $count += 1
Wend


func resetClip($val = "")
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
EndFunc

And here's a sample output. Everything looks great until the last row.  This was not the last row in the data and should not have stopped here, but you can see the PON is either still on the clipboard, or the mouse click did not update the coordinates and copied the MouseClick commands didn't move to the coordinates:

expandcollapse popup

0: Date: 2/9/2015  PON:173714357INN0001
1: Date: 2/9/2015  PON:173708683DID0001
2: Date: 2/9/2015  PON:173708683DID0003
3: Date: 2/9/2015  PON:173546121INN0001
4: Date: 2/9/2015  PON:173701848INN0001
5: Date: 2/9/2015  PON:173454114INN0001
6: Date: 2/9/2015  PON:173708683DID0002
7: Date: 2/9/2015  PON:173714373DID0001
8: Date: 2/9/2015  PON:173709684INN0001
9: Date: 2/9/2015  PON:173621964INN0001
10: Date: 2/9/2015  PON:173710242CHC0001
11: Date: 2/9/2015  PON:855440C    
12: Date: 2/9/2015  PON:173531625INN0001
13: Date: 2/9/2015  PON:173717963DID0001
14: Date: 2/9/2015  PON:173717560DID0001
15: Date: 2/9/2015  PON:173713978DID0001
16: Date: 2/9/2015  PON:173710221INN0001
17: Date: 2/9/2015  PON:173713002INN0001
18: Date: 2/9/2015  PON:173456352RGN0001
19: Date: 2/9/2015  PON:173712598DID0001
20: Date: 2/9/2015  PON:173709807DID0001
21: Date: 2/9/2015  PON:173616984INN0001
22: Date: 2/9/2015  PON:173646606INN0001
23: Date: 2/9/2015  PON:173649988INN0002
24: Date: 2/9/2015  PON:173712524INR0001
25: Date: 2/9/2015  PON:173710904DID0001
26: Date: 2/9/2015  PON:173717004INN0001
27: Date: 2/9/2015  PON:173712871INN0001
28: Date: 2/9/2015  PON:173426776RGN0001
29: Date: 2/9/2015  PON:173418924RGN0001
30: Date: 2/9/2015  PON:173712465INR0001
31: Date: 2/9/2015  PON:173712465INR0002
32: Date: 2/9/2015  PON:173712524INR0002
33: Date: 2/9/2015  PON:173710788INR0001
34: Date: 2/9/2015  PON:173710788INR0002
35: Date: 2/9/2015  PON:173712737DID0001
36: Date: 2/9/2015  PON:173717129INN0001
37: Date: 2/9/2015  PON:173718061DID0001
38: Date: 2/9/2015  PON:173717663INN0001
39: Date: 2/9/2015  PON:173718181DID0001
40: Date: 2/9/2015  PON:173714071DID0001
41: Date: 2/9/2015  PON:173692375INN0002
42: Date: 173692375INN0002

 

Edited October 31, 2015 by mitchelwb

[jdelaney]

If you want any sort of accuracy, you will stay away from the generic functions like MouseClick, or ClipGet.

Use, ControlGetText instead...or if you are using a browser, use a browser specific function.

As always, specifics help, tell us what you are attempting to do, and on what.

Side note...winwaitactive, with no timeout param, will never return an @‌error value.  In fact, it doesn't return any @‌error.  It will also never exit that function until a window matching is active.  Make sure to read the help file for each function to see what they do.

Anyways, you ControlClick, or if it's a browser, _IeAction with action=click.

Edited October 31, 2015 by jdelaney

[mitchelwb]

I wish this was a browser.  The data grid I'm trying to pull data from is in a custom built proprietary app that I think was built using Power Builder.  There are two of us working on this and I'm forced to access the app in Citrix (The other guy has a desktop version of it but isn't having much more success).

So if the problem is in using generic version of mouseclick and clipget, what's the alternative? 

[jdelaney]

Use the AutoIt spy tool.  Focus on the control you are manipulating, and then go to the summary tab and post that.  (delete any sensitive info)

[mitchelwb]

I'm using Citrix to access the app.  I only get the title, handle, and coordinates of the window.  the other guy using a desktop version gets a few more values when clicking on the datagrid, but nothing of use.  I know that's not a very helpful answer, but we've tried using the spy tool to find any advantage we can and there just isn't anything more than a single control handle.  There are no controls or handles inside the data grid. 

If I can get a copy of the Summary tab, I will, but I've got to find a solution ASAP and it may be sometime tomorrow at best before I can get a copy of what he's getting.  I honestly don't think it'll be of any help though.

[jdelaney]

Do keyboard commands do anything?  Can you ControlSend copy commands to grab row data or cell data?  Do arrows keys do anything?

Edited October 31, 2015 by jdelaney

[mitchelwb]

I can navigate up and down the data grid one row at a time with the up and down arrows, but I can't navigate through the data.  if I double click to select a value and press down, it moves to the next row, but does not stay focused on the column I was on.  So I have to do the double click on every cell that I want to read a value from.  That's the only way to get the cell I need selected.

Once the cell is selected, double clicking again will highlight a word portion of the value..  In both cases, this is ok.  On the date field, there is a time stamp, but I don't need it and the space between the date and timestamp means that when I double click that coordinate, I only highlight the date.  On the PON field, that's the only value and it's a solid string, so double clicking it highlights the entire PON.  Sending a ^a does not work.  I've tried using arrow keys and shiftdown and up to try and collect it, but it runs too much risk of losing focus and highlighting the whole row and causing it to scroll wildly horizontally.

I did some more testing and basically came to the conclusion that my mouseclicks aren't always getting done.  Even though I send two sets of double clicks (one to select the field, one to highlight the data) I still often don't get a new value.  I'm outputting the value of my clipboard before getting the new value and it's definitely getting cleared, so it's a matter of my mouse clicks just aren't getting placed correctly by Autoit, or that the application is not registering them (or not fast enough)

This is the most recent version that is running and actually seems fairly bulletproof, but it sure seems like I'm having to do a lot of work just to get it to do what it's supposed to do.  You may be able to sense my frustration in the code.

expandcollapse popup

Global $desiredDate = "2/9/2015"
Global $yCoord = 290
Global $yStep = 15
Global $recDisplay = 9  ;number of rows to read before starting to scroll
Global $totalRec = 0
Global $PONList = ""
Global $delay = 250
Global $dcDelay = 12
Global $count = 0
Global $clipReset = "chicken"
Global $pattern = "\d+\/\d+\/\d+"


$oCoastal = WinActivate("Find FOC")
WinWaitActive($oCoastal)

if @error Then
    msgBox(1, "winactivate error", @error)
    Exit
endif

AutoItSetOption("MouseCoordMode", 0)
AutoItSetOption("SendKeyDownDelay", 100)

ClipPut("")

While 1
    ;get date
    MouseClick("left", 284, $yCoord, 2, $dcDelay)
    sleep($delay)

    $thisDate = getMouseDCVal(284, $yCoord, $dcDelay, $pattern)

    ConsoleWrite($count & ": Date: " & $thisDate)

    ;check date matches what we want
    if StringRegExp($thisDate, $desiredDate) = 0 Then ;just check to see if the desired date exists inside what was copied
        ;we have reached the end of our records
        MsgBox(1, "Found " & $totalRec & " total records", $PONList)
        Exit ;or exitloop
    endif

    ;get pon
    MouseClick("left", 81, $yCoord, 2, $dcDelay)
    Sleep($delay)

    $thisPON = getMouseDCVal(81, $yCoord, $dcDelay, $pattern, false)

    ConsoleWrite(" PON:" & $thisPON & @CRLF)

    ;Check if we want this record
    if StringCompare(StringMid($thisPON, 10, 1), "D") = 0 Then
        ;We want this record!
        $totalRec += 1
        $PONList = $PONList & ", " & $thisPon
    EndIf

    if $recDisplay >= 1 Then
        ;still in the first 10 records and need to move the mouseclick position down by a record
        $yCoord += $yStep
        $recDisplay -= 1
    Else
        ;reached the bottom of the data grid and need to click the down arrow in the bottom right of the grid to advance one record
        MouseClick("left", 650, 438,1)
    endIf
    $count += 1
Wend


func resetClip($val = "")
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Local $damnyou = ClipGet()
    ConsoleWrite("  ~" & $damnyou & "~  ")
EndFunc

Func getMouseDCVal($xPos, $yPos, $mcDelay, $matchPattern, $equalsPattern = true)
    Local $copyVal = ""

    while 1
        MouseClick("left", $xPos, $yCoord, 2, $mcDelay)
        Send("^c")
        $copyVal = clipGet()

        $matchResult = StringRegExp($copyVal, $matchPattern)
        if not  then
            if $matchResult = 1 Then
                $matchResult = 0
            Else
                $matchResult = 1
            EndIf

        if StringCompare($copyVal, $clipReset) <> 0 and StringLen($copyVal) > 7 and $matchResult The
        endifn
            resetClip($clipReset)
            return $copyVal
        EndIf
    WEnd

EndFunc

 

[JohnOne]

func resetClip($val = "")
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Local $damnyou = ClipGet()
    ConsoleWrite("  ~" & $damnyou & "~  ")
EndFunc

That is just ridiculous.

func resetClip($val = "")
    Do
        ClipPut($val)
        Sleep(10)
    Until ClipGet() == $val
EndFunc

Should suffice.

If you are having trouble with that not doing what it is supposed to then it suggests some other app is using the clipboard

Edited November 1, 2015 by JohnOne

[jdelaney]

When you are in a cell, you can do a Control+A to select all the text, right?  If you can do that, then you can fairly accurately grab data.  The less you have to use mouseclicks, the better.  Use the Arrow keys to navigate.

[mitchelwb]

func resetClip($val = "")
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Do
        ClipPut($val)
    Until StringCompare(ClipGet(), $val) = 0
    Local $damnyou = ClipGet()
    ConsoleWrite("  ~" & $damnyou & "~  ")
EndFunc

That is just ridiculous.

func resetClip($val = "")
    Do
        ClipPut($val)
        Sleep(10)
    Until ClipGet() == $val
EndFunc

Should suffice.

If you are having trouble with that not doing what it is supposed to then it suggests some other app is using the clipboard

You're darn right it's ridiculous.  But I was at my wits end trying to make it work like I expected.  And I didn't know if it was a clipboard issue or if it was a click issue.  I knew it wasn't something else using the clipboard though because the value that kept coming up was the previous value I had copied.  I knew that block of code was just dumb, but at least I felt it was pretty safe to assume I had reset the clipboard.  And outputting the value to the console proved it.  I was definitely getting the clipboard reset.  I think the problem was with the clicks.

[mitchelwb]

When you are in a cell, you can do a Control+A to select all the text, right?  If you can do that, then you can fairly accurately grab data.  The less you have to use mouseclicks, the better.  Use the Arrow keys to navigate.

that's just it.  Ctrl-A didn't work.  a double click would set focus on the cell, but ctrl-A did not select all.  I could right click on it and Select All was an option, but even on the right click menu, there were no hot keys.  So the option was either to double click a second time to highlight, or to right click and then calculate a second location for another left click to get Select All. 

Believe me, we tried absolutely every key combo we could think of to get it to highlight and let us select fields without having to do the mouse clicks but it just wasn't gonna happen.

 

When I added the regex match to verify that we were pulling the right data, that seemed to finally make it solid.  What I think was happening was that the app UI just wasn't keeping up.  It acted like it had a bad memory leak.  After a fresh reboot, it would run pretty snappy, but within 15 minutes, it would drag.  Popping up the next windows that we needed to go in to when we came across records that needed to be inspected would get slower and slower.  To the point that they sometimes took up to 30 seconds before they would pop up.  And it would take an entire reboot, not just restarting the app to get it flowing again.  And so I think what was happening was that the script we were running was sending the double-click and the UI just wasn't updating.  Even after the second set of double clicks, if the UI still hadn't updated and the copy command was sent, it copied what had previously been highlighted.  That's why I was working so hard to clear the clipboard, I had to prove it was either the clicks or the clipboard, and I couldn't otherwise prove it was the clicks.

 

[JohnOne]

If I were you, I'd write a completely new app to do whatever it is that this lousy app is doing.

You are using it so you obviously know what it does.

Edited November 2, 2015 by JohnOne

[mitchelwb]

If I were you, I'd write a completely new app to do whatever it is that this lousy app is doing.

You are using it so you obviously know what it does.

if only that were an option. I'm working as a consultant for another company and we're trying to extract data using this terrible system because we can't get direct access to the data.  It would be much easier to do this with a SQL query.

[JohnOne]

Then is it the memory that's leaking? look at it in device manager.

Does it only do this when automated, and not when manually operating?

[mitchelwb]

we watched the task manager, but being under the gun, we only did it briefly.  The app itself didn't grow, but there was one generic process that did seem to keep growing.  It never really seemed like it grew to a critical size though. 

There was more to it than just this piece though.  In the script that I've posted, we were just looking for the records.  When we found them there was a whole series of steps that had to be done in this app, one of which included another data grid just like the one we were fighting with here.  the deeper we would get in to that process, the more it slowed down as well.  I only posted asking about this portion because the rest of the process had actual control handles we could work with except for the other data grid, but then it was just the same as this logic.

[JohnOne]

Well it all sounds like complete madness, consultant or not, I'd consult them that they're being stupid in pursuing this course.

Take a look into https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/