Detect screen change (NOT FOR A GAME!)

[Atamisk]

1 hour ago, junkew said:

I think you can reach your goal with multiple solutions and imagesearch/pixelchecksum I would not start with (although it should be possible)

Alternatives

• IUIAutomation search on the forum plenty of links
  With simplespy you should get a basic code example to start with how to inspect an element.
• WebDriver
• javascript (injected thru addressbar)

  javascript:if (document.getElementsByName("q")[0].value="abc") {alert(searchEl[0].value)}

   

• pixelchecksum
•  

Those are gonna be my next steps. I want to be able to pull the information directly from the webpage code somehow so I can deploy this little script to others on my team and help pull up our answer call %'s. Makes us look better to management and I get a bit of credit for coming up with a solution that "made the numbers go up" (cuz thats all they friggen care about anyway)

from what I pasted on the first picture with the page code, anything that might look like it could contain something for what I'm talking about? I know it's a barren bunch of <div> stuff so not much to go off of but the system has to inject the change somehow into the page. 

[Atamisk]

17 hours ago, AlessandroAvolio said:

Open Scite, type the function name, set the cursor on the word, press F1, you get definition and examples. 

An example of a program that might work  

expandcollapse popup

#include <AutoItConstants.au3>

Main()

Func Main()

    Local Const $aScreenArea[4] = [39, 355, 123, 429]   ; Get coords with AutoIt v3 Window Info tool, Mouse tab
    Local $iCheckSumZero
    Local $iCheckSumNotZero

    Local $iCheckSum

    MsgBox(64, "Collecting graphic info", "Ensure number zero is on screen, inside $aScreenArea portion before continue")

    $iCheckSumZero = PixelChecksum($aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3])

    While 1

        $iCheckSum = PixelChecksum($aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3])

        While $iCheckSum == $iCheckSumZero
            Sleep(400)
            $iCheckSum = PixelChecksum($aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3])
        WEnd

        $iCheckSumNotZero = $iCheckSum

        SoundPlay(@WindowsDir & "\media\tada.wav", $SOUND_NOWAIT)

        $iCheckSum = PixelChecksum($aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3])

        While $iCheckSum == $iCheckSumNotZero
            Sleep(400)
            $iCheckSum = PixelChecksum($aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3])
        WEnd

    WEnd

EndFunc

 

Alessandro,

I've marked this as the solution. It's working 100% after finding the correct screen area with your other script trick. Since "read the screen" is a solution for my specific issue ATM I consider that perfect for a solution to the original question. 

I believe The next step is to figure out how the page displays the number change and capture it somehow. Once I can do that I could deploy this to the rest of my coworkers to help bring our call answered %'s up even more. 

We are all T1 guys for a mid-sized MSP in Illinois. Pity us. 
 

[Atamisk]

For anyone else running into this same issue of the mouse position being off, I figured it out! Well, You guys figured it out but I found the post that had the needed info in it. So my DPI was set to 125% and as stated here
 

So I changed everything back to 100% and used the mouse position from auto IT window program and it worked! it was giving the correct position. So in case anyone runs into this problem with using mouse position, change your windows DPI back to 100%!

[Atamisk]

Also I thought of something else. There was another post here
 

About drawing a rectangle and getting a printout of those mouse coordinates. Would there be a way to mesh the two programs together to first give the user a box to highlight the specific area on their screen then whatever the result is input that into your field?
Local Const $aScreenArea[4] = 

Right now the cord that fits that number so far is 
[4220, 325, 4235, 350]
but I'm not sure what that translates to in a square size. Nor would I have any idea how to utilize the program listed above to "paste" the coordinates into the 2nd field. 

[junkew]

• Most likely it follows the win* functions
  a 4-element array containing the following information:

    $aArray[0] = X position
    $aArray[1] = Y position
    $aArray[2] = Width
    $aArray[3] = Height

• For UIA you can use elementfrompoint function (which has some limitations if other objects are in front/overlayed.
• For drawing a rectangle you could use a window with just a 1 pixel border around it, 100% transparancy
• In HTML page you can directly copy your text. Whats your goal? See in FAQ 31 some starting points but it can become difficult to get text from whatever location you select.

 

 

[Atamisk]

Main goal is to read that number change from a 0 to anything else and play a sound. The 2nd goal is to make it deployable to other techs. The initial idea was to use a screen capture because that is what I'm most familiar with (and therefor the idea to mesh to two programs together to make the screen area selectable for the program).

However, if I could pull the info directly from the webpage code somehow and deploy this to a few of the other front line techs that would be even better so it would be a seamless read. We all use the same webpage to display waiting calls. If I can find that number change in the code then the whole "select this area of the screen" becomes a moot point. 

[Atamisk]

I think I found it!!
I forgot I can dive into the page code deeper and was able to isolate this section of the code

Right now it shows a number. How it changes that number is still unknown. I'll play with it more at the end of the work day when i can disrupt less techs. But it gives me a spot to "read" the page code with and isolate the necessary number change if possible. 
Is there a faq or some prebuilt code to read the webpage changes every 2 seconds or something close that I can look at?

[junkew]

 

Something like below could get you going. Type the javascript:.... into your addressbar, be carefull when copy/pasting as then javascript: will be trimmed off



document.querySelectorAll("[myAttribute='aValue']");

javascript:alert(document.querySelector("[a-id='WaitingForConversations']"),value);

 

[Atamisk]

1 hour ago, junkew said:

javascript:alert(document.querySelector("[a-id='WaitingForConversations']"),value);

Just copy and paste that into the address bar of the website? I tried (and made sure to type in the missing javascript portion) and it didn't do anything. I tested it out on a different window and nothing there either. I'm sorry I'm a bit dense when it comes to this stuff. 

[AlessandroAvolio]

I don't know Javascript, so I can't help with such type of solution, but another way to get screen area coords, requires anyway a little bit of user input, is to highlight a starting point near the object, by using Chrome searching tool CTRL+F and searching for text label, then calculate final coords by adding some offset:

expandcollapse popup

#include <ScreenCapture.au3>

Main()

Func Main()

    Local $hTime
    Local Const $iTimeOutMilliseconds = 8000
    Local Const $iHighlightColor = 0xFF9632 ; orange
    Local $aHighlightTopLeftPosition

    Local Const $iOffsetX = 100
    Local Const $iOffsetY = 50
    Local $aScreenArea[4]

    MsgBox(64 + $MB_TOPMOST, "Highlight label with searching tool on Google Chrome", "Please highlight ""Contact Waiting"" in orange with Google Chrome searching tool (CTRL+F) before continue.")

    $hTime = TimerInit()
    $aHighlightTopLeftPosition = PixelSearch(0, 0, @DesktopWidth, @DesktopHeight, $iHighlightColor) ; search on primary display

    While IsArray($aHighlightTopLeftPosition) == 0
        Sleep(800)
        If Int(TimerDiff($hTime)) > $iTimeOutMilliseconds Then
            MsgBox(64 + $MB_TOPMOST, "Highlight not found", "Highlight ""Contact Waiting"" label not found. The program will be closed.")
            Exit
        EndIf
        $aHighlightTopLeftPosition = PixelSearch(0, 0, @DesktopWidth, @DesktopHeight, $iHighlightColor) ; search on primary display
    WEnd

    $aScreenArea[0] = $aHighlightTopLeftPosition[0]
    $aScreenArea[1] = $aHighlightTopLeftPosition[1] - $iOffsetY
    $aScreenArea[2] = $aHighlightTopLeftPosition[0] + $iOffsetX
    $aScreenArea[3] = $aHighlightTopLeftPosition[1] - 1

    Preview($aScreenArea)

    Return 0

EndFunc




Func Preview(Const $aScreenArea)

    Local Const $sZeroFileName = "zero.bmp"
    Local Const $sZeroFilePath = @ScriptDir & "\" & $sZeroFileName

    _ScreenCapture_Capture($sZeroFilePath, $aScreenArea[0], $aScreenArea[1], $aScreenArea[2], $aScreenArea[3], False)

    SplashImageOn("Zero in selected screen area", $sZeroFilePath, 400, 300, Default, @DesktopHeight * 0.10)
    Sleep(1000)
    MsgBox(64 + $MB_TOPMOST, "Confirm zero is in selected screen area", "Ensure zero is shown in the window above.")
    SplashOff()

EndFunc

 

Edited October 9, 2021 by AlessandroAvolio

[major_typo]

Hi - I've been writing a similar "PixelChecksum" based script that will detect colour change over a predetermined area (small box). All is working fine except for one thing - the change of colour is to be generated by Zoom's pen function. Aka: my program is running along side a zoom meeting window (using partial screen share). The problem: invitees can use their pen tool to colour my "box" but the PixelChecksum function doesn't acknowledge/detect the colour change which is showing on the screen. It is as if my autoit program is running on a layer beneath zoom's coloured pen strokes. IE The program works fine when using paint to colour the "box", but doesn't register a change when the screen is coloured via an external zoom invitee using his pen tool to colour my "box". Was thinking that this zoom layer may be similar to the mouse which also moves over (and we can see the mouse) but doesn't trigger the PixelChecksum - Any ideas?

[junkew]

Javascript thru addressbar

Learn the basics

• go to www.google.com
• Type in addressbar: javascript:alert(document.body.innerText);void(0);

As you then should see the text displayed the "sky is the limit" as you can inject or run small javascript pieces

Doing it that way

• - Can be challenging as you need experience
• + No need to install additional drivers/extensions
• + Allways available in any browser

 

A pixelchecksum solution is a good solution but as you had some challenges with x/y coordinates and you wanted more the direction probably is with javascript 

Pixelchecksum is probably 100 times easier then any of the solutions with javascript, webdriver, uia automation

Just to give you more complexity: https://www.autoitscript.com/autoit3/docs/functions/InetGet.htm