Screen scraping

[Nine]

Screen scraping means a way to read / write on the representation of the current screen.  You can, with this UDF, read the actual foreground of the screen or a background window.  Hidden or minimized window are not supported.  I tried to optimize the use of AutoIt as much as I could, but some actions require the performance of a DLL.

I have found multiple threads and others UDF discussing this subject but never found an optimized approach to this issue without gathering everything inside a large DLL. 

The examples I am providing show how to use a background window (even if it is in the foreground when you run it directly with Scite) or use the current screen.  Reading current screen is faster (about 20-30 ms faster) than reading a background window.

You need to first initialize the structure based on width and height.  You can reuse the structure as long as you want if the dimensions do not change.  You will need to make a new initialize if dimensions are changing.   You do not need (this way) to have the actual size of a window.  You can decide to reduce the dimensions to increase performance.

Supports x86 and x64.

The following functions are included in the UDF : 

_GetScreen_Initialize($iWidth, $iHeight)
_GetScreen_GetScreen()
_GetScreen_GetWindow($hWnd, $iFlag = 0)
_GetScreen_GetPixel($iX, $iY)
_GetScreen_SetPixel($iX, $iY, $iValue)
_GetScreen_SearchArea($iLeft, $iTop, $iRight, $iBottom, $iColor, $iStart = 1)
_GetScreen_SearchAll($iColor, $iStart = 1)
_GetScreen_PixelReplaceArea($iColorFrom, $iColorTo, $iLeft, $iTop, $iRight, $iBottom)
_GetScreen_PixelReplaceAll($iColorFrom, $iColorTo)
_GetScreen_CheckSumArea($iLeft, $iTop, $iRight, $iBottom)
_GetScreen_SaveFile($sFileName)
_GetScreen_GetBitMap()

Version 2022-04-17

* Added new optional parameter $iFlag to _GetScreen_GetWindow

• 0 = full window for Win 7 and over
• 1 = client area for Win 7 and over
• 2 = full window for Win 10 and over (this can be used with applications (like Chrome) that do not support  WM_PRINT or WM_PRINTCLIENT messages)

Version 2021-11-28

* Corrected a bug in SearchArea

Version 2021-08-30

* Fixed bugs related to areas

* Added support to repetitive searches in the same area

* Added a function to make a CheckSum of an area

* Added an additional example

Version 2021-08-24

* Removed unused structure

* Harmonized code

 

Example 1 : show various functions using the _GetScreen_GetWindow (background support)

Example 2 : show new functions using _GetScreen_GetScreen() (foreground only but faster)

 

All comments are very much welcome.

 

GetScreen.zip

Edited April 22, 2022 by Nine

[Musashi]

6 hours ago, Nine said:

Let me know if you have used it...

Yes, I have used it . However, my concentration is almost zero at this time of day. So I can only tell you that it works for me. The screenshot is a bit blurry in some areas.

Spoiler

(EDIT : Excerpt of the screenshot packed into a spoiler.)

Edited August 21, 2021 by Musashi

[Nine]

Yes it is because of the change of color (replaceAll).  It changes blue to yellow which makes it blurry 

[argumentum]

...I've saved as PNG and is better. Colors are brighter and the file is smaller.
Thanks for sharing this.

[markyrocks]

care to share the dll source?

[Nine]

4 minutes ago, markyrocks said:

care to share the dll source?

FreeBasic, pretty simple in fact...only few lines per function.

[Loc]

Thanks for this. Great 😊

[Loc]

I tried _GetScreen_GetPixel function but it returns a different color than au3info even when Aero is on or off for WIN 7

[Nine]

Not for me.  I'm also on win7.  Probably you are not setting info tool or UDF correctly.

Edited August 22, 2021 by Nine

[Loc]

I use: _GetScreen_GetWindow($hWnd) _GetScreen_GetPixel($iX, $iY) It does not return the color of the handle. seems to return the color of all on the screen

[Nine]

Same here. Make sure you are at 100% scale.

[Loc]

So the UDF can't get the color of the handle when it's overridden? I see when I save the file, it captures the handle, why can't I get the color when it's obscured 😰

[Nine]

Listen. You do not understand.  It is working perfectly fine.   You need to setup info tool for window not screen.  You need to make window scale 100%.  It is working for background and foreground.  Stop repeating it is not working and show the code and results and the setup of info tool, etc.

[Loc]

I find myself a bit annoyed. Because I re-read it with google translate, so I don't understand it very well. I have adjusted the coordinate information to windows, so I got the correct color. Since all the code I used before was taking the coordinates with the client, now that I switch to windows, I have to start all over again. Is there a way to make the UDF use coordinates from the client?

[argumentum]

you could substract the client x,y from the screen x,y ( at 0,0 usually ) and get those ? Or add the client X,Y to the screen ?
There is code in the help file that has the x,y example  ( GUICtrlCreateContextMenu > Func ClientToScreen($hWnd, ByRef $x, ByRef $y) )

Edited August 23, 2021 by argumentum
added link

[Nine]

3 hours ago, Loc said:

so I got the correct color

Glad you made it work.

3 hours ago, Loc said:

Is there a way to make the UDF use coordinates from the client?

Yes, but I will not, it would make my UDF slower.  This is out of question. 

However, here the way you can calculate the delta of Window and Client Area :

#include <StructureConstants.au3>
#include <Constants.au3>
#include <WinAPIConv.au3>
#include "..\GetScreen\GetScreen.au3"

Local $hWnd = WinGetHandle("[CLASS:SciTEWindow]")

; this calculates the screen position of the window
Local $aWin = WinGetPos($hWnd)
If @error Then Exit MsgBox($MB_SYSTEMMODAL, "", "Window not found")
Local $Width = $aWin[2], $Height = $aWin[3]

; this calculates the screen position of the client area of the window
Local $tPoint = DllStructCreate($tagPOINT)
$tPoint.X = 0
$tPoint.Y = 0
_WinAPI_ClientToScreen($hWnd, $tPoint)
Local $xDiff = $tPoint.X - $aWin[0], $yDiff = $tPoint.Y - $aWin[1]

MsgBox($MB_SYSTEMMODAL, "", "Diff from Window to Client = " & $xDiff & "/" & $yDiff)

_GetScreen_Initialize($Width, $Height)
_GetScreen_GetWindow($hWnd)

Local $color = _GetScreen_GetPixel(52+$xDiff, 122+$yDiff) ; 52 and 122 are client coordinates
MsgBox($MB_SYSTEMMODAL, "", Hex($color, 6))

You just need to calculate it once (at the beginning of the script) and reuse the deltas for each call to get pixel...

Edited August 23, 2021 by Nine

[supersonic]

Nice share!

Any reservations about the publication of the FreeBasic code?

That would be awesome...

[Nine]

New version available.  Added DLL source code to Zip file.  

[supersonic]

Thank you, sir, for providing the FreeBasic source!

 

[Nine]

New version available.