Tesseract (Screen OCR) UDF

[seangriffin]

Hi,

I have a problem with using this UDF within the loop. It seems to me, that some part of memory is not freed after the functions finishes. Runnig it in the loop causes memory outage and results in error message:

_WinAPI_CreateCompatibleBitmap: W magazynie brak miejsca dla wykonania tego polecenia

wich could be translated like:

_WinAPI_CreateCompatibleBitmap: There is no enough room in the storage to run this task

I've tried to find the place where the memory leak could be possible, but I failed. I'd be greatfull for any clue or help with solving this issue.

Some more explenation:

I have 1GB of RAM, and about 600MB of that memory is free at the script startup. After about 5 minutes and about 500

$tmp = _TesseractScreenCapture(0, "", 1, 3, 690, 168, 900, 750, 0)

function calls, I got the above error. The memory state at this time is about 400MB of free memory.

That is a concern. I recall that I found the code that converts a HBITMAP (produced by AutoIT's _ScreenCapture_CaptureWnd and _ScreenCapture_Capture functions) to a regular bitmap in another forum topic. I'd have to say, I didn't really understand these WinAPI calls for bitmap manipulation, but they did seem to work. I also haven't tried running the UDF in such a large loop before. There must be a WinAPI or GDIPlus call missing to cleanup memory somewhere, but I couldn't say where. Maybe someone with knowledge of these functions can help?

Here is the snippet of script that seems to have the leak:

; Convert the image to a bitmap

$hImage2 = _GDIPlus_BitmapCreateFromHBITMAP ($hBitmap2)

$hWnd = _WinAPI_GetDesktopWindow()

$hDC = _WinAPI_GetDC($hWnd)

$hBMP = _WinAPI_CreateCompatibleBitmap($hDC, ($pos[2] * $scale) - ($right_indent * $scale), ($pos[3] * $scale) - ($bottom_indent * $scale))

_WinAPI_ReleaseDC($hWnd, $hDC)

$hImage1 = _GDIPlus_BitmapCreateFromHBITMAP ($hBMP)

$hGraphic = _GDIPlus_ImageGetGraphicsContext($hImage1)

_GDIPLus_GraphicsDrawImageRect($hGraphic, $hImage2, 0 - ($left_indent * $scale), 0 - ($top_indent * $scale), ($pos[2] * $scale) + $left_indent, ($pos[3] * $scale) + $top_indent)

$CLSID = _GDIPlus_EncodersGetCLSID($Ext)

Edited October 3, 2009 by seangriffin

[seangriffin]

Sean (or anyone else),

I'm trying to work with the UDF, but I don't seem to be able to make it work. It seems to be broken for me at the base level. I'm testing the CaptureToTIFF() function, and it only outputs a blank .tif file.

I have the latest version of Tesseract installed in my program files directory.

Here's my script:

#include <Tesseract.au3>

$left_indent = 0
$top_indent = 0
$right_indent = 0
$bottom_indent = 0
$show_capture = 1

CaptureToTIFF("Calculator", "", "", "test01.tif", "", $left_indent, $top_indent, $right_indent, $bottom_indent)

Apologies for the long delay in replying.

Change your code to the following:

CaptureToTIFF("Calculator", "", "", "test01.tif", 1, $left_indent, $top_indent, $right_indent, $bottom_indent)

You have to specify the 4th parameter ($scale) as 1. You may also need a WinActivate and WinWait prior to calling CaptureToTIFF, just to be on the safe side.

I tried this and it works.

[seangriffin]

this is an awesome script. it works very well for me.

but, i had some probs with the scaling too.

how di i figure out these ident values from given pixel cords?

as for now im running it over and over again, while adjusting the values, to get the desired part of the screen captured using _TesseractWinCapture.

would it be possible to change the tesseract functions to use normal pixel cords, to become more user friendly?

Yes I thought someone would pick this up eventually. I'd have to admit, I find it a little unfriendly myself, but I do like the indentation concept. Reason being, it creates consistency between the 3 functions (_TesseractScreenCapture(), _TesseractWinCapture() and _TesseractControlCapture()). In this way, I tried to make it more friendly, so the user can switch between recognition at the screen, window or control level with the same parameter layout in the functions. Indentation is best suited to _TesseractControlCapture() though, where use of the function without indentation can cause unwanted pixels to be included, from areas such as the border around the control. Swapping the functions to absolute pixel coordinates I fear would make the functions less friendly.

The part of the script for identifying the area to capture to a TIFF (in CaptureToTIFF()) is not great. It works, but the indentation values and scale values that must be provided don't seem to quite make sense to me. But this is the best script I can write, given I don't understand the WinAPI and GDIPlus bitmap functions that well. I did try to rewrite it at one stage and gave up. I just couldn't seem to get the captured area to line up exactly as I would have expected. At least the functions work (as you say), albeit using unusual indent values.

I was hoping that a coordinate selection tool could be incorporated into the UDF somehow? I've seen some great tools in the forums already, that can magnify the desktop and display mouse coordinates realtime. That would take the guess work out of indentations.

Edited October 3, 2009 by seangriffin

[seangriffin]

I am confused. I have a window pos:0,0 size:800,573 and i need to read region at pos:230,360 size:90,30.

Please, could you help me with params?

$x=_TesseractScreenCapture(0,"",1,2,?,?,?,?,0)

msgbox(0,"",$x)

If you are recognising text in a window, then I would be using _TesseractWinCapture() instead of _TesseractScreenCapture(). I can't work out the indent values for _TesseractScreenCapture() unless I know your screen width and height (ie. the resolution of your desktop, such as 1024 x 768).

If your desktop resolution was 1024 x 768 pixels, then the indent parameters for _TesseractScreenCapture() would be:

230, 360, 884, 438

because:

884 = (1024 - 230 + 90)

438 = (768 - 360 + 30)

If you use _TesseractWinCapture() then the indent parameters would be:

230, 360, 660, 243

because:

660 = (800 - 230 + 90)

243 = (573 - 360 + 30)

[seangriffin]

I would like to use this UDF to do a OCR capture of a customized Tooltips. I cannot directly read the tooltip string using WinGetText()..so I'm thinking of feeding _TesseractControlCapture() with the Tooltip window info. But I only can get the following info from this tooltip. How do I put them into _TesseractControlCapture() parameters?

Window Handle: 0x002E01C6

Window Class: Afx:1da0000:8

Window Title:

Window Text:

Window Process: 1960

My General Help & Support Post

Hi VAG,

I get a feeling those window details aren't of the tooltip control, but of a window (perhaps the tooltip expressed as a window?). I haven't tried capturing a tooltip before. I'm guessing a tooltip has a window handle only, and no control handle, but I could be wrong. In which case I'd be using _TesseractWinCapture() instead.

If the Window Handle 0x002E01C6 is of the tooltip window, then use _TesseractWinCapture() with the following parameters:

_TesseractWinCapture(HWnd("0x002E01C6"), "", etc ...

this might work because my script just uses WinGetHandle($win_title, $win_text) to get the handle to the window (or tooltip perhaps, in this case).

[IndyUK]

Hi Sean,

I've been using your script in a project of mine that controls a Java application. However, ocassionally I get an error that crashes the AutoIT script. I've attached the screenshot for you to take a look at. Would you happen to know what might be causing this? I'm assuming it has something relating to with a postion of memory that is being allocated to the screen capture but isn't enough. The part of my script where the error occurs is as follows;

$iPos = 0
    While $iPos <> 1 
        $iPos = _TesseractScreenFind("Using the case", 1, 0, "", 1, 3, 35, 138, 955, 750, 0)
        Sleep($ScreenDelay)
    WEnd

I'd appreciate any help or pointers with this.

Thanks

[carlleese]

Hi

Is it possible to get this script to work as a bot on this typing game called "Typing Maniac" on face book

http://apps.facebook.com/typing-maniac/

I would be very grateful if anyone can give me any help on this.

Carl

Edited October 15, 2009 by carlleese

[tooper]

Great work!!!

I add some changes:

For use with Spanish text, English, or other languajes ...

Global $languaje = "-l spa" ; "" for use English or "-l <languaje>"

For differents routes I add this:

Global $tesseractRoute = @ProgramFilesDir & "\tesseract\tesseract.exe"   ; ... @WindowsDir & "\tesseract.exe"

For Hide de comand line window of Tesseract:

ShellExecuteWait($rutatesseract, $capture_filename & " " & $ocr_filename & " " & $languaje, "", "", @SW_HIDE)

The original code:

ShellExecuteWait(@ProgramFilesDir & "\tesseract\tesseract.exe", $capture_filename & " " & $ocr_filename)

I have a problem with Spanish text and special characters.

The problem is: the text file create by Tesseract is ASCII, but _FileReadToArray read the file as UTF8 and introduce erroneus charcters in characters witch accent

Ejem. in txt file "petición realizada" in array "petición realizada"

Any idea?? to fix...

PD: Sorry my poor english...

Edited October 22, 2009 by tooper

[tooper]

Great work!!!

I add some changes:

For use with Spanish text, English, or other languajes ...

Global $languaje = "-l spa" ; "" for use English or "-l <languaje>"

For differents routes I add this:

Global $tesseractRoute = @ProgramFilesDir & "\tesseract\tesseract.exe"   ; ... @WindowsDir & "\tesseract.exe"

For Hide de comand line window of Tesseract:

ShellExecuteWait($rutatesseract, $capture_filename & " " & $ocr_filename & " " & $languaje, "", "", @SW_HIDE)

The original code:

ShellExecuteWait(@ProgramFilesDir & "\tesseract\tesseract.exe", $capture_filename & " " & $ocr_filename)

I have a problem with Spanish text and special characters.

The problem is: the text file create by Tesseract is ASCII, but _FileReadToArray read the file as UTF8 and introduce erroneus characters in characters witch accent

Ejem. in txt file "petición realizada" in array "petición realizada"

Any idea?? to fix...

PD: Sorry my poor english...

[jays]

Okay, i'm installing it now and so far i'm not having any problems. I'd just like to say thanks for the guide.

Regards,

Jays

Edited October 22, 2009 by Jos
spam removed

[don134]

Yes I thought someone would pick this up eventually. I'd have to admit, I find it a little unfriendly myself, but I do like the indentation concept. Reason being, it creates consistency between the 3 functions (_TesseractScreenCapture(), _TesseractWinCapture() and _TesseractControlCapture()). In this way, I tried to make it more friendly, so the user can switch between recognition at the screen, window or control level with the same parameter layout in the functions. Indentation is best suited to _TesseractControlCapture() though, where use of the function without indentation can cause unwanted pixels to be included, from areas such as the border around the control. Swapping the functions to absolute pixel coordinates I fear would make the functions less friendly.

The part of the script for identifying the area to capture to a TIFF (in CaptureToTIFF()) is not great. It works, but the indentation values and scale values that must be provided don't seem to quite make sense to me. But this is the best script I can write, given I don't understand the WinAPI and GDIPlus bitmap functions that well. I did try to rewrite it at one stage and gave up. I just couldn't seem to get the captured area to line up exactly as I would have expected. At least the functions work (as you say), albeit using unusual indent values.

I was hoping that a coordinate selection tool could be incorporated into the UDF somehow? I've seen some great tools in the forums already, that can magnify the desktop and display mouse coordinates realtime. That would take the guess work out of indentations.

If anyone is interested I have done a little mod to change from indentation to real co-ords. Of course this changes the entire screen capture rule as you now need to enter some co-ords rather than nothing but you then simply enter 0,0,-1,-1 to get screen shot grab. I haven't updated any of the notes as I haven't had time but simply put instead of indents you put co-ords in that’s the only change for usage, everything is the same.

Example function I created for use with this modified UDF. Note this function increases scale if it didn't find what I was looking for and try again up to whatever you want but watch the memory usage, that’s why I stop at 5. I found this invaluable as I didn't want to always try huge scales or be stuck to one scale either I found I only needed this every now and then as most things get found at scale 2.

$VTSApplet = A variable that holds the name of window I want to check in

$ctrl_id = A variable that holds the CLASSNN of the control I want OCR to read from i.e. "Edit1"

$SearchString = A variable that holds what I am looking for i.e. "John Smith"

$left, right, top, bottom are literal co-ordinates taken from auotit wininfo tool. If using WindowFind or WindowCapture then use window co-ords, if using ControlFind or ControlCapture use control co-ords and for ScreenFind or ScreenCapture, yep you guess it use literal screen co-ords.

Func OCR($VTSApplet, $ctrl_id, $SearchString, $Left, $Top, $Right, $Bottom)
    $OCRScale = 2
    $OCR = ""
    While $OCR = "" And $OCRScale < 6
        $OCR = _TesseractControlFind($VTSApplet, "", $ctrl_id, $SearchString, 1, 0, "", 0, 0, 1, 1, $OCRScale, $Left, $Top, $Right, $Bottom, 0)
        If $OCR = "" Then
            $OCRScale = $OCRScale + 1
        EndIf
    WEnd
    Return $OCR
EndFunc   ;==>OCR

I fixed a couple of other minor bugs like it will now allow spaces in filenames for captures and such, just in case anyone wants to enter their own filenames like I did and then stop the UDF from deleting these captures for debugging etc... This was so useful as I didn't need to do a ControlCapture straight after a ControlFind failure I just had to open the files up to see what OCR read and the tiff image to ensure I got the co-ords right. Very useful just let me know if anyone wants to know how to do this.

Can anyone help with how I get post or upload this modified UDF or should I e-mail to someone to check first? I have a question in return to for these changes in that I urgently need to figure out why the kernel page memory usage isn't getting dropped for the compatible bitmap in the capture tiff section of the UDF, I noticed a few others have asked the same question in this forum and I hope someone has worked this out. I will keep trying to debug this and post if I find a solution but for now this is a real show stopper for me at the moment. I am using this as a performance monitor at the moment on a large scale UK national network i.e. once a object is clicked I constantly smash an screen area until the text I know should then appear is found by this UDF. It works extremely well but unfortunately I have to leave a big gap between checks/tests to allow the memory allocation to timeout and die. So for my function test tool I can't do this as I need to be constantly checking things but then I get the dreaded "Not enough storage to process the command" or something similar and a crash of autoit.

Thanks for reading hope I can help some of you and you can help me with this memeory issue.

Regards

Aaron

[don134]

If anyone is interested I have done a little mod to change from indentation to real co-ords. Of course this changes the entire screen capture rule as you now need to enter some co-ords rather than nothing but you then simply enter 0,0,-1,-1 to get screen shot grab. I haven't updated any of the notes as I haven't had time but simply put instead of indents you put co-ords in that’s the only change for usage, everything is the same.

Example function I created for use with this modified UDF. Note this function increases scale if it didn't find what I was looking for and try again up to whatever you want but watch the memory usage, that’s why I stop at 5. I found this invaluable as I didn't want to always try huge scales or be stuck to one scale either I found I only needed this every now and then as most things get found at scale 2.

$VTSApplet = A variable that holds the name of window I want to check in

$ctrl_id = A variable that holds the CLASSNN of the control I want OCR to read from i.e. "Edit1"

$SearchString = A variable that holds what I am looking for i.e. "John Smith"

$left, right, top, bottom are literal co-ordinates taken from auotit wininfo tool. If using WindowFind or WindowCapture then use window co-ords, if using ControlFind or ControlCapture use control co-ords and for ScreenFind or ScreenCapture, yep you guess it use literal screen co-ords.

Func OCR($VTSApplet, $ctrl_id, $SearchString, $Left, $Top, $Right, $Bottom)
    $OCRScale = 2
    $OCR = ""
    While $OCR = "" And $OCRScale < 6
        $OCR = _TesseractControlFind($VTSApplet, "", $ctrl_id, $SearchString, 1, 0, "", 0, 0, 1, 1, $OCRScale, $Left, $Top, $Right, $Bottom, 0)
        If $OCR = "" Then
            $OCRScale = $OCRScale + 1
        EndIf
    WEnd
    Return $OCR
EndFunc   ;==>OCR

I fixed a couple of other minor bugs like it will now allow spaces in filenames for captures and such, just in case anyone wants to enter their own filenames like I did and then stop the UDF from deleting these captures for debugging etc... This was so useful as I didn't need to do a ControlCapture straight after a ControlFind failure I just had to open the files up to see what OCR read and the tiff image to ensure I got the co-ords right. Very useful just let me know if anyone wants to know how to do this.

Can anyone help with how I get post or upload this modified UDF or should I e-mail to someone to check first? I have a question in return to for these changes in that I urgently need to figure out why the kernel page memory usage isn't getting dropped for the compatible bitmap in the capture tiff section of the UDF, I noticed a few others have asked the same question in this forum and I hope someone has worked this out. I will keep trying to debug this and post if I find a solution but for now this is a real show stopper for me at the moment. I am using this as a performance monitor at the moment on a large scale UK national network i.e. once a object is clicked I constantly smash an screen area until the text I know should then appear is found by this UDF. It works extremely well but unfortunately I have to leave a big gap between checks/tests to allow the memory allocation to timeout and die. So for my function test tool I can't do this as I need to be constantly checking things but then I get the dreaded "Not enough storage to process the command" or something similar and a crash of autoit.

Thanks for reading hope I can help some of you and you can help me with this memeory issue.

Regards

Aaron

Fixed Memory Issue

Two issues and one precaution added.

1. Add _WinAPI_DeleteObject($hBitmap2) to the Save TIFF function as shown below. This is the main issue you should see massive improvements by adding this.

Example:

; Save TIFF and cleanup
_GDIPlus_ImageSaveToFileEx($hImage1, $sOutImage, $CLSID, $pParams)
_GDIPlus_ImageDispose($hImage1)
_GDIPlus_ImageDispose($hImage2)
_GDIPlus_GraphicsDispose ($hGraphic)
_WinAPI_DeleteObject($hBitmap2) ;Add this line to delete bitmap from cache
_WinAPI_DeleteObject($hBMP)
_GDIPlus_Shutdown()

2. Precaution add $tSCROLLINFO = 0 to ALL the capture functions (_TesseractControlCapture, _TesseractWinCapture & _TesseractScreenCapture) just before the return, as shown below.

Example:

; Store a copy of the capture
if $last_capture.item(Number($hwnd)) = "" Then
 $last_capture.item(Number($hwnd)) = $final_ocr
EndIf
$tSCROLLINFO = 0 ;Add me just in case
Return $final_ocr

3. See the following for a small memory leak in Screen_Capture. This is fixed in next release of autoit but if you are like me and running 3.3.0.0 then you will need to change this manually as explained in this forum My link http://www.autoitscript.com/forum/index.php?showtopic=97843 read this forum entry for full details.

Hope this helps and not I have tested this by running a loop at full whack (99% cpu) and increasing the capture scale up to 9x and the back to 2x for over an hour and memory was stable and no messages about _WinAPI_CreateCompatibleBitmap: Not enough storage

Example loop to test for memory leak fix:

#include <Tesseract.au3>


Opt("WinTitleMatchMode", 3)
_TesseractTempPathSet(@MyDocumentsDir & "\Temp")

$Outlook = "Inbox - Microsoft Outlook" ;change this to whatever window you want to test against
$SearchString = "^&*$%£$" ;some text it won't find to force loop, change this to something real if will find if want to test it working rather than infinate loop to test for memory fix.
$OCRScale = 2
Local $OCR = ""
While $OCR = ""
    _TesseractTempPathSet(@WindowsDir & "\Temp")
    $OCR = _TesseractWinFind($Outlook, "", $SearchString, 1, 0, "", 1, $OCRScale, 2, 2, 50, 100, 0)
    If $OCR = "" Then
        $OCRScale = $OCRScale + 1
    Else
        MsgBox(64, "Found", $OCR, 2)
    EndIf
    If $OCRScale = 9 Then $OCRScale = 2
WEnd

Cheers

Aaron

Edited October 27, 2009 by don134

[JohnOne]

Hi

Has anyone got this to work under windows 7 at all ?

[jeanbern]

I'm trying to just do a test case using the following code

#include <Tesseract.au3>

Sleep(1000)
$Capture = _TesseractScreenCapture(0)
If IsArray($Capture) Then
    _ArrayDisplay($Capture, "$Capture")
Else
    ConsoleWrite("String $Capture = " & $Capture)
EndIf

but it's returning an empty string

I know this isn't much to work with, but what could be a common error that I might have made and/or how would I get this working for me

Thanks,

JB

[JohnOne]

I'm trying to just do a test case using the following code

#include <Tesseract.au3>

Sleep(1000)
$Capture = _TesseractScreenCapture(0)
If IsArray($Capture) Then
    _ArrayDisplay($Capture, "$Capture")
Else
    ConsoleWrite("String $Capture = " & $Capture)
EndIf

but it's returning an empty string

I know this isn't much to work with, but what could be a common error that I might have made and/or how would I get this working for me

Thanks,

JB

I dont know if its a common problem but its the same for me on win7

Xp its absolutely superb, but if I boot on same machine with win7 and try the same code on the same document I get nothing.

earlier in the thread is was noted that an error may occur when setting the last param to 1 as it used xp fax viewer

but still with it set to 0 no joy.

[jeanbern]

I dont know if its a common problem but its the same for me on win7

Xp its absolutely superb, but if I boot on same machine with win7 and try the same code on the same document I get nothing.

earlier in the thread is was noted that an error may occur when setting the last param to 1 as it used xp fax viewer

but still with it set to 0 no joy.

I'm using xp

I don't know, it might be possible that I haven't downloaded all the proper files

I'll keep tinkering but if in the meantime someone can think of anything I'd appreciate it

Thanks

[litlmike]

I'm trying to just do a test case using the following code

#include <Tesseract.au3>

Sleep(1000)
$Capture = _TesseractScreenCapture(0)
If IsArray($Capture) Then
    _ArrayDisplay($Capture, "$Capture")
Else
    ConsoleWrite("String $Capture = " & $Capture)
EndIf

but it's returning an empty string

I know this isn't much to work with, but what could be a common error that I might have made and/or how would I get this working for me

Thanks,

JB

I also have this issue. I am using Windows Server 2003. Any suggestions? Edited December 7, 2009 by litlmike

[madScientist]

How can I train Tesseract ?

[madScientist]

Doesn't seem i can edit my post so please excuse me for double-posting, just wanted to say thank you for all your hard work in providing a Tesseract implementation, saved me a lot of work. I am trying to train it for certain fonts and if i find an easy or scriptable way i will post it here.

[pabloruiz55]

Hi! I tried using it but it doesn't properly recognize word. I can't get it to recognize even some phrase written in the notepad. Shouldn't it have to do that almost right without having to train it?

Many thanks, great work!