Get Caret Position with all input by UI Automation

[jimmy123j]

I found IME status viewer ImTip is interesting, so I try to create same function by AutoIT

And I try to use function WinGetCaretPos, but seems not support Non-standard Window Control

Search some thread said UI Automation TextPattern2 with GetCaretRange supported, so I try to use UIA_Constants to detect VS Code and Forum Search Bar

But always show not Object error, please help me understand the issue

Or if you have other good ideal for Get Caret Position, please give me your suggestion

Thanks for help, Win10 and Win11_21H2 btw

     

expandcollapse popup

#include "UIAutomatim\UIA_Constants.au3"

Opt( "MustDeclareVars", 1 )

; ====

Global $UI_Object, $Element, $Pattern, $Caret_Position

Global $Element_Handle, $Element_Focus, $Element_Value_Ava, $Element_Text_Ava

Global $Element_Old_Handle, $Element_Old_Focus

Global $Temp

; ====

$UI_Object = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation)

; ==

While 1

        $UI_Object.GetFocusedElement($Temp)
        $Element = ObjCreateInterface($Temp, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement)

        ; ==

        $Element.GetCurrentPropertyValue($UIA_NativeWindowHandlePropertyId, $Element_Handle)
        $Element.GetCurrentPropertyValue($UIA_ControlTypePropertyId, $Element_Focus)
        $Element.GetCurrentPropertyValue($UIA_IsValuePatternAvailablePropertyId, $Element_Value_Ava)
        $Element.GetCurrentPropertyValue($UIA_IsTextPatternAvailablePropertyId, $Element_Text_Ava)

        ; ====

        If $Element_Value_Ava = True And $Element_Text_Ava = True And $Element_Old_Handle <> $Element_Handle Or $Element_Old_Focus <> $Element_Focus Then

                $Element_Old_Handle = $Element_Handle
                $Element_Old_Focus = $Element_Focus

                ; ==

                If $Element_Focus = "50004" Or $Element_Focus = "50020" Then        ; $UIA_EditControlTypeId = 50004, $UIA_TextControlTypeId = 50020

                        ConsoleWrite("Focus : " & $Element_Focus & ", " & $Element_Handle & @CRLF )

                        $Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)
                        $Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2)

                        ; ==

                        If Not IsObj($Pattern) Then
                                ConsoleWrite("Pattern not Object" & @CRLF )
                                ExitLoop
                        EndIf

                        ; ==

                        $Pattern.GetCaretRange(True, $Caret_Position)

                        ConsoleWrite("Caret :" & $Caret_Position & @CRLF )

                EndIf


        EndIf

        ; ====

        Sleep(100)

WEnd

; ====

Exit

 

 

Edited May 2, 2023 by jimmy123j
Add OS Info.

[Nine]

Works for me until the GetCaretRange method.  There is a bug in the UIA_Constants.au3.

The definition should be :     "GetCaretRange hresult(bool*;ptr*);"

So the call is :   $Pattern.GetCaretRange($bool, $Caret_Position)

Now I do not know why you cannot get the pattern object.  I tested successfully on Google Chrome and Windows File Explorer (search box).

On Win10 here.

 

[jimmy123j]

@Nine Thanks for help, work on Win10, but still not object on Win11 ( can't understand why ... )

and sorry can't provide the image, cause Win10 is my company's notebook (PIP issue)

anyway, new 3 questions after object issue solved

1. I refer the thread used CompareEndpoints and got a $Caret_Position with number as 31474641, how to transfer to coordinate (x, y) data
2. everytime I re-focus same Element like Forum Search Bar the $Caret_Position (31474641, 314754678, 31474417 ... etc.) always different, is that normal ?
3. I refer AutoHotKey's script seems use the AccessibleObjectFromWindow object, is that AccessibleObjectFromWindow better for my purpose ?

sorry for foolish question, I'm not a professional programmer ... try & error to learn by myself

expandcollapse popup

#include "UIAutomatim\UIA_Constants.au3"

Opt( "MustDeclareVars", 1 )

; ====

Global $UI_Object, $Element, $Pattern, $Bool, $Caret, $Document_Range, $Caret_Position

Global $Element_Handle, $Element_Focus, $Element_Value_Ava, $Element_Text_Ava

Global $Element_Old_Handle, $Element_Old_Focus

Global $Temp

; ====

$UI_Object = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation)

; ==

While 1

        $UI_Object.GetFocusedElement($Temp)
        $Element = ObjCreateInterface($Temp, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement)

        ; ==

        $Element.GetCurrentPropertyValue($UIA_NativeWindowHandlePropertyId, $Element_Handle)
        $Element.GetCurrentPropertyValue($UIA_ControlTypePropertyId, $Element_Focus)
        $Element.GetCurrentPropertyValue($UIA_IsValuePatternAvailablePropertyId, $Element_Value_Ava)
        $Element.GetCurrentPropertyValue($UIA_IsTextPatternAvailablePropertyId, $Element_Text_Ava)

        ; ====

        If $Element_Value_Ava = True And $Element_Text_Ava = True And $Element_Old_Handle <> $Element_Handle Or $Element_Old_Focus <> $Element_Focus Then

                $Element_Old_Handle = $Element_Handle
                $Element_Old_Focus = $Element_Focus

                ; ==

                If $Element_Focus = "50004" Or $Element_Focus = "50020" Then        ; $UIA_EditControlTypeId = 50004, $UIA_TextControlTypeId = 50020

                        ConsoleWrite("Focus : " & $Element_Focus & ", " & $Element_Handle & @CRLF )

                        ; ==

                        $Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)
                        $Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2)

                        ; ==

                        If Not IsObj($Pattern) Then
                                ConsoleWrite("Pattern not Object" & @CRLF )
                                ExitLoop
                        EndIf

                        ; ==

                        $Pattern.DocumentRange($Document_Range)

                        $Pattern.GetCaretRange($Bool, $Caret).CompareEndpoints(0, $Document_Range, 0, $Caret_Position)

                        ConsoleWrite("Caret Pos :" & $Caret_Position & @CRLF )

                EndIf


        EndIf

        ; ====

        Sleep(100)

WEnd

; ====

Exit

 

Edited May 3, 2023 by jimmy123j

[Nine]

1 and 2.  GetCaretRange creates a IUIAutomationTextRange element that you actually need to convert into an object.  Once the object is created you can access its value with the different methods offered.  So yes it is normal that each call to GetCaretRange gives a different number (element).

I don't think this line is right : $Pattern.GetCaretRange($Bool, $Caret).CompareEndpoints(0, $Document_Range, 0, $Caret_Position) ; as you would need to do it in 2 steps.

3.  I don't know what is best for you, you need to test the different options to see what suits better your requirements.

 

Edited May 3, 2023 by Nine

[jimmy123j]

Thanks for replay !!

1 and 2.  GetCaretRange creates a IUIAutomationTextRange element that you actually need to convert into an object.  Once the object is created you can access its value with the different methods offered.  So yes it is normal that each call to GetCaretRange gives a different number (element).

>> But I re-focus the element with same status, so I'm confuse ...

e.g. Same Search Bar with 3 chr. "ABC" and caret at same position

1st GetCaretPos is 33781234, 2nd is 33782345, 3rd is 33742548 ... not same every time, and transfer to x, y may not the same

but my purpose is get caret position, and it should not be same ? 

I don't think this line is right : $Pattern.GetCaretRange($Bool, $Caret).CompareEndpoints(0, $Document_Range, 0, $Caret_Position) ; as you would need to do it in 2 steps.

>> this line is only successfully got value return, and I refer it by the thread (sorry I can't find now)

could you mind povide the right example for me ?

; ==== $Range not Objet error

    $Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)
    $Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2)
    $Range = ObjCreateInterface($Pattern, $sIID_IUIAutomationTextRange, $dtag_IUIAutomationTextRange)

    ; ==
    
    $Pattern.DocumentRange($Document_Range)
    $Pattern.GetCaretRange($Bool, $Caret)
    
    $Range.CompareEndpoints(0, $Document_Range, 0, $Caret_Position)

; ==== $Range not Objet error

    $Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)
    $Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2)
    $Range = ObjCreateInterface($Temp, $sIID_IUIAutomationTextRange, $dtag_IUIAutomationTextRange)

    ; ==
    
    $Pattern.DocumentRange($Document_Range)
    $Pattern.GetCaretRange($Bool, $Caret)
    
    $Range.CompareEndpoints(0, $Document_Range, 0, $Caret_Position)

; ==== $Caret_Position get Number as 33471234

    $Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)
    $Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2)

    ; ==
    
    $Pattern.DocumentRange($Document_Range)
    $Pattern.GetCaretRange($Bool, $Caret).CompareEndpoints(0, $Document_Range, 0, $Caret_Position)

 

3.  I don't know what is best for you, you need to test the different options to see what suits better your requirements.

>> understand, I'll keep trying ( try & error is hard work )

Edited May 3, 2023 by jimmy123j

[Nine]

Here the code I use.

expandcollapse popup

#include "Includes\UIA_Constants.au3"

Opt("MustDeclareVars", 1)

Sleep(5000)  ; enough time to get the focus on the right text box

Global $UI_Object, $Element, $Pattern
Global $Element_Handle, $Element_Focus, $Element_Value_Ava, $Element_Text_Ava
Global $Temp

$UI_Object = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation)
$UI_Object.GetFocusedElement($Temp)
$Element = ObjCreateInterface($Temp, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement)

$Element.GetCurrentPropertyValue($UIA_NativeWindowHandlePropertyId, $Element_Handle)
$Element.GetCurrentPropertyValue($UIA_ControlTypePropertyId, $Element_Focus)
$Element.GetCurrentPropertyValue($UIA_IsValuePatternAvailablePropertyId, $Element_Value_Ava)
$Element.GetCurrentPropertyValue($UIA_IsTextPattern2AvailablePropertyId, $Element_Text_Ava)

ConsoleWrite($Element_Handle & @CRLF)
ConsoleWrite($Element_Focus & @CRLF)
ConsoleWrite($Element_Value_Ava & @CRLF)
ConsoleWrite($Element_Text_Ava & @CRLF)

$Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)

Global Const $dtag_IUIAutomationTextPattern2ex = $dtag_IUIAutomationTextPattern & _
    "RangeFromAnnotation hresult(ptr;ptr*);" & _
    "GetCaretRange hresult(bool*;ptr*);"

$Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2ex)
ConsoleWrite("isobj pattern = " & IsObj($Pattern) & @CRLF)

Local $CaretRange, $bool, $CaretPosition, $DocumentRange, $hResult, $sText, $TextRange
$hResult = $Pattern.DocumentRange($DocumentRange)
ConsoleWrite($hResult & "/" & $DocumentRange & @CRLF)
$hResult = $Pattern.GetCaretRange($bool, $CaretRange)
ConsoleWrite($hResult & "/" & $CaretRange & @CRLF)

$TextRange = ObjCreateInterface($CaretRange, $sIID_IUIAutomationTextRange, $dtag_IUIAutomationTextRange)
ConsoleWrite("isobj textRange = " & IsObj($TextRange) & @CRLF)
$hResult = $TextRange.CompareEndpoints(0, $DocumentRange, 0, $CaretPosition)
ConsoleWrite($hResult & "/" & $CaretPosition & @CRLF)

$TextRange = ObjCreateInterface($DocumentRange, $sIID_IUIAutomationTextRange, $dtag_IUIAutomationTextRange)
ConsoleWrite("isobj textCaret = " & IsObj($TextRange) & @CRLF)
$hResult = $TextRange.GetText(100, $sText)
ConsoleWrite($hResult & "/" & $sText & @CRLF)

Code works but CompareEndPoints only gives 0 if caret at beginning of the text box or 1 if caret is after beginning.  It does not tell the number of characters.

[jimmy123j]

@Nine Thanks for example, and yes, only return 0 / 1 by CompareEndPoints

I try to use AccessibleObjectFromWindow object and return same as WinGetCaretPos function

unfortunately, I have no ideal how to complete get Caret Position script, so sorry to waste your time

maybe someone can give me other suggestion

even script stuck now, but I'm so happy to learn about to use some funtions of UIAutomation / Accessible Object

many thanks your help, have a nice day

btw, I try the script today, and confirmed that TextPattern2Id not work on Win11

 

[Nine]

Just thought of something, you could hack it with this procedure :

#include "Includes\UIA_Constants.au3"

Opt("MustDeclareVars", 1)

Sleep(5000)  ; enough time to get the focus on the right text box

Global $UI_Object, $Element, $Pattern, $Temp

$UI_Object = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation)
$UI_Object.GetFocusedElement($Temp)
$Element = ObjCreateInterface($Temp, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement)
$Element.GetCurrentPattern($UIA_TextPattern2Id, $Temp)

Global Const $dtag_IUIAutomationTextPattern2ex = $dtag_IUIAutomationTextPattern & _
    "RangeFromAnnotation hresult(ptr;ptr*);" & _
    "GetCaretRange hresult(bool*;ptr*);"

$Pattern = ObjCreateInterface($Temp, $sIID_IUIAutomationTextPattern2, $dtag_IUIAutomationTextPattern2ex)
ConsoleWrite("isobj pattern = " & IsObj($Pattern) & @CRLF)

Local $DocumentRange, $hResult, $sText, $TextRange
$hResult = $Pattern.DocumentRange($DocumentRange)
ConsoleWrite($hResult & "/" & $DocumentRange & @CRLF)
$TextRange = ObjCreateInterface($DocumentRange, $sIID_IUIAutomationTextRange, $dtag_IUIAutomationTextRange)
ConsoleWrite("isobj textCaret = " & IsObj($TextRange) & @CRLF)
$hResult = $TextRange.GetText(100, $sText)
ConsoleWrite($hResult & "/" & $sText & @CRLF)

Send("+{END}^c{LEFT}")
ConsoleWrite(StringInStr($sText, ClipGet()) - 1 & @CRLF)

All is left for you is to check if caret is at the end of the text box...