Getting tooltip text from a Chromium based browser

[ZeroToHero]

Hello!

I have been trying to get AutoIt to read tooltip text from a Chromium based browser (Yandex broswer, to be precise). I'm 99% sure that this testing code was working on Windows 7, but after updating to Win10 something went wrong. Here's the testing code I am using.

#include <GUIToolTip.au3>

HotKeySet("{ESC}", "Close")
Example()

Func Example()
    While 1
        Sleep(100)
        _Read_Tip()
    WEnd
EndFunc

Func _Read_Tip()
    Local $aTipList = WinList("[CLASS:tooltips_class32]")
    Local $aRet
    For $i = 1 To $aTipList[0][0]
        If WinGetProcess($aTipList[$i][1]) = WinGetProcess("[ACTIVE]") Then
            $aRet = _GUIToolTip_GetCurrentTool($aTipList[$i][1])
            If $aRet[8] <> "" Then MsgBox(0, "Visible Tip", $aRet[8])
        EndIf
    Next
EndFunc 

Func Close()
    Exit
EndFunc

This code does catch some of the other programmes' tooltips though. I think the source of the problem that some tooltips are not actually windows of class "tooltips_class32" (for instance, Total Commander's tooltips). How does one intercept such tooltips? I tried to catch them with Au3Info, UIASpy without success.

I have tried tons of different browsers and they all seem to be working same way. With the exception of IE11, but for some reason it won't open the page I'm trying to automate.

[Danp2]

What's the URL so that we can take a look for ourselves?

[ZeroToHero]

Basically any URL will do. Even the basic tooltips are not caught. 

Here's an example (sorry for photo instead of printscreen, but it's acutally the way to capture the tooltip).

 

[Nine]

It only works on Win7 and IE.   Tested on other combinations, it doesn't work otherwise.  Also your code was not working, so this is the code that I used :

#include <GUIToolTip.au3>
#include <Array.au3>

HotKeySet("{ESC}", "Close")

Example()

Func Example()
  While 1
    Sleep(300)
    _Read_Tip()
  WEnd
EndFunc   ;==>Example

Func _Read_Tip()
  Local $aTipList = WinList("[CLASS:tooltips_class32]")
  Local $aRet
  For $i = 1 To $aTipList[0][0]
    If BitAND(WinGetState($aTipList[$i][1]), $WIN_STATE_VISIBLE) Then
      $aRet = _GUIToolTip_GetCurrentTool($aTipList[$i][1])
      If $aRet[0] <> 0 Then ConsoleWrite("found" & "/" & $aRet[8] & @CRLF)
    EndIf
  Next
EndFunc   ;==>_Read_Tip

Func Close()
  Exit
EndFunc   ;==>Close

 

Edited October 2, 2020 by Nine

[ZeroToHero]

So basically there is no easy way to do that on win10? I mean there should be some way to read these tooltips via OCR, but that seems like an overkill. 

I also tried looking through everything generated by WinList(), but there were no windows with class similar to browser's one. 

[Nine]

5 minutes ago, ZeroToHero said:

but there were no windows with class similar to browser's one

Yes that is what I also found.  The class only appears under Win7 AND under IE.

But I believe you could use UIAutomation to read those tooltips, although I did not test it.   Before investing too much time into it, just use UIA spy to see if it can identify the tooltip.

[Nine]

Tested it with UIA spy and it actually is finding the tooltip.  So you have a solution to get it IMO.

[Nine]

Here to start you up (couldn't help trying it) 

expandcollapse popup

#include <Constants.au3>
#include "Includes\CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Example()

Func Example()
  ; Create UI Automation object
  Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
  If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
  ConsoleWrite("$oUIAutomation OK" & @CRLF)

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement($pDesktop)
  $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
  If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
  ConsoleWrite("$oDesktop OK" & @CRLF)

  ; --- Find Chrome window ---

  Local $pCondition
  $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "Chrome_WidgetWin_1", $pCondition)
  If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
  ConsoleWrite("$pCondition OK" & @CRLF)

  Local $pChrome, $oChrome
  $oDesktop.FindFirst($TreeScope_Descendants, $pCondition, $pChrome)
  $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
  If Not IsObj($oChrome) Then Return ConsoleWrite("$oChrome ERR" & @CRLF)
  ConsoleWrite("$oChrome OK" & @CRLF)

    ; --- Find Tooltip element ---

  $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "tooltips_class32", $pCondition)
  If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
  ConsoleWrite("$pCondition OK" & @CRLF)

  Local $pTooltip, $oTooltip
  While True
    $oDesktop.FindFirst($TreeScope_Descendants, $pCondition, $pTooltip)
    $oTooltip = ObjCreateInterface($pTooltip, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If IsObj($oTooltip) Then ExitLoop
  WEnd

  ConsoleWrite ("Found tooltip" & @CRLF)

  Local $sText
  $oTooltip.GetCurrentPropertyValue($UIA_NamePropertyId, $sText)    ; $UIA_NamePropertyId extracts the text
  MsgBox ($MB_SYSTEMMODAL,"",$sText)

EndFunc

Works perfectly on Win7.  Cannot test it on Win10, I do not have Chrome. I'll see if I have time to test it on Win10/Edge later...