Detect button press/hold/release on Win10 touch screen device..

[Bumperscoot]

Hi There!

 

I have an AutoIT application that needs to run on a remote tablet platform and will be used as a remote IFB Talkback/Cough/Mute "switch". To do this, a host/producer will push the T/B GUI button, and while depressed, command an audio router (remotely via IP) to make a specific priority crosspoint "sum".  This sum is released  when the button is released.
I have this working perfectly with a wired mouse, however I am stuck on making this work with touch screen input (touch screen mousedown events are on release).

I have tried both WM_TOUCH and WM_TOUCHHITTESTING, but each has it's issues..
I am able to process the dwFlags from WM_TOUCH's TouchInfo, however I can only register the Form GUI for messages. I've tried buttons, pictures, and dummy objects to no avail.

I can detect the Touch "hit" with WM_TOUCHHITTESTING, and can differentiate between unique locations, however I have no idea how to get the "down" or "hold" status..

Has anyone successfully created a touchscreen Press/Hold event in AutoIT? 
Any hints, references, or examples are greatly appreciated!

Thanks SO much in advance,

Steve

[Nine]

I cannot test it since I don't have a touch screen computer, but a fast search revealed that WM_TOUCH is obsolete and you should use WM_POINTER exclusively.  Looking at the numerous notifications, I trust it may be a better solution.

[Bumperscoot]

Hi Nine!

Thanks for the reply!
I read that too (It was the first hit in my rabbit-hole google search), however I still believe the issue is registering/handling the messages on GUI items (buttons, frames, pics, dummies, etc).  I'm not a native Win32 API guru, so it will take me some time to decipher WM_POINTER (I'm looking for some example code to pull apart and play with).

I"m still all ears for solutions!

 

Steve

[Bumperscoot]

Hi Nine!

Thanks for the reply!
I read that too (It was the first hit in my rabbit-hole google search), however I still believe the issue is registering/handling the messages on GUI items (buttons, frames, pics, dummies, etc).  I'm not a native Win32 API guru, so it will take me some time to decipher WM_POINTER (I'm looking for some example code to pull apart and play with).

I'm still all ears for solutions!

 

Steve

[Nine]

Alright, I have look into the WM_POINTER* notifications, and it is quite interesting.  Especially since you can enable mouse as a pointer.  So I am willing to make a deal with you.  I will prepare base code at the condition that you help me test the code on a touch screen machine.  Deal ?

[Bumperscoot]

Absolutely Can Do!
Only on the condition you answer my dumbass questions so I can learn this too..

[Nine]

Great.  My intention is to make an UDF of this code eventually.  So please share the code you will develop so I can integrate it.

Here a very first draft of the code, only a few functions are included, but it is a working start, let me know how it goes on touch screen.

expandcollapse popup

#include <GUIConstants.au3>
#include <WinAPIShellEx.au3>
#include "WM_POINTER.au3"

EnableMouseInPointer(True)

Global $hGUI = GUICreate("WM_POINTER")
Global $idButton = GUICtrlCreateButton("OK", 250, 350, 100, 25)
Global $hDll = DllCallbackRegister(WM_POINTER_SUB, 'lresult', 'hwnd;uint;wparam;lparam;uint_ptr;dword_ptr')

_WinAPI_SetWindowSubclass(GUICtrlGetHandle($idButton), DllCallbackGetPtr($hDll), $idButton, 0)
GUISetState()

While True
  Switch GUIGetMsg()
    Case $GUI_EVENT_CLOSE
      ExitLoop
  EndSwitch
WEnd

_WinAPI_RemoveWindowSubclass(GUICtrlGetHandle($idButton), DllCallbackGetPtr($hDll), $idButton)
DllCallbackFree($hDll)

Func WM_POINTER_SUB($hWnd, $iMsg, $wParam, $lParam, $iID, $pData)
  Local $iPointerID, $iX, $iY
  Switch $iID
    Case $idButton
      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("Button OK was pressed down with type " & $iPointerType & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
      EndIf
  EndSwitch
  Return _WinAPI_DefSubclassProc($hWnd, $iMsg, $wParam, $lParam)
EndFunc   ;==>WM_POINTER

 

WM_POINTER.au3

Edited April 20, 2022 by Nine

[Bumperscoot]

Nine,

Someone needs to promote you to Ten!
You rock.. Completely!

Here's what I found...
I created a WMPOINTER test folder with your UDF code and the test application you started..
If I understand it correctly, "_WinAPI_RemoveWindowSubclass" removes the subclass processing  from $idButton and keeps that object from eating those subclass messages.. Yes?

The code works awesome..
WM_POINTER_SUB processed the POINTERDOWN messages for the button control for both wired mouse and touch input.
I did try to process the IS_POINTER_INCONTACT_WPARAM (I received a boolean return?) to detect a continued "press & hold" or mouse-down, however it appears unpredictable. I need to play with that a bit more to fully understand what it's doing..

The icing on this cake was  POINTERUP..

I modified your function here:

Func WM_POINTER_SUB($hWnd, $iMsg, $wParam, $lParam, $iID, $pData)
  Local $iPointerID, $iX, $iY
  local $iPointerContact

  Switch $iID

      Case $idButton

      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("Button OK was pressed down with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
      EndIf

        if $iMsg = $WM_POINTERUP Then
            $iPointerID = GET_POINTERID_WPARAM($wParam)
            $iPointerType = GetPointerType($iPointerID)
            ConsoleWrite("Button OK was pressed up with type " & $iPointerType & " : " & $iPointerID &  @CRLF)
        endif


  EndSwitch

Not very original, but I'm sure you get the idea..
This works incredibly well with both mouse AND touchscreen input.
here's the console messages from a wired mouse left press:

Button OK was pressed down with type 4 : 1
Location X 1047/694
Button OK was pressed up with type 4 : 1
Button OK was pressed down with type 4 : 1
Location X 1047/694
Button OK was pressed up with type 4 : 1
Button OK was pressed down with type 4 : 1
Location X 1047/694
Button OK was pressed up with type 4 : 1

This processed both the down and up beautifully with an ID of 1 (the mouse seems to always return that value).
Here are console messages from the touch screen:

Button OK was pressed down with type 2 : 1146
Location X 1070/697
Button OK was pressed up with type 2 : 1146
Button OK was pressed down with type 2 : 1147
Location X 1071/698
Button OK was pressed up with type 2 : 1147
Button OK was pressed down with type 2 : 1148
Location X 1065/703
Button OK was pressed up with type 2 : 1148

The ID value for down/up pairings match ( a good sanity check) and are very consistent!
I can absolutely use this for a button press!!

I need to play with the other messages in more detail.. This is AWESOME!!
Unfortunately, I am leaving tomorrow to be on travel until late next week, so I won't be able to play until then.
I can't thank you enough for the head start, I need to understand WIN32 and subclassing better than I do!
I'll PM you (if you don't mind), I'd love to pick your brain about HOW to code against the guts of the WinAPI like this..
You Rock my friend.. THANK YOU!!

Steve

 

 

Edited April 21, 2022 by Bumperscoot

[Bumperscoot]

Nine (+1),

I had a go at playing with IS_POINTER_INCONTACT_WPARAM ...

Here's the WM_POINTER_SUB code I modified:

 

Func WM_POINTER_SUB($hWnd, $iMsg, $wParam, $lParam, $iID, $pData)
  Local $iPointerID, $iX, $iY
  local $iPointerContact

  Switch $iID

      Case $idButton

      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("Button OK was pressed down with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
      EndIf

        if $iMsg = $WM_POINTERUP Then
            $iPointerID = GET_POINTERID_WPARAM($wParam)
            $iPointerType = GetPointerType($iPointerID)
            ConsoleWrite("Button OK was pressed up with type " & $iPointerType & " : " & $iPointerID &  @CRLF)
        endif

  EndSwitch

  if $imsg = $WM_POINTERUPDATE  then
        $iPointerContact = IS_POINTER_INCONTACT_WPARAM($wParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("UPDATE " & $iPointerType & " : " & $iPointerID & " " & $iPointerContact & @CRLF)
  endif

Return _WinAPI_DefSubclassProc($hWnd, $iMsg, $wParam, $lParam)
EndFunc   ;==>WM_POINTER

This works..
However.. $WM_POINTERUPDATE is only fired on the initial action and repeats if motion is detected from that initial action on either Mouse or Touch.
For the wired mouse example, $iPointerContact evaluates to "false" when the mouse is moved over the button and/or is in motion over the button.
Conversely, $iPointerContact returns a single (slightly delayed) "true" when the wired mouse left button is held down, and repeats "true" when the mouse in motion with the left button down.

For the touch input, $iPointerContact returns true only when the touch input is on/over the button (clicked), and continues to return true as long as the touch is in contact with the screen. When the touch input is released (i.e., I lift my finger off the button), $$WM_POINTERUPDATE messages cease and there is no "false" return.

It's late here, so I hope this makes sense.. Still Very awesome!

I'll talk to you soon..

 

Steve

[Nine]

Fantastic.  Glad it is working nicely on touch screen.  Nice job BTW.

Here my take on a hold approach.  It is important to use the dummy control as you want messages to be properly processed.  If you shortcut processing by setting a global variable inside the callback you may get undesirable effects after awhile.

expandcollapse popup

#include <GUIConstants.au3>
#include <WinAPIShellEx.au3>
#include "WM_POINTER.au3"

EnableMouseInPointer(True)

Global $hGUI = GUICreate("WM_POINTER")
Global $idButton = GUICtrlCreateButton("OK", 250, 350, 100, 25)
Global $idDummy = GUICtrlCreateDummy()
Global $hDll = DllCallbackRegister(WM_POINTER_SUB, 'lresult', 'hwnd;uint;wparam;lparam;uint_ptr;dword_ptr')

_WinAPI_SetWindowSubclass(GUICtrlGetHandle($idButton), DllCallbackGetPtr($hDll), $idButton, 0)
GUISetState()

Local $bPush = False

While True
  Switch GUIGetMsg()
    Case $GUI_EVENT_CLOSE
      ExitLoop
    Case $idDummy
      $bPush = GUICtrlRead($idDummy)
  EndSwitch
  If $bPush Then
    ;do something here
    ConsoleWrite("button is pushed and held" & @CRLF)
  EndIf
WEnd

_WinAPI_RemoveWindowSubclass(GUICtrlGetHandle($idButton), DllCallbackGetPtr($hDll), $idButton)
DllCallbackFree($hDll)

Func WM_POINTER_SUB($hWnd, $iMsg, $wParam, $lParam, $iID, $pData)
  Local $iPointerID, $iX, $iY
  Local $iPointerContact

  Switch $iID
    Case $idButton
      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("Button OK was pressed down with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
        GUICtrlSendToDummy($idDummy, True)
      ElseIf $iMsg = $WM_POINTERUP Then
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("Button OK was pressed up with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        GUICtrlSendToDummy($idDummy, False)
      ElseIf $iMsg = $WM_POINTERUPDATE Then
        $iPointerContact = IS_POINTER_INCONTACT_WPARAM($wParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("UPDATE " & $iPointerType & " : " & $iPointerID & " " & $iPointerContact & @CRLF)
      EndIf
  EndSwitch

  Return _WinAPI_DefSubclassProc($hWnd, $iMsg, $wParam, $lParam)
EndFunc   ;==>WM_POINTER_SUB

Currently the switch $iID is useless because there is only one control subclassed.  If you add other controls, then it will be useful.  And to answer your question, yes all events on the button are subclassed within the callback function.  Also it is important to release ressources at the end of the script to ensure everything closes normally.

Edited April 21, 2022 by Nine

[Bumperscoot]

Hey Nine (+1) !!

 

Couldn't resist.. 
I did this quickly (code below), but you get the idea. I also shortened up the messages..
Question:
Isn't setting a dummy control the same as setting a global status "flag" (I.e. a boolean var) in the Callback, then processing it in the main loop?
Again, thanks for all your help!!!

Steve

expandcollapse popup

#include <GUIConstants.au3>
#include <WinAPIShellEx.au3>
#include "WM_POINTER.au3"

EnableMouseInPointer(True)

Global $hGUI = GUICreate("WM_POINTER")
Global $idButtonTB1 = GUICtrlCreateButton("TB1", 200, 50, 100, 100)
Global $idDummyTB1 = GUICtrlCreateDummy()

Global $idButtonTB2 = GUICtrlCreateButton("TB2", 200, 200, 100, 100)
Global $idDummyTB2 = GUICtrlCreateDummy()

Global $hDll = DllCallbackRegister(WM_POINTER_SUB, 'lresult', 'hwnd;uint;wparam;lparam;uint_ptr;dword_ptr')

_WinAPI_SetWindowSubclass(GUICtrlGetHandle($idButtonTB1), DllCallbackGetPtr($hDll), $idButtonTB1, 0)
_WinAPI_SetWindowSubclass(GUICtrlGetHandle($idButtonTB2), DllCallbackGetPtr($hDll), $idButtonTB2, 0)

GUISetState()

Local $bPush1 = False
local $bPush2 = false

While True
  Switch GUIGetMsg()
    Case $GUI_EVENT_CLOSE
      ExitLoop
    Case $idDummyTB1
      $bPush1 = GUICtrlRead($idDummyTB1)
      ConsoleWrite(" TB1 button is pushed and held" & @CRLF)

    Case $idDummyTB2
      $bPush2 = GUICtrlRead($idDummyTB2)
      ConsoleWrite(" TB2 button is pushed and held" & @CRLF)

  EndSwitch

    If $bPush1 Then
        ConsoleWrite("TB1 is pushed" & @CRLF)
    Else
        ConsoleWrite("TB1 is released" & @CRLF)
    endif

    if $bPush2 then
        ConsoleWrite("TB2 is pushed" & @CRLF)
    Else
        ConsoleWrite("TB2 is released" & @CRLF)
    EndIf

WEnd

_WinAPI_RemoveWindowSubclass(GUICtrlGetHandle($idButtonTB1), DllCallbackGetPtr($hDll), $idButtonTB1)
_WinAPI_RemoveWindowSubclass(GUICtrlGetHandle($idButtonTB2), DllCallbackGetPtr($hDll), $idButtonTB2)

DllCallbackFree($hDll)

Func WM_POINTER_SUB($hWnd, $iMsg, $wParam, $lParam, $iID, $pData)
  Local $iPointerID, $iX, $iY
  Local $iPointerContact

  Switch $iID
    Case $idButtonTB1
      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("TB1 was pressed down with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
        GUICtrlSendToDummy($idDummyTB1, True)
      ElseIf $iMsg = $WM_POINTERUP Then
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("TB1 was released with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        GUICtrlSendToDummy($idDummyTB1, False)
      endif

    Case $idButtonTB2
      If $iMsg = $WM_POINTERDOWN Then
        $iX = _WinAPI_LoWord($lParam)
        $iY = _WinAPI_HiWord($lParam)
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("TB2 was pressed down with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        ConsoleWrite("Location X " & $iX & "/" & $iY & @CRLF)
        GUICtrlSendToDummy($idDummyTB2, True)
      ElseIf $iMsg = $WM_POINTERUP Then
        $iPointerID = GET_POINTERID_WPARAM($wParam)
        $iPointerType = GetPointerType($iPointerID)
        ConsoleWrite("TB2 was released with type " & $iPointerType & " : " & $iPointerID & @CRLF)
        GUICtrlSendToDummy($idDummyTB2, False)
      endif

  EndSwitch

  Return _WinAPI_DefSubclassProc($hWnd, $iMsg, $wParam, $lParam)
EndFunc   ;==>WM_POINTER_SUB

 

[Nine]

18 minutes ago, Bumperscoot said:

Isn't setting a dummy control the same as setting a global status "flag" (I.e. a boolean var) in the Callback

I already said in my previous post, DON'T.  Please read carefully again what was explained.  Another reason not to use a global variable, is exactly not to use a global variable.  One should always try to reduce as much as possible usage of global variables.  In a long and complex script it may become quite fastidious to follow a large amount of global variables.  This is just a best practice to eliminate global variables when feasible...

18 minutes ago, Bumperscoot said:

Couldn't resist.. 

Edited April 21, 2022 by Nine

[Earthshine]

problem with global variables is this, when you are developing something new or altering something that already exists, it's too easy to change the value of it. See, other things rely on a certain value to be a certain thing at a certain time, and now you come along and whack it with some new function thinking--hey, i can use this variable to do what i want locally here.... and you break the rest of the program

any program that relies on global variables is basically retarded and should never be published.

globals serve a purpose, maybe one or two but not your stuff, each function should create--or be passed EXACTLY what it needs to do it's job locally. then as you add new functions you never break the application, be it an installer or whatever you are doing.