Delivering touch events to Lua

[jfrickmann]

Before we start coding, I think that it would be good to have a general discussion among the Lua script writers about how we would like to have touch events delivered to the scripts.
I have looked around in the code to find out how the touch system is implemented. I found the basic interface definition in touch.h:

enum TouchEvent
{
    TE_NONE,
    TE_DOWN,
    TE_UP,
    TE_SLIDE,
    TE_SLIDE_END
};

struct TouchState
{
  unsigned char event;
  short x;
  short y;
  short startX;
  short startY;
  short deltaX;
  short deltaY;
  short lastDeltaX;
  short lastDeltaY;
};

And I found some onTouch functions implemented in libopengui in the Window class, which AFAIK is the base class for most other GUI elements:

bool Window::onTouchStart(coord_t x, coord_t y)
bool Window::onTouchEnd(coord_t x, coord_t y)
bool Window::onTouchSlide(coord_t x, coord_t y, coord_t startX, coord_t startY, coord_t slideX, coord_t slideY)

[jfrickmann]

I think that a straightforward interface would be to deliver touch events in the same argument that delivers key events, i.e. add the following to the list of possible events:

EVT_TOUCH_FIRST
EVT_TOUCH_BREAK
EVT_TOUCH_SLIDE

Also add a touchState argument to refresh(...). If we did not get one of the above events, then touchState == nil, otherwise it is a table with the relevant coordinates.

[pfeerick]

Just linking this here for consideration also, as there is also recent discussion on this in OpenTX...

Issue: opentx/opentx#8598
Related PR: opentx/opentx#8599

[wimalopaan]

I think there should be also a higher level interface.

For slide events I propse the same as with key events something like slide-left/rifht/up/down events as a simple technic for simple use cases.

Raw tap events with coordinates may be useful.
But I think it would be more useable for lua to have the possibility to define a button (a button area e.d. id = luaCreateButton(...)), and then also tap events on this area can be delivered like key events.

[raphaelcoeffic]

So basically having a list "hit-boxes" matched in C code? libopenui provides this on window by window basis, but it is going to be very cumbersome to provide a full windowing API to the LUA code to allow for libopenui to be leveraged properly.

[lshems]

I'm primarily interested in getting touch events converted to key events with the same functionality.

That might ask some thinking in how to link what to what, but would at least solve touch enabling in one go for all current use cases.

A little bit similar to windows, where in the end all mouse actions are also executable using keystrokes.

I could think of tap for enter, swipe left/right page up/down, swipe up/down plus/minus. Although the last one would be better with two finger slide. Supported? I don't know. What to use for exit? Tap on another editable item? Double tap on popupped items?

Just some thoughts

[raphaelcoeffic]

I could think of tap for enter, swipe left/right page up/down, swipe up/down plus/minus. Although the last one would be better with two finger slide. Supported? I don't know. What to use for exit? Tap on another editable item? Double tap on popupped items?

This is possibly a good idea to provide touch support to LUA Apps which have not been designed to use that new API.

[lshems]

Not only Lua. If done low level, the entire GUI becomes directly touch aware.

But if done by touch conversion to virtual events like that, you can't differentiate anymore between virtual events triggered by touch and those by buttons.

I could imaging to have a 'button emulation' option, that would directly enable touch but without specific touch GUI adaptations. The other option would be 'Touch GUI'.

[wimalopaan]

I'm primarily interested in getting touch events converted to key events with the same functionality.

I would go for different events, so that they are distinguishable in the script.
Then lua authors can add the tap event easily by reacting on those event-codes in the same manner.

[jfrickmann]

It would be fairly easy to convert touch events to key events in your scripts, and the advantage of doing it yourself, is that you can do whatever you want ;-) E.g.

local function myEvents(event, touchState)
  if event == EVT_TOUCH_FIRST or event == EVT_TOUCH_LAST then
    return 0
  elseif event == EVT_TOUCH_TAP then
    return EVT_VIRTUAL_ENTER
  elseif event == EVT_TOUCH_SLIDE then
    if touchState.swipeRight then
      return EVT_VIRTUAL_NEXT_PAGE
    elseif touchState.swipeLeft then
      return EVT_VIRTUAL_PREV_PAGE
    elseif touchState.swipeDown then
      return EVT_VIRTUAL_NEXT
    elseif touchState.swipeUp then
      return EVT_VIRTUAL_PREV
    else
      return 0
    end
  end
  return event
end

function refresh(widget, event, touchState)
  event = myEvents(event, touchState)
  ...
end

[jfrickmann]

I agree that we should keep different events. If we started adding code to support a specific use case, then we would also be limiting people who want something different. Let's build on what is already there, and add OllieW's "extended" events also.
It can be done in the onTouch functions that we will add to the LuaWidget class - no need to mess with the underlying drivers.
How about the following. We add the following values to events:

EVT_TOUCH_FIRST
EVT_TOUCH_BREAK
EVT_TOUCH_SLIDE
EVT_TOUCH_TAP

The latter is returned if EVT_TOUCH_BREAK happens within 250ms after another touch event.
The swipe events are special cases of EVT_TOUCH_SLIDE, and they would be added to the touchState table as a true value, e.g. swipe right:

event == EVT_TOUCH_SLIDE and touchState.swipeRight

That way, people can choose if they want to swipe or slide in their script :-)

[lshems]

For the lua scripts I don't have any issues with the approach. I was just trying to think of what I would do to enable touch for my scripts, which is exactly as Jesper suggests (I use the same approach to mimic buttons with switches for widgets).

But the same approach would also work for the entire GUI if you could do the same event replacement trick.

[jfrickmann]

I have started working on a Lua GUI design for SoarETX, and I wonder if there is interest in developing a common Lua utility library with GUI functions etc. that comes preloaded on the SD card?
One interesting thing about the Lua widget design - since all Lua widgets on the SD card are always loaded into memory, you can define a global Lua function in a small dummy widget to load a library file. That way, the library is globally available, even if the dummy widget is not instantiated!
Also, since all widgets are always loaded, it can take up A LOT of memory! Therefore, my widgets will be very small files that load the rest of the code only if the widget is instantiated. I can share a simple pattern for how to accomplish that.

[lshems]

Well, I'm interested, but at the same time cautious. Cautious because I now have the freedom to make my own choices regarding everything in my justfly project. I'm a little bit stubborn and like to progress in the way I think is best.
Cautious also because I don't want to opensource everything in my project just to find out nobody is really putting effort in, and then being stuck forever to remain opensource.

Opensourcing is a one way track, some companies have found out the hard way.

So unless I really are convinced by arguments put forward that opensource is the best way to manage my project, I will not opensource my project.

The added value of such a lib would thus be limited to me, because you then really enter a gray zone, of using lua libraries developed as opensource specifically for edgetx, for a closed source project. Is still defendable, but I don't even want to start the discussion.

So either people must convince me, or my support is limited in a sense that I'm always willing to share my time and knowledge, as it is fun. But I wont be pushed because I would use it myself.

I hope you understand my position.

Opensource is not a holy grail suitable for all projects.

[jfrickmann]

Guido, I am not surprised to hear that you will do your own thing ;-)
Having a shared library would by no means force anyone to use it, but it could be helpful for those who would rather get started writing their own widget instead of creating a GUI from scratch. It would also save some memory, if people do not take special precautions to write widgets that only load the main code on demand.
But this is indeed the advantage of doing things in Lua instead of baking it into the OS - that it is optional instead of mandatory!

[pfeerick]

This is exactly the direction we would like to see things go also, as that means you guys can focus more on the front end/UI stuff, giving more time to work on the underlying code.

[lshems]

Ok.lets help people. Fully in favour of that.

I think a widget loader would be nice.

Loading a telemetry script or even a simpler format. Just a function script Lua.

Forget about background and run, and certainly init. Just a script that is called.
With preemption it doesn't even need to be in sync with the mixers. You can loop your screen write commands like you wish.

So basically have them run a 'BASIC' script.
We could even package the LCD functions into something more basic like. Like draw(x,y,something).

This is what could invite people to write their own scripts for widgets.

For me, that is something I would like to invest in.

[jfrickmann]

The widgets live in a different Lua state than the other scripts, and I have only implemented preemption for the other scripts. So no preemption in widgets for now, and I cannot apply the same solution for widgets as I did for the other scripts. Maybe later, but for now I am focused on how to design the Lua code to start making useful stuff with widgets, and it will also be nice when we get theme colors working in ETX so we can start looking at the aesthetics.

[lshems]

Oh, didn't know that. That's a pity. So the entire preemption thing is for mixerscripts, telemetry scripts and one-time scripts.

Ok. Well, for me that means still a great achievement for getting mixer and telemetry scripts more reliable and perhaps even fit for flight.

[lshems]

Ok, well, two steps back on my wild ideas. But yes, still interested to help. Just try to set out a goal, a basic design, ... What you want to achieve.

I'm sure we can make something that will help people do things easier with such a library.

So yes please, share your first ideas.

[jfrickmann]

OK, I have uploaded the widget scripts that I made so far here: https://github.com/jfrickmann/SoarOTX/tree/SoarETX/SD-CARD/WIDGETS
It is very much work in progress, but the general scaffolding is beginning to take shape!

[lshems]

Ok, just checked the scripts.

I am not sure I understand where you're intending to go.

Perhaps we need to have a chat so we can share thoughts first. Is discord open for voice chat?

Guido

[pfeerick]

Yes, there is a voice chat and accompanying text channel ;)

[lshems]

We found it. Thanks!

[JimB40]

Okay I had some research about touchUI APIs in other platforms.
So far we talk only about 'single touch' (one finger) events

First approach is to include only 4 basic events TOUCH_START, TOUCH_MOVE, TOUCH_END, TOUCH_CANCEL.
This allows to code anything else.

Still I think It may be better to code some of additional functional events in C++ leaving possibility for LUA dev to write customised solution using those basic four above mentioned.
By additional functional events I mean
TOUCH_TAP -> single short tap - used mainly for selection
TOUCH_LONG -> single long tap (press) - used mainly for contextual menus
TOUCH_DOUBLE_TAP - double short tap - use case is usually to select default option
TOUCH_SLIDE -> this is when user press screen & move - changing sliders value, scrolling etc
TOUCH_SWIPE -> this is quick slide finished with finger removal -> changing views, invoking menus, quick scrolling etc

From what I read those additional events need timing configuration that should be set and tested with specific hardware.
There is also tolerance (deadband) to consider ie when user press screen and touch coordinates changes within predefined scope API still returns TOUCH_LONG instead TOUCH_SLIDE

[JimB40]

Mapping touch events to key events equivalents can be easily done within script, and existing scripts adjusting to touchUI is inevitable.

On the other hand predefined mapping can cause various problems.
Take as an example UI selection menu. In key events scenario it's most probable that plus/minus will select menu line while enter executes code connected to this menu line. If we initially map TOUCH_TAP to EVT_VIRTUAL_ENTER tapping on non-selected menu line will invoke selected menu line code, which is not desired behavior.

[jfrickmann]

What has been implemented is the following:

• EVT_TOUCH_FIRST happens one time when the user plants a finger on the screen.
• EVT_TOUCH_TAP happens if the finger is lifted within 250ms.
• EVT_TOUCH_BREAK happens if the finger is lifted after 250ms.
• EVT_TOUCH_SLIDE happens continuously while the finger is moved around, and neither TAP nor BREAK happens after a slide.

In addition to the (x, y) that comes in touchState with all touch events, SLIDE also gives (startX, startY) which is where the first SLIDE happened, and (slideX, slideY) which is the movement since the previous SLIDE.
If the user swipes, then touchState also comes with e.g. swipeUp=true or swipeRight=true etc.
I have so far hacked about 400 lines of Lua GUI library code, and this covers my needs. But I am fairly agnostic and open to discussing, if there is consensus for an alternative protocol. The above is based on @olliw42's solution that @pfeerick linked to in a previous post.

[jfrickmann]

Here is yet another proposal. We can have two types of virtual key events: EVT_TOUCH_* and EVT_SLIDE_*.

When a finger touches the screen, we get EVT_TOUCH_FIRST, and if it stays there, then we get the usual key event sequence: EVT_TOUCH_LONG, EVT_TOUCH_REPT(s), and then EVT_TOUCH_BREAK when the finger is lifted. With EVT_TOUCH_BREAK we also get a Taps field added to the touchState table, counting the number of times we have tapped the screen within a short time (zero if we did not tap). So in principle, we can get single, double, triple etc. taps.

If the finger starts sliding around, then we get EVT_SLIDE_FIRST, followed by EVT_SLIDE_REPT(s), and finally EVT_SLIDE_BREAK. In the case of a swipe, we get toucState.swipeRight=true etc. (as in the current model).

What do you think about that proposal? It is more feature complete, but also a little more complicated to use. I will be happy to implement it and put it up here in a new PR so people can compare to the existing implementation.

Please discuss 🙂

[wimalopaan]

I would like this approach too: most work should be done in C++-Code instead of lua.

Cool would be a way to define some sort of "button", say a rectangualar area in wich to receive an event, and that the lua-events transports the button "id" too.

[jfrickmann]

Event widgets.zip
Here is the latest update of my widgets to test events. Please take a look at LibGUI to see how I have implemented buttons and other screen elements in Lua. The advantage of doing it in Lua rather than baking it into C++ is 1) It is much simpler 2) It is flexible 3) It does not force everyone to use a single, fixed model.

And the beautiful thing is that, since all widgets are loaded all the time, the shared GUI library is always available as long as the widget is on the SD card, i.e. you do not need to add the widget to your screen to use the library functions. The only purpose of the widget is to demonstrate how the library functions are used.

And since both the library and the widgets are loaded on demand, they take up almost no memory when they are not used.

Please go to PR #498 to grab firmware for testing. (Look under Checks, and Artifacts in the upper right corner)

[lshems]

I have no issues if I need to convert touch myself to the corresponding key events.

That's a standard function I had to make to allow switched to simulate buttons. So no issue there:)

[lshems]

It would just have been my first approach if the entire opentx GUI including scripts would have to be prepared for touch. But the opentx GUI is already following a custom approach. So that's fine.

[jfrickmann]

I have moved this to a new pull request: #489

[jfrickmann]

The PR have been merged into the main branch! So please check the nightlies, if you want to play with events 🎉