Overhaul button drawing/styling

[Julusian]

For anyone stumbling upon this curious about what this is;

We want your examples for complex buttons or weird scenarios that are hard to do in the current system.

---

There are a few threads around different aspects of this already, but things have changed in companion since they were opened and there are additional requirements now.
The ones I found: #262, a snippet of #2387, #2237, #3067, #2367

I want to focus on the aspects of:

1. Supporting multiple aspect ratios of drawing
2. Supporting more drawing options without cluttering the ui

A lot of this builds heavily on what @dnmeid has already written, primarily around adding layers.

I think that doing anything around this is going to 'break' advanced feedbacks, because they are very tied to the current structure of properties we use for drawing. This structure has become a bit of a mess as new bits have been added over time, but has been unable to be tidied up the layout due to it being used in feedbacks and the module-api.

I would propose that this would probably be best achieved as a new button type. We wouldn't touch existing buttons (for now), once this is done we could look at auto converting some/all existing buttons across, but I expect that not everything will translate perfectly.
This way, we don't break things for existing users, and don't limit ourselves by trying to minimise this breakage.
We can then drop the old rendering in a future major version, or sooner if we can figure out acceptable automatic translation.

I feel like this should be implemented largely in the open in main, rather than hidden on a branch until it is complete. We can label the new type as alpha/beta, and if we want to do a stable release we could leave it in or hide the ui option

Layers

As has already been proposed, I agree that buttons should have layers. I would suggest that each layer should do a single thing, ie, a layer can be a png layer or a text layer. This will allow us to add more options within each layer without making the ui too cluttered. And at the same time, we can add new layer types without cluttering up the ui,
The user should be free to add and remove as many layers as they like into this stack, the one they will not be able to remove is the fixed background layer which will be where the background colour is set.
UI clutter is a big concern I have around improving the drawing, the existing style properties don't feel very organised and any changes to the current system feel like they will make it worse unless we can categorise things better..

Once we have this structure, adding many of the things from https://github.com/thedist/companion-module-utils would be much simpler, as they would be their own layer types.

A thought for myself, perhaps we could utilise the layering for some extra performance enhancements? We could draw each layer as its own bitmap and then flatten afterwards. Doing this is excessive, but we could utilise it for any costly layers, or to avoid repeating ourselves too much when it comes to gifs or frequent redraws. Not in scope of this, but something to think about in the future.

What about feedbacks

Now for one of the more radical ideas I have here; we should move the feedbacks to be defined as part of this layering.
For boolean feedbacks, this would be a pretty trivial way of adding an enable rule on each layer. This will likely need a way of linking layers to accommodate this fully, or a way of defining some 'button local' variables based on feedbacks. I am not sure on the best approach here.
For 'advanced' feedbacks, as written above I dont think these will translate well to this new model. I would be tempted to call these 'legacy feedbacks', and be able to mark layers as 'override for legacy feedbacks'. And at the same time, define a new 'advanced' feedback type which would fit better into this model.

I forgot to consider here that feedbacks override the defaults currently..
Perhaps then each layer type should have its own feedback stack to allow applying overrides to itself? That feels like it will be a bit clunky, but if we encourage doing it via local variables (or only allow via local variables?), to be able to share feedbacks across layers then it wouldnt be terrible? Definitely better than forcing enabling/disabling of layers
I am very open to ideas here

Aspect ratios

Finally, we need to be able to handle different aspect ratios of buttons. The Stream Deck Studio has slightly widescreen buttons that we 'letterbox', the Neo has the lcd strip that we ignore, the Plus has the LCD strip that we leave space around each button, the loupedecks have vertical strips that we ignore, and more quirks will come in the future.

I am not 100% sure how this should work, but I think long term we need to allow the user control over how different aspect ratios render.
The MVP could be to just perform the same operations on the different aspect ratio, and hope it works out. This will likely do for many users, as they will likely only look at buttons on one type of surface. We would need to figure out how to show this to them in the UI, but I dont think that is in scope of this task.

For a proper solution, I feel like we would need to have a separate stack of layers to draw, but that will get tedious if you just need to tweak one thing. Perhaps they share a stack and layers can be marked as 'only for X', and 'except for X'. I'm not sure, and I think it would be best to leave this to a follow up task.

The UI

I am not sold on these ideas, just my immediate thoughts for future me.

For showing this in the UI, I think this would be a case of adding this as a new tab in the action step tab list. Probably as the first and default tab.
The 'progress' and 'rotary actions' toggles should probably be moved to an 'options'/'settings' tab there too.

This tab could be split with the top portion showing a larger preview of the button, with a layer list beside it.
With the editor for the currently selected layer below.

[Julusian]

define a new 'advanced' feedback type which would fit better into this model.

From memory, there times where advanced feedbacks are used are:

1. module hasn't been updated to boolean feedbacks
2. feedback produces a highly custom imageBuffer
3. feedback produces text/colour values based on some device state.

For 1, we can ignore these, they should be migrated to boolean.
For 2, we should continue to support this, perhaps a new 'imageBuffer' feedback type, which does just this? There is the question here on how to handle different aspect ratios, do we want to tell the feedback about all the aspect ratios it should do, or tell the module about 'fake' feedbacks for the non-standard ones.
For 3, perhaps we need a way for a feedback to produce variables? If the feedback produced some 'button local' variables, they could then be used by the user in one of their layers.

I don't know if modules do 2&3 in the same feedback, hopefully not. Or if they do, then maybe any feedback should be able to produce these variables (including boolean feedbacks). We would then need somewhere to park feedbacks which are used just for their variables, but that should be solvable.

[Julusian]

Some other things will 'break' as part of this

The feedback 'use another buttons style' won't work fully, as one of those 'advanced' feedbacks. I expect we will leave this 'broken' when going across types, and add a new one for this new button type.

The 'set button text' and similar api endpoints and actions wouldn't nicely translate across to this. We could take the opportunity to remove them, or we could expand them to allow choosing a layer index (which will be brittle) defaulting to the first text layer. I would lean towards removing them, everything they support will be possible with custom variables instead in this.

Satellite API can provide text and text colour to clients. We need to define a rule on how those should be determined from a multi-layer button.

[dnmeid]

I appreciate this summary and it is mostly also what I'd suggest.
There is maybe this one to link where I maintained the work of this project so far https://github.com/orgs/bitfocus/projects/4 but this can also transfer to Companion plan.

• Layers: d'accord with the general specs. I'd maybe add the possibility of (nested) layer groups. So one could also switch on/off a complete group or have it switched by a feedback/variable
• Feedbacks: I'd propose to pretty much rely on the local variables here. Let's assume every feedback just writes its original data to a local variable. Now there is this expressions everywhere idea (Extend expressions to action options #2345) and that means you can simply switch the enable of a layer to be fed from an expression and use the boolean of a boolean feedback. Or you can switch the color to an expression and can use the color part of an advanced feedback by accessing it from the json. Or using the text of a feedback in a text layer and combine it with some other text. That way we could even fit all todays functionality in the new layers and wouldn't necessarily need a new button style. The styling options in the feedback itself could be dropped and would be replaced by the style of the layer or what you write in the expression. The downside I see, is much more work for the user to style the button. A lot users want to have more flexibility but not more work. I can only think of some nice workflow enhancements to streamline the setup process, e.g. drag and drop a feedback to an option and it will switch to an expression with the according local variable filled. Or having the expression prefilled with the most likely variable when switching to it.
  Another issue I see, is to transfer the current way of feedback stacking, i.e. higher feedbacks overwrite lower ones. One can easily stack the boolean sources of boolean feedbacks. Also it would be possible to just have e.g. one solid color layer per boolean feedback and switch it on/off from the feedback. This leads to the same result, but I think this may be too much effort for many users. Even worse with stacked advanced feedbacks. What so far has been an easy to use feedback will then be some programming like $(this:fb3.text) || $(this:fb2.text) || $(this:fb1.text) . We could maybe offer a pre-stacked preudo-feedback like$(this:advanced_feedbacks_stacked).
  We could also offer this as layer types, eg. a layer for all stacked advanced feedback backgrounds, a layer for all stacked advanced feedback images, a layer for all stacked advanced feedback texts.
  I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks, like a string feedback that returns a string or a number feedback that returns a number and this will have the same workflow like the boolean feedbacks, just with different datatypes.
• Aspect ratios: My thoughts here are two possibilities: First possibility is to have sizes either in pixels/pt or in percent, second is to add something like CSS breakpoints, i.e. some conditional logic what to use for a given size or a/r.
  There are two questions. How can we make a UI for the conditional logic and should we switch complete layers or only some properties. When we have expression possibilities in size properties we automatically get the ternary operator with it. We would just need to add the button size and/or a/r to local variables and then you could use it in the expression. This would also mean that a button image can only be rendered at surface level but I think that is anyway what we need to do. I'm not sure if we should do it per surface or per surface type or per output type and where in the code (thinking of the deamon idea here).
  For the fixed/proportional value entry I see two possible solutions. One would be to simply have a dropdown per option where you can switch if the entered value or the value from the expression is pixels, percent (0-100) or a float (0.0 - 1.0) and the other one would be some automatic switching for regular options like css does where you can enter 17 or 17px for pixels and 0.5 or 50% . For expressions we would need to either force the users to generate a size string then with the named suffixes and/or offer functions to convert to special data types like asPixels() or asPercent().
  The first solution is easier to understand, the second is more flexible as you could also switch between fixed and proportional sizing by logic. I wouldn't necessarily offer expression possibility for the dropdown of the first solution because this would be overkill to have possibly two independent expressions to generate one size.
• breaking stuff: This stuff would not necessarily break the existing button/feedback system and can in theory be upgraded automatically. It would not nessarily break the advanced feedbacks but automatic upgrades may not look very nice.
  Im not sure if two different button types would be a good solution. If we keep the current button as it is, we would also need to keep the feedbacks as they are, that means we would e.g. still need the style options for a boolean feedback. If we keep the button mostly as ist is but make the feedbacks part of that legacy button, we would have two different feedback presentations. In the legacy button there would be style options and in the new one we wouldn't show them. Additionaly I'd consider the feedbacks more to be part of the control itself if they feed local variables and not part of the graphics, because local variables can be used everywhere in the control.
  Satellite text and color transmission I think is a easy solution, I'd provide a as good as possible backward compatible solution and add the new features to it. I expect a good adoption rate of users that make use of these special features and others will just not update.
  For the set text and similar endpoints I'd suggest to just always direct them to the most probable layer, e.g. we could say one has to call a solid color layer Background and then it will receive the background color and if that name is not found it will be the lowest solid color layer. This I'd just do to not break existing configurations, but I'd also discourage using such workflows in the future and prefer user variables instead.
  "use another button's style" could use the same logic of mapping existing layers to legacy style options. That would at least make sure that automatically updated configs are still usable. But also in the long term I would deprecate it. I think the whole thing of borrowing a style from a different button is kind of a workaround and we should think of what the real goal behind it is and if there maybe is a better solution.

There's now not an answer for all of the open questions, but I tend to write too much and I think we'll find more solutions along the way if we agree first on the concept.

[Julusian]

Layers: d'accord with the general specs. I'd maybe add the possibility of (nested) layer groups.

Yeah I was thinking about that too, but didn't write it.

Sounds like this could be started on then, either as unit tests, or a standalone browser based tool so that we can play around with it and see how aspect-ratios behave.

Feedbacks: I'd propose to pretty much rely on the local variables here.

No objection from me on this. It will be harder for users, but that might just be the cost of necessity. This whole change will need some UX focus, hopefully we can mitigate confusion with some tooltips or better inline help/or links to help.

I can only think of some nice workflow enhancements to streamline the setup process

An idea: make it one click to create a new local variable and jump to where to setup the value

What so far has been an easy to use feedback will then be some programming like

For advanced feedbacks, I don't mind making them awkward/hard to use. In a way, I want to be 'hostile' against using them, to motivate pushing modules to newer feedback types that fit our new systems.
As I see it, we either make it painful to use and hope it pushes modules away from it, or it could be annoying to use and modules wont bother changing as quickly as its fine and works.

The other challenge with advanced feedbacks, is that we don't know what properties they will give us. And the properties they give us can change each time they give us a value. So we either need to do something to semi automatically link all the properties, or leave that for the user to do and hope they know what the feedback will give them.

But you're right, this will mean that replacing the text will become more programming like; which I think is unavoidable its just a question of how we expose/encourage that programming to be done (expressions vs logic feedbacks vs override layers). I don't think we need to decide on this immediately, this feels like something that can be iterated on once we have something to look at and play with.

I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks

Yep, I agree. It should be able to produce multiple values, with a predefined list of values to expect in the feedback definition. (to allow for cases where today they return a bgcolor and fgcolor or whatever)

Aspect ratios: My thoughts here are two possibilities: First possibility is to have sizes either in pixels/pt or in percent,

Thats a good point, maybe using things like this would get us 99% of the way there. Throw in using expressions, and some new variables as input to the expression of the width and height of the current drawing, then complex formula become possible.
We probably don't need to make this 'simple' to use, I suspect it will be a minority case where fine control is needed over the drawing. Maybe something about em/rem too for sizing? I don't fully know what they do, but know theyre relative to something..
Making expression output be consistent for this will need some work, if things like randomInt are used but all doable.

I'm going to skip over the rest of this for now, as its not critical for a MVP, and I think will be easier to reason about once we have something we can play/test ideas with.

breaking stuff: This stuff would not necessarily break the existing button/feedback system and can in theory be upgraded automatically.
If we keep the current button as it is, we would also need to keep the feedbacks as they are, that means we would e.g. still need the style options for a boolean feedback

Well, the style properties of boolean feedbacks belong to companion, and they are already ignored when used as conditions, so I don't see this as a problem. Sure, it would be nice to clean them from the code, but thats not an urgent thing.

Im not sure if two different button types would be a good solution.

Long term no, but for a couple of releases I think it would be a good idea to. And maybe depending on user feedback and our feeling on how complex the new system is to use, I would be open to keeping a slightly simplified form of it if there is enough demand for it.

Satellite text and color transmission I think is a easy solution, I'd provide a as good as possible backward compatible solution and add the new features to it

My main uncertainty with it was what text do we send. But I suppose it would be taking the text from the top or bottom most enabled text layer.
I am very hesitant to add anything resembling the full layer blob to the satellite protocol, as then suddenly the semver of the satellite api becomes tied to how we represent this layer drawing. We will always need to support forward converting the structure for imports and presets, but for satellite that would mean converting it to older versions too.
I much prefer making companion do the compositing when wanting an actual pixel buffer; for me the text/color options are there to support simple surface types that don't have full rgba screens and can only do simple things like showing a line of text possibly with colours.

---

So to summarise, I think I have enough answers to be able to start playing around with this.

I think the next steps are:

• Define the layers data-structure, and implement the rendering, possibly as its own tool.
• Do a minimal version of this button type, using purely static data.
  • During this stage, the editor should probably show a few different aspect ratios at all times, square, SDS, really tall and really wide, just so we can get a feel for what does/doesnt work.
• Add local variables to buttons. Some thought might be needed on how to tell users that these won't work in most actions
• Expand this to be minimally 'dynamic', being able to enable/disable layers and put expressions in every field.
• Then it will be time to pickup discussions on what to add to improve the UX, and the other interactions that have been neglected (eg advanced feedbacks)

And I am going to insist that for now this should be its own type, as it means we can push it to the betas/experimental before it is feature complete, and allow testing/experimenting in parallel with other usage. Based on the steps above, I could probably convert 90% of my office/test setup across (I follow the latest betas there, it has found a bunch of bugs I missed during development, so I want to keep it to betas).
I am open to discussing whether it will be released like this when we are nearer to it being ready for general use. I don't think that having both will be a massive cost (short-term), we should try and write the code to be as separated as possible anyway.

[dnmeid]

I can go with the new button being a different type during development, but I'd really prefer it to replace the current button before it is released and not have both in a couple of releases. I don't see a big stopper where we wouldn't have an upgrade path. My main concern is not having two button types in parallel, but having two feedback workflows in parallel. If we make a clean transition, some users will praise it, most users will just take it and some will dislike it. If we offer both workflows in parallel, it will feel much more like we're taking somthing away later instead we're improving it. The frustration will be higher. We can see this now a little with the delay to wait change. Just because the space where delays had been is empty now, many users have the feeling that we have taken something away.
I can see your point of having it in the main branch for better access and testing but if we feel that this will not make it in the next release we should keep it in develop branch. We can merge in main as soon as we are sure that it will be ready with the next release and that means at least some months of testing time in a beta.

I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks

Yep, I agree. It should be able to produce multiple values, with a predefined list of values to expect in the feedback definition. (to allow for cases where today they return a bgcolor and fgcolor or whatever)

I'd rather vote for one value per feedback like a function also has only one return value. Just like the boolean feedback today returns a boolean, but with some companionabel types. E.g. text, color, number, image. Internally we can derive the type either by the type of the value or we pack it in a JSON with additional descripor for ambigous types. I wonder if we should introduce something like ENUM, or how do we want to deal with switching the alignment from a feedback (do we want to have this at all or drop it).
For the UX I'd go more with the atomic approach, when you want to e.g. change a foreground color and a background color, use the same feedback two times. For the real advanced stuff we can allow objects like arrays or JSON as the return type. This would need some skills of the users but it would be the replacement of today's advanced feedback. I'm also fine with this being so complex and actually motivates the module devs to break down the still existing advanced feedbacks into smaller feedbacks.
As a side effect this would lead to the opportunity of a better unification of module variables and feedbacks. A feedback usually is associated with a control and a variable with a connection, but besides that they do the same thing: returning values. A feedback gives a local variable and a variable gives a global variable with the connection namespace. We could use the existing definitions for module variables as well. A current variable would translate into a feedback without options but modules then could also offer to publish user defined variables. Today this already often done with feedbacks. Some feedbacks only exist to generate a variable, but you have to put it on a unused control to work.

[Julusian]

Something else to do as part of this:

Stop using \n in the ui for a line break. We could just as easily turn the field into a multiline field (it already does this when in expression mode), and use real line breaks.

Doing this is messy for presets, but if we do it at the same time as introducing a new preset type, we can leave the old behaviour for old presets, with new behaviour for new presets