feat: overhaul drawing

[Julusian]

Starting point for the button drawing overhaul #3293

Let me introduce the worst ui you have ever seen:

But it works, and is a starting point

cc @dnmeid

Required follow ups before this can be released:

• Deal with any TODO-layered

[Julusian]

The UI is making a bit more sense now. Leaving it at that for today

[dnmeid]

Wow, you're fast. I wanted to start that stuff for so long but always had an excuse for not doing it. I'll try to add some pieces from time to time.
I had a brief look at it, especially the interface.

First of all I think the way we are looking at the concept now, mabe we are using the term "layer" in a wrong way. Usually in a drawing app you are using objects or elements and you can arrange them in layers (and groups). If we are going to have only one element per layer, we could technically still call it a layer (like e.g. openGL does), but I'd recommend to call it an element instead so we won't have a hard naming time if we want to introduce real layers later.

I suggest to drop the "canvas" and replace it with a "matte". A matte is an element that always fills the whole area of the button (no adjustable size) and draws a color or maybe later a gradient.
All elements should be optional. Without any element you will get a transparent image. On a Streamdeck the base color will be black, so if you want to have a different background color, you need to add a matte. I envision that this graphics will also be combined later with different backgrounds, e.g. use the graphics of one control as an element in a diferent control. Or webbuttons or satellite may want to have some different overall background...

I would make the topbar a separate element. So the user is free to add it or not, or reposition it in a later version or maybe we add some styling options in the future. I think "statusbar" would be the better name then. Later we can separate the yellow border from it, for now as long as there is no position or size control all status decorations can stay in one element.

For the image element we should take care from the start that there will be more possible formats than PNG. Just for the naming.

For the UI I suggest to show the topmost layer on the top and the bottom layer at the end of the list. I'm not sure if in the definition we should have the same order, I tend to say technically it makes more sense to have the lowest element first in the array because it is drawn first. So only the presentation would change.
The add buttons should be above the list.

[Julusian]

Wow, you're fast. I wanted to start that stuff for so long but always had an excuse for not doing it.

A lot of it is very early at this point, I did a draft of the renderer using the existing Image class and didn't test it much until the ui was functional. And at this stage the functionality of what each 'layer' can do is limited to what we currently support.

And thanks for this input, this is exactly what I was hoping would happen. I find it much easier to make something quick and crude and refine it being able to talk about concrete things rather than large concepts/ideas.

First of all I think the way we are looking at the concept now, mabe we are using the term "layer" in a wrong way. Usually in a drawing app you are using objects or elements and you can arrange them in layers (and groups). If we are going to have only one element per layer, we could technically still call it a layer (like e.g. openGL does), but I'd recommend to call it an element instead so we won't have a hard naming time if we want to introduce real layers later.

Good point. Yeah I am happy to rename it all to elements. I would be tempted to refer to the 'layer group' as a layer then (once that exists)

I would make the topbar a separate element. So the user is free to add it or not, or reposition it in a later version or maybe we add some styling options in the future. I think "statusbar" would be the better name then. Later we can separate the yellow border from it, for now as long as there is no position or size control all status decorations can stay in one element.

I'm not sure about this. While I can see the appeal of what you are proposing, whether the top bar is shown affects the vertical position/height of other drawing. So this would mean we either need to drop this behaviour, or accept that any layer can affect some basic properties about the canvas, rather than just the first 'layer'.
If if its own element, then should the user be able to add it multiple times?

I suggest to drop the "canvas" and replace it with a "matte". A matte is an element that always fills the whole area of the button (no adjustable size) and draws a color or maybe later a gradient.
All elements should be optional. Without any element you will get a transparent image.

This kind of ties into my thoughts about the topbar. I am using the canvas element as a place to dump any settings that affect how this is drawn. I think that this is something that will be useful.

I am open to a matte type, it will probably make more sense once we can do gradiants, to limit where we clutter with these properties. But again it feels like one of those types where the only valid place to use it is as the second layer, so its just extra steps?
I suppose I am not thinking very forward here, but it will be much easier to split a layer out later than try to merge them.

For the image element we should take care from the start that there will be more possible formats than PNG. Just for the naming.

Yep, I made sure to name the it generically. It didn't accept a jpeg when I tried it, but I didn't look into it. Might be a stray guard somewhere checking for PNG, the actual drawing won't care since we dropped pngjs and let the canvas handle parsing image buffers.

For the UI I suggest to show the topmost layer on the top and the bottom layer at the end of the list. I'm not sure if in the definition we should have the same order, I tend to say technically it makes more sense to have the lowest element first in the array because it is drawn first. So only the presentation would change.
The add buttons should be above the list.

Yeah, this part of the ui has had no attention yet, just dumped out some html to be able to prove it worked.

[Julusian]

Looking much more like a proper ui now

[dnmeid]

I'm not sure about this. While I can see the appeal of what you are proposing, whether the top bar is shown affects the vertical position/height of other drawing. So this would mean we either need to drop this behaviour, or accept that any layer can affect some basic properties about the canvas, rather than just the first 'layer'.

At the moment there is no possibility of user positioning or sizing an element. So I guess the current system reflects the behavior of the old buttons, that means a text or image element is sized according to if the topbar is shown or not. So in this regard it wouldn't make any difference if the topbar is an elemet of its own.
At the time we introduce position/size we will have to deal with that anyway, regardless if the topbar is part of a canvas or separate.
I would drop the show/hide topbar automation and replace it by some style presets. The style preset would be a direct replacement for the selection of the button type "page up", "page down", "regular". We can offer premade sets of elements that reflect the same button look and behavior. A "page up" would add the arrow or plus as an element and a text element with the page name and a text element with the page number. Instead of one simple button we can offer a preset for a button with the topbar and a preset for a button without a topbar. The amount of clicks stays the same, no matter what you choose. Also this is compatible with current button preset definitions, where modules can set this property.
When you decide later to change the style, you manually have to adjust the elements. So changing this would be slightly more effort, but the overall process of button styling will consume more time, e.g. for adding an image you first will need to add the image element and then select the image.

Users that really want to have a show/hide topbar automation can use a custom variable and make the style dependent on the variable, e.g. text element y-position = $(custom:showtopbar) ? 16 : 0, topbar opacity = $(custom:showtopbar) ? 1.0 : 0.0

BTW: having talked about style presets. Actually this can be just a preset offered by the internal module and the "create button" dropdown can remain working as a way of assigning these internal presets to a control.

If if its own element, then should the user be able to add it multiple times?

Yes, why not. I would try to make as less exceptions as possible. It may not make sense to add five topbars, but I'd allow it. I'd say all elements are optional and all can be added as often as you like.
Maybe when we have groups/layers/containers/divs (whatever we will call 'em) a user wants to use these as different graphical states and he wants to have three groups containing a topbar and one group without a topbar.

[Julusian]

At the time we introduce position/size we will have to deal with that anyway, regardless if the topbar is part of a canvas or separate.

I intend to do this before calling this overhaul ready for general usage, so I think we should think about it as if it was already implemented.

Users that really want to have a show/hide topbar automation can use a custom variable and make the style dependent on the variable, e.g. text element y-position = $(custom:showtopbar) ? 16 : 0, topbar opacity = $(custom:showtopbar) ? 1.0 : 0.0

Does this mean that we won't support auto vertical centering the text anymore? Unless all the drawing is aware that there is a topbar, then numbers will need to be manually adjusted by a pixel amount (and we want to discourage using pixel values) using numbers that users are just expected to guess?

Sure, we can make the default button be pre-configured, but then users will need to think about this offset for any change they make. Changing the text alignment to bottom, now they need to zero that value. Or to top, and they need to put it to 16. Or to middle, and to 8.
Add a new image layer, and set the value yourself.

Having to compensate for this would be incredibly frustrating, and a step backwards

The style preset would be a direct replacement for the selection of the button type "page up", "page down", "regular".

I'm very open to this idea as a follow up, I'm not sure about whether it should for the top bar setting or not.

[dnmeid]

Does this mean that we won't support auto vertical centering the text anymore?

No, I'd like to continue that. The text element has an overall position and dimensions and the alignment of the text is done within these boundaries.

Unless all the drawing is aware that there is a topbar, then numbers will need to be manually adjusted by a pixel amount (and we want to discourage using pixel values)

No, the preset would give you a button with the correct numbers in it. Right, we should have this expressed in percent instead of pixels, but the logic is the same. The topbar fills 18%, then there is 82% remaining for the other elements.
Maybe I should add another thought I have on the group thing. It should work more like a <div> and have its own position and size. Maybe we should call it a "container"? The position/size of the elements within that container should be relative to the container. That means when our preset has a container for the part not occupied by the topbar, the user can easily add elements to it and they will all automaticall fit 100% of the container.
As far as I know typical user workflows most of them do not often change the topbar visibility. They adjust their preference and then maybe they'll have a few special buttons with different topbar setting.

using numbers that users are just expected to guess?

Well, I think that every element has some initial values when you add it. When the statusbar is a single element without other decorations, I think it will have X:0, Y:0, W:100%, H:18% by default. So you have a hint what the numbers are.

Sure, we can make the default button be pre-configured, but then users will need to think about this offset for any change they make. Changing the text alignment to bottom, now they need to zero that value. Or to top, and they need to put it to 16. Or to middle, and to 8.

I don't get this point. The text alignment is not related to the outer dimensions or position of the element.

[Julusian]

Every field can now accept expressions. I reworked the data structure to be able to represent this more cleanly.

In this fun example, the button is redrawing every second with the font size constantly changing

I think my next step is going to be refactoring the drawing and working towards the button preview in this editor being rendered client side. That way it will be easier for users to play with different aspect ratios, and for us to draw some additional lines to indicate boundaries and things

[Julusian]

Maybe I should add another thought I have on the group thing. It should work more like a

and have its own position and size. Maybe we should call it a "container"? The position/size of the elements within that container should be relative to the container.

Right yeah, that makes more sense now, that would work and be useful anyway. But I would worry that forcing the user to make sure that every element they create ends up in this container might be an annoyance.
Even if we add buttons to be able to add elements directly inside containers, it will be very easy to click the wrong one and then wonder later why the text on some buttons isn't vertically centered.

But my technical concerns would go away with containers, so this really comes down to do we want the topbar to be a fixed/rigid part of the button drawing process, or do we want it to be an element that the user is free to place and move around freely, and to almost always force use of a container.

I don't get this point. The text alignment is not related to the outer dimensions or position of the element.

With containers this is void; or I suppose a proper way of defining the bounds of some text would negate my thoughts here too

[dnmeid]

this really comes down to do we want the topbar to be a fixed/rigid part of the button drawing process, or do we want it to be an element that the user is free to place and move around

Correct, so the options are:

• leaving the topbar (or other decorations) something that the button does when switched on and elements can ever only use the remaining space
• offering always the whole button area to elements and the topbar is an user adjustable element itself

First of all I'd not say that the topbar as a separate element would force the use of a container. A container will make it easier for other elements to stay within the area not occupied by the topbar, but even without a container you can do it. Maybe you put a matte in the background, a text element on top of that and the topbar on the very top. You only have to tweak the position or size of the text to your liking and that doesn't necessarily needs calculating pixels or percent, you can just shift it until it looks good.
Also I would say that the possibilities of a container compensate for the overhead effort of using it.

I think the biggest advantage of a fixed topbar is that it remains possible to switch the topbar on and off and the remaining button just rescales automatically (if the elements used percent).

On the other hand the biggest advantage of a separate topbar is the styling flexibility. You can position the bar at the top or at the bottom or somewhere else. You can have some containers with a topbar and some without and switch between them programatically (e.g. per step). You can resize the topbar to your liking. Currently it is 18% of the height, when I think of e.g. a 640x400px display, I don't think that a 18% topbar will be what you want. Maybe you want it to be only 5% high and only 20% wide.

We could of course offer both possibilities, the fixed topbar and a statusbar element at the same time. I just think then we are carrying unnecessary code and UI stuff over, because actually only one system is needed. Maybe it's an idea to have both possibilities for a testing period and then we see what we like more or how some test users react before releasing it.

[Julusian]

First of all I'd not say that the topbar as a separate element would force the use of a container. A container will make it easier for other elements to stay within the area not occupied by the topbar, but even without a container you can do it.

Yes, that is all true, but I would probably recommend to do to to make things easier.

I am thinking that the default layers on a newly created button should be the 'canvas' layer (if that remains), a 'matte' layer (if that exists), a 'text' layer, and this status bar layer. The text layer might be in a container, to encourage users to add their stuff there for easier positioning, if not the text layer will need the position adjustments.

The probably most common change to make to this layer structure will be adding a background image. Which means adding a layer, moving it into the container or adjusting its height and moving it below the statusbar.

Maybe the 'solution' to my concern is to also create that image layer by default. Then to match the functionality of today it wont require fiddling with layer order, or positions.

You can resize the topbar to your liking. Currently it is 18% of the height, when I think of e.g. a 640x400px display, I don't think that a 18% topbar will be what you want. Maybe you want it to be only 5% high and only 20% wide.

I hadn't considered displays of this kind of size. I was expecting this to be drawn at a fixed height still (which yes, adds its own problems of mixing percent and px)

Maybe it's an idea to have both possibilities for a testing period and then we see what we like more or how some test users react before releasing it.

I can get behind this idea. Supporting the old flow is very little code, seeing as it is used for the old buttons and simply adjusts what the min/max x/y values are.
And until all this code lands in beta I will not be taking any care to not break the data structure.

Regarding the matte type, since containers, I think this is an appropriate way to go as this will start to go towards being able to draw regions in different colours.
Should it be 'matte' or 'solid' or 'rectangle'? This is the start of different drawing primitive shapes, so we should probably name it in a way that will make sense once there are other shapes

[dnmeid]

Should it be 'matte' or 'solid' or 'rectangle'?

Let's collect some ideas for elements to see how tey can build a useful set with distinct functionality.
Top priority

• Container: Can hold other elements. Has X,Y,W,H. Maybe rotation? Child elements use the extends as their 0%-100% range, but can also draw outside of the container area. E.g. container x is at 5px and child x is at -5px, actual x will be at 0.
• Matte: draws a color or a gradient (maybe gradient as a separate element?). Has color. Does always draw to the extends of the parent element
• Box: Has XYWH, rotation, fill color/gradient, border width, border color, border alignment, corner radius?. Maybe we make the box the container? Easier sometimes but more unused options most often
• Text: Has text, font size, font, horizontal alignment, vertical alignment, color, style, line height. Maybe later animation like scroll? Maybe enable interpretion dropdown for plain|md|html, maybe formatting option like in excel.
• Image: Has XYWH, rotation, image, alignment horizontal, alignment vertical, crop/fit. Can use different bitmap and vector image formats. Maybe gif support? (usually super ugly but highly requested by users). Maybe Images are taken as a reference from a image storage. Drag and drop from the OS should be possible either to the image option of an existing element or directly in the layer stack.
• Statusfield: Has XYWH, rotation, color, border color. Inner elements will adopt to size/aspect of element.

Medium priority

• Line: Has X1, Y1, length, angle, width, color, style, cap start, cap end
• Gauge: Has XYWH, rotation, value, style (horizontal, vertical, circular...), inactive color, active color, indicator width, indicator color, border width, border color
• Circle: Has XY, diameter, angle start, angle end, fill color/gradient, border width, border color, border alignment
• Icon: Has XYWH, rotation, icon from a super exhaustive set of vector icons, overlay color

Low priority

• Gauge: add possibility to use custom style for power users or to export from presets
• Videostream: like image, but can show a videostream (e.g. CITP, NDI, RTMP)
• Web: Has XYWH, rotation, scale, refresh. renders html from URL (probably with CSS but without JS). This may add limited value to a tiny button but will be very useful at larger areas
• Reference: Has XYWH, rotation, source, element, alignment horizontal, alignment vertical, crop/fit. Can reuse other containers or elements of the same graphics or from other buttons or complete other buttons

Any more?

All elements should have a Name/ID, Show/hide toggle, opacity and draw mode.
When you select an element, an outline should be shown in preview.

[Julusian]

Re that list, I'm not sure I see the value in matte, it looks like it is a subset of box, so may as well just use box?
I think that everything should have XYWH, including text. Because it will act as the bounding box, for text that means reducing the region for autosize to try filling.

When you select an element, an outline should be shown in preview.

yeah this is why I wanted to figure out client side rendering, I'm not sure what you mean by outline, but at the very least we should show XYWH bounding lines for the selected layer, hopefully drawn a few pixels outside the canvas so that they are always identifiable

---

Now playing around with client side rendering, which is working, and has some 'fun' results due to the fixed number calcs

[dnmeid]

I'm not sure I see the value in matte, it looks like it is a subset of box, so may as well just use box?

Yep, matte is a simplified box. My thought is that people would use a box with 100% size and without a border a lot as it translates to the old background. So maybe it is worth having this as an element that doesn't eat up much screen real estate in the list as a full featured box.
We can start with implementing the full featured box and see how it feels though, maybe I'm wrong and it's better to have only one element of this kind.

I think that everything should have XYWH, including text.

You're absolutely right with the text. I guess I missed it there.
But do you mean also the circle or the line with everything? I could go with the circle. It then would be more like the ellipse tool of Powerpoint. I prefer the XY for the center though.
For the line I don't see XYWH. It would actually translate to X1, Y1, X2-offset, Y2-offset. So a line from 20,40 to 50,25 would have to have 20,40,30,-25. I think that feels awkward. I think a lengh/direction approach would be best for the typical usecases I can think of when it comes to expressions. If you want to move the line, you need to adjust only one point and rotation is much easier with lenght/direction.

I'm not sure what you mean by outline, but at the very least we should show XYWH bounding lines for the selected layer, hopefully drawn a few pixels outside the canvas so that they are always identifiable

Exactly that

[Julusian]

But do you mean also the circle or the line with everything? I could go with the circle. It then would be more like the ellipse tool of Powerpoint. I prefer the XY for the center though.

no youre right, not everything, but many/most things. I suppose I meant that everything should in some way define its bounds/size (in the case of a line, that will be the two end points), nothing should only be able to fill the parent

---

But ok, I think this is getting along well as a POC. It is in no way polished or complete, but enough is done to be able to prove the concept and flow.

The preview drawing looks reasonably sane, but not 100% correct (there are some hacks in there, such as skewing the font size), and something is off in the scaled topbar maths.
It now also has fonts client side (the emoji one is playing up, or that could just be firefox)
And it is using 'live' values for any expressions (it is setup so that the client can subscribe to any expression and be notified when it changes)

I think the next step is getting feedbacks integrated into this structure. Based on my notes in the discussion, that means we need local variables to be able to handle boolean feedbacks, with advanced feedbacks being TBD.
I am starting to wonder about when would be best to merge this PR into develop, it is already pretty large in LOC

[Julusian]

I've had an idea regarding satellite and surfaces which can only do a 'simple mode' color and text, that I want to write down but may not want any discussion yet.
While we can try to pick the best element to use for this, we will often be 'wrong', so the user either needs to be able to specify which it will use, or needs a way to override it.
So perhaps on the canvas layer is some properties used for just this purpose. These won't be drawn in the normal way, they will just be data that we pipe out for the surfaces. Default should be automatic logic to pick the top/bottom elements, but with this opt-in override.

I see this as being needed for:

• some satellite devices
• xkeys (they can only do solid/flashing colours)
• blackmagic atem micro panel (only solid/flashing colours)
• some buttons on the loupedeck/streamdeck (only solid colours)

The other option for users would be to learn which elements the logic will pick, and to create some new elements which are positioned outside the canvas just to feed these surface types. Doable, but annoying and error prone. And as it is not uncommon for surfaces to have non lcd buttons, we should really cater to it nicely

Again, I am in no rush to do this, this is very much a polishing stage thing to do, currently all these protocols will pretend that these new buttons are blank. And I intend to keep the canvas element around, as I'm confident there will be something to go on it later

[dnmeid]

While we can try to pick the best element to use for this, we will often be 'wrong', so the user either needs to be able to specify which it will use, or needs a way to override it.

Yes, I've been thinking about that too and come to the same conclusion. And it is the same with the conversion of old buttons to new buttons or the mapping of the existing API or existing styling actions.
My thought is to do all of this in the same way. I'd go with some special element names. E.g. when the user calls a text element Text, we know that a set text action should set the text of that element. For surfaces we can publish what the keywords are, e.g. LED for a LED background.
When we don't find an element with the special word, we still can do the best effort guess.
We should give guidlines to the surface module developers so that there is some consistency in the names and also in the interpretation. E.g. if I have a 7-segment display, even if the element to use is clear, there should be a comon standard in how to interpret the data then.

[dnmeid]

we need local variables to be able to handle boolean feedbacks

I just had this coming to my mind.
When a feedback stores its result in a local variable and the options of a feedback can be variables (or expressions), wouldn't it be possible to write a feedback loop with potential hazardous results?