FX Layers

[mrtripie]

Procedural non-destructive effects that automatically adjust themselves as you draw and automatically respond to animations could be quite useful for speeding up your workflow. It could be used for things such as:

• Outlines
• Cast Shadows
• Glow Effects
• 3D Effects
• Color Adjustments
• Patterns
• Warping effects

I believe I've also seen it mentioned the ability to create custom shader effects is a possible future feature. This will allow for all kinds of custom effects to be created. And this could also be included as part of this feature.

Another potential idea would be particle effects that could be used for quickly creating pixel art VFX such as smoke, fire, explosions, and pouring water, which could then have manual adjustments placed on top of them with standard layers. It may make sense for this to be a separate layer type though as the other effects would most likely all be shader based.

This is how I think it should work in Pixelorama:

• As Pixelorama is a tool that is often going to be used for making games, and many developers will prefer to skip the export process by using a plugin to directly import .pxo files into their game engine, I think it is important to prevent the .pxo format from becoming too much more complicated for other programs to import. For this reason, I believe the format should be something like this:
  • A special FX Layer type. This would be similar to Adjustment Layers in programs like Photoshop/Affinity Photo or Filter Layers in Krita.
  • The .pxo's JSON data for these layers would include the effect settings such as which effect type, the effect's shader params, and if the effect type is a custom shader, which shader to use. (Perhaps these custom shaders would have an option to be packed into the .pxo, or a link to shader file via a filepath).
  • The .pxo file should include the final image data output of these FX Layers. This way import plugins can treat these layers exactly the same as regular layers, just importing the image data as normal. No need to recreate Pixelorama's built in effects, and for engines other than Godot, no need to recreate Godot's shader language to properly import layers with custom shaders.
  • Effects attached directly to a normal layer (such as Layer Effects in Photoshop/Affinity Photo or Filter Masks in Krita) would add additional complexity that I think would probably be best to be avoided. If these are added, maybe it would be best to do it similar to Krita, as a special mask type, that may be parsed similar to regular masks if added to Pixelorama. It would also include the final image data of the effect.
• From the UI perspective, there would probably be an FX+ button next to the Create a New Layer button which would add an FX Layer. As the layers wouldn't be able to be edited manually, there would be no need to need for a lock option, so an FX button could replace the Lock toggle. Clicking the FX button would open a dialog with the effect's options.
• In the options dialog there would be a drop down for the effect type, including several built-in options, a custom shader, and maybe even particles.
• Next if the user chose Custom Shader, they would have the options to choose one (maybe from a file dialog, or something similar to the brushes/patterns selector), create a new one (with a text editor that would have an option to save to file) and options to pack/unpack it from the .pxo file.
• After that there would be any params from the effect: Shader Params such as colors, vectors, scalars, textures, etc. for both built-in and custom shader effects, or particle system options if that's included.
• It would also be nice if there was an option to work based on the layer below, all layers below, if layer groups are implemented, just ones in the same group, etc.
  • My guess is that this might work something like this: There's a viewport that contains a sprite with the layer's texture as the sprite's texture. This sprite would have a material on it with the effect's shader. In the shader you should just be able to use the TEXTURE constant to reference the layer(s) the effect is based on. I'm not sure if there would be a fast way to get the results of all lower layers as a texture though.
• If Pixelorama gets blend modes, it may make sense to include an "Overwrite" blend mode, that would completely ignore the layers below it. This would be useful for things such as color adjustment effects, where if there's semi-transparent pixel, it would overwrite the lower layers, rather than getting blended on top of them.

[novhack]

I really like the idea. I have a few ideas in addition:

• Rather than applying effects to layers underneath the effect layer I would implement effects as specialised layer groups (as described here Layers hierarchy #607) and apply effects to all layers that are children of this effect group.
• I would like an ability to bake effects so they are permanently applied to layers they affect.
• This whole idea of FX layers is a nice superset for my proposal Lighting system and normal map layer groups #617 and I think that it could be just another type of effect layer.

In regards of GUI:

• I would not show effects in the main canvas. We have a second preview next to the main canvas which would be awesome to display effects applied.
  • This would make it possible to work in an unaffected main canvas and see all effect changes applied in the preview next to it.
  • We could allow users to also draw in this effects preview but I would mirror the cursor in the main canvas to make it clear where pixels are actually being placed - kinda like tiling mode has it now.
  • If users really wanted to have the effects view in a place of the main canvas I would allow it with some toggle button but I would make it very clear that the main canvas and effects preview are switched.

[mrtripie]

Trying to mockup how this might look.

I think there's a bit of a struggle with the Timeline/Layers UI to fit everything in if FX Layers are a type of Group Layer:

• Group Layers want a collapse toggle, a visibility toggle (for all child layers), and maybe a lock toggle to lock all children.
• FX Layers want a visibility toggle, and a FX Options button. I'm not sure if it wants a linked cels enabled button.
• FX Groups want all those combined, collapse, group visibility, fx visibility, fx options, and maybe a lock and a linked cels, which is a lot. I didn't put them all in the mockup.

FX Layers could be combined with Group Layers where an FX Layer in a group will only use layers below it that are inside its group. They could also have in their options an option to use all layers below, or just the first one below.

If you want an FX Group to use the whole image, it will move all the layers down the hierarchy, eating up horizontal space.

Of course, using a Group Layer to limit an FX Layer will eat up extremely valuable vertical space, so maybe it is better to just have a ton of options on an FX Group.

---

As for a button for converting either an FX Layer to a Pixel Layer, or an FX Group to a regular Group (optionally modifying the layers below to match the effect), maybe it would make sense to replace the Merge Below button with that button when you have an FX Layer/Group selected.

---

For viewing the FX in the side canvas, I'm not sure where to put the toggles, for the mockup I have them just slapped in the corner of the canvas (I imagine them only showing up if you have an FX Layer in your project).

I've imagined before using the side canvas for a tilemap preview, so maybe it could have different "Preview Modes", a Normal mode, a Tilemap mode, and a FX mode (when in this mode, and the side canvas is visible, then turn off FX in the main canvas, and maybe have a FX option in the view menu for when its not in this mode).