(Potential fix for Niri WM) Use wlr-layer-shell for the screenshot overlay window on Wayland

[SuperUserA]

Related bug: #4895 (tile reflow on Niri when Flameshot opens)

On Wayland, the Flameshot screenshot overlay currently opens as a fullscreen
xdg_toplevel. This causes visible tile reflow on tiling window managers
(Niri, Hyprland, Sway, etc.) because the compositor recalculates the layout
when the window is mapped and unmapped.

The proper Wayland approach for overlay-type windows (screenshot tools,
notification daemons, launchers) is to use wlr-layer-shell. This is already
the standard used by:

• dunst (notifications)
• rofi (launcher)
• waybar (status bar)

Using layer-shell would make the Flameshot overlay:

• Floating above all windows (no tiling, no reflow)
• Not requesting fullscreen (no layout disruption)
• Working identically across all compositors without per-WM config

Currently, users on tiling WMs (Niri on my case) must manually add a window rule:

window-rule {
    match app-id="flameshot"
    open-floating true
    open-fullscreen false
}

This is a workaround, not a fix. With layer-shell, no user-side config would
be needed and the overlay would behave correctly from first use.

Scope:

• Replace the fullscreen xdg_toplevel with a layer-shell surface (overlay layer) on Wayland
• Keep the existing X11 behavior unchanged
• The overlay should still cover the full output (or selected output)

This is a medium-scope change (requires adding a wlr-layer-shell client
implementation to the Wayland backend) but could resolve #4895 and
similar reports on all tiling compositors permanently.

[SuperUserA]

@borgmanJeremy This might be worth a look, given the connection with the issue we discussed here #4895

[borgmanJeremy]

The problem is this would be very DE specific and not support X11. If there is a Qt function that does something similar but abstracts it I think that would be a better route.

[SuperUserA]

The problem is this would be very DE specific and not support X11. If there is a Qt function that does something similar but abstracts it I think that would be a better route.

Regarding X11 support: the existing X11 code path wouldn't be changed or affected. Flameshot already has platform-specific handling, and on X11 it uses override-redirect (Qt::BypassWindowManagerHint), which already bypasses window managers properly.

Regarding a Qt-level abstraction: unfortunately, I think upstream Qt does not provide an abstraction for layer surfaces. In qtwayland, all top-level QWidgets map strictly to standard xdg_toplevel surfaces (and Qt::BypassWindowManagerHint is not supported for top-levels on Wayland). This is why even KDE had to write their own external layer-shell-qt library.

However, to address the DE compatibility concern: wlr-layer-shell is supported across Sway, Hyprland, Niri, River, Labwc, and KDE Plasma, but it is notably not supported by GNOME (Mutter).

Rather than making it a hard requirement, the clean approach would be a runtime capability check in the Wayland backend:
1. Check wl_registry for zwlr_layer_shell_v1.
2. If available (Niri, Hyprland, Sway, KDE, etc.), map the capture overlay as a layer-shell surface on the overlay layer (preventing any tiling reflow or window management).
3. If unavailable (GNOME), fall back to the existing xdg_toplevel behavior.

This keeps X11 untouched, avoids breaking GNOME, and completely resolves the tiling reflow issues on tiling compositors without requiring manual window rules.