Fix jittering during window resize on MacOS for WGPU/Metal

[aspcartman]

Overview

As mentioned in #903 the window resizing on macos in egui was hit by something heavy some long time ago due to, probably, changes in winit 0.29->0.30. I have not been able to narrow down to the root cause of that back then. But even though the issues were reported for glow initially, they are present with winit/wgpu/metal triplet as well today.

A.mp4

But it appears for the metal the solution is well known. (thx @mattia-marini #903 (comment))

The CAMetalLayer - one that wgpu draws to - has to draw the frames in sync with CoreAnimation, framework that drives the whole GUI rendering pipeline on apple platforms. The desynchronization between the two pipelines lead to the visual jitter. CA is unaware that a new frame is coming and tries to stretch the existing window content.

This behavior is controlled by presentsWithTransaction flag. The support for it has been implemented in wgpu many years ago, 2021 it seems, including the needed drawing logic alterations, and is exposed to the metal backend users as a pub field in the metal surface struct. Note it was not escalated to higher levels of abstraction as this is a backend-specific feature.

Yet enabling it comes with side-effects, best described in Zed GPUI Article and potential latencies (@Wumpf #903 (comment), gfx-rs/wgpu#8109). The metal layer is desynced with CA for most performance and control, but that limits the ability to play well with native UI elements, including window itself.

So the proposed solution is to enable the present_with_transaction only during the resizing - same solution as Zed did. I've made a prototype and.. Well. It works as expected.

B.mp4

I have made changes to egui-wgpu: made Painter responsible for toggling the surface properties in response to window resize begin & end; and to eframe wgpu integration: detection of resize start and end w/ pokes to the painter.

Things to address

• Metal surface struct field access
• Conditional compilation
• Confirm resize begin-end detection is fine
• Wrap up the code for merge

Accessing the field

The field is currently accessed through a pointer cast. The API is specific for Metal and thus is not exposed to upper levels, so if one has no typed handle to the metal surface struct one has to downcast.

    unsafe {
        if let Some(hal_surface) = surface.as_hal::<wgpu::hal::api::Metal>() {
            let raw = (&*hal_surface) as *const wgpu::hal::metal::Surface
                as *mut wgpu::hal::metal::Surface;
            (*raw).present_with_transaction = present_with_transaction;
        }
    }

I think it won't change on the wgpu side in the future. It's a stable API for 4 years already :)
So having this abomination around might need an explicit approval. Personally I am completely fine with it.

Conditional compilation

The wgpu::hal::api::Metal and wgpu::hal::metal::Surface are under metal flag in wgpu crate. Currently egui does not import wgpu backend by default or has feature flags for that. That's what defaulting to wgpu in eframe & #7615 recently hit into.

This PR also needs backend feature flags in egui for it to work as it depends on those metal types. The

#[cfg(all(target_os = "macos"))]

is not enough and code won't compile if wgpu/metal is not enabled. But it's impossible to cfg('wgpu/metal') afaik. Any solutions? Should it wait @emilk for the metal flag?

Window lifecycle events

Winit does not provide events for window resizing other than just a plain Resized(new_size). In context of the issue at hand it's required to be aware that resize is in progress across frames. It seems that during the window resize winit fires an exclusive stream of Resized events. I have assumed that only one window viewport can be resized at a single moment of time and added tracking of it's id. When an event arrives and it's not a resize - well, resize has ended it seems.

It is safe for a random event to occur during the resizing and trigger the end prematurely. That will only lead to a possibility of a visual jitter occurrence at that frame.

Is it fine?

---

PS: So happy this thing stopped lagging on resize, it's a miracle. ^_^
PSS: The current draft will not build in CI because of the feature flag thingy.

[github-actions]

Preview available at https://egui-pr-preview.github.io/pr/7641-osxmetalresizejitterfix
Note that it might take a couple seconds for the update to show up after the preview_build workflow has completed.

View snapshot changes at kitdiff

[aspcartman]

@Wumpf would love to hear your feedback on this =)

[Wumpf]

thanks for setting this up! will have a more thorough look and think in the coming days

is not enough and code won't compile if wgpu/metal is not enabled. But it's impossible to cfg('wgpu/metal') afaik. Any solutions?

You should be able to poke the wgpu adapter/device to ask it at runtime whether it's a metal device 🤔
Checking for the feature flag is technically not enough anyways since someone could run with GL or MoltenVK.

Edit: Ah sorry, you already have that check via cast. The problem is that you don't know whether the types are available. That's nasty 🤔

[emilk]

This looks very promising! Thanks for working on this ❤️

[Wumpf]

Thanks for investigating this deeper and coming up with a !

Played around a bit with it and noticed that egui's fps counter goes up during resizing. In fact if I always enable present with transaction, it renders 240fps instead of the expected 120fps in continuous mode. That kinda explains how this can work: the screen (and presumably resize events) happen at 120fps and we're practically running into a sampling problem of the currently correct screen size. Fascinating!
That also implies that the concern about latency isn't exactly warranted, well that is if your application can keep up.

Unfortunately the jitter is still there a liiiitttle bit on external monitors. But on the builtin one it's gone completely. Interestingly, it's the entire window jittering which never happens on the internal monitor - notice how the top left of the windows moves as well (running via usbc @ 120hz):

occasional.jitter.on.external.monitor.downsized.mov

The metal layer is desynced with CA for most performance and control, but that limits the ability to play well with native UI elements, including window itself.
So the proposed solution is to enable the present_with_transaction only during the resizing - same solution as Zed did. I've made a prototype and.. Well. It works as expected.

Reading through the material I come to the same conclusion as well so far and agree that the proposed solution is the way to go forward. Let's make sure to have everything documented in the code!

I can explore a bit how we can expose this in wgpu directly more easily, but the next time we can land semvar breaking changes is end of december, would be nice to figure this out without that :)
On that note, in egui_wgpu's codeflow I'd really like to have that setting be part of configure - changing it requires reconfiguration and while I think what you wrote is correct (i.e. should never change the setting without reconfiguring) it seems a bit brittle. Similarly, if we were to expose this proper in wgpu, I think it should be some form of backend specific parameter on wgpu::SurfaceConfiguration 🤔

[Wumpf]

Looking around a bit, doing platform specific features on configure in wgpu will be a breaking change and one we'd have to debate a bit first (are there other platforms having it, does it make sense to maintain such a hook there, etc.). And really to solve the problem at hand we'd need to have some notion of metal features outside of metal-feature-gated code.
Pretty much the only place in wgpu (outside of hal related things) that has this kind of explicit per-backend information is the instance backend options and that one is ofc not suitable.

Either way, we're probably better off either..

• just always enable the metal feature in egui_wgpu for macos(/ios/tvos/visionos ?). It has no effect outside of metal supporting platforms and on the flipside it's highly unlikely you don't want a Metal backend on those platforms
• add a new feature flag, something like smooth_metal_resizing, which drags in wgpu/metal. It's on by default (bc of above rationale) and is what we check for transactional present. That allows at least some way to still opt out

[aspcartman]

I was not able to understand why the fps increases 2x. I tried to poke around w/ the debugger, even into places I am not supposed to poke debuggers in. No avail. I find it weird as I expect CADisplayLink (my educated guess what drives the render loop) to be locked to current display preferred FPS. In the power safe mode it's supposed to be 60fps, not 120fps it does.

That kinda explains how this can work: the screen (and presumably resize events) happen at 120fps and we're practically running into a sampling problem of the currently correct screen size. Fascinating!

I doubt that it's a sampling problem caused by fps mismatch but rather, I think, it's just about 'clock' sync. But that's only intuition I can put on the table as I have no data to support it. Maybe it can be checked by increasing FPS forcefull w/ (EDIT: without!) present_with_transaction and see if the resize jitter gone? No idea how to do that though. Also force-lowering FPS with that thingy enabled can provide some insight..

But anyway. As for the solution, I agree. I want to spell out the possible ways to go:
A) wgpu exposing the feature in the non-backend-specific configuration as a generalized optional flag like "syncs_with_window_render_loop` or something.

• Removes the need to of the pointer-casting-abomination.
• Maybe the flag is useful for opengl too? The jittering is still there for that target and maybe there's a similar fix possible for it?
• Maybe it even won't be an API breaking change?

B) Introduce the metal feature flag to egui/frame.

• I think that would also solve the Make wgpu the default renderer for eframe and egui.rs #7615 issue.
• Also that means exposing all the possible backends of wgpu as well there. It makes sense from the ergonomics perspective, but allows the user to have a broken configuration: he can enable metal on wgpu crate, but not enable it on eframe crate. That combination has to lead to a compilation error.
• In cases where users do not use metal on osx, that will add the later + possibly force-switch them to metal accidentally (I guess...). Right now the metal/opengl is chosen by a flag in wgpu. eframe/metal + wgpu/opengl => compilation error? Breaking change, but for eframe maybe thats fine.

C) Introduce the osx_smooth_thingy flag.

• But I find it to be a dirty solution.
• Opens the door to a series of bugfix-feature-flags
• Cheapest option of all

D) Enable wgpu/metal on osx target as required dependency.

• What if the user does not want to metal around? Rare, but for games it makes sense.
• Enforcing stuff is fun.

I'd vote for (A) if it can be done as a non-breaking API change, and (B) as it fixes the defaulting to metal issue and allows future backend specific configurations. And those are not muturaly-exclusive. What do you think?

Lastly we are ignoring an elephant in the room. Why on earth the winit 0.29 just works out of the box, what have happened there? It was fine for all backends back then. Now it requires us to do the fancy dancing. As we are considering the options above I think I should go and compare 0.29 & 0.30:

• confirm it's all good for all backends and just breaks on winit 0.29->0.30
• take into account egui changes - it's huge
• figure out the reason

Osx/ios development experience helps there, still it will take me a while.

[Wumpf]

A)

Maybe the flag is useful for opengl too? The jittering is still there for that target and maybe there's a similar fix possible for it?

It uses egl, no idea if any of this is applicable there. I kinda doubt it. But also Apple has marks OpenGL support as deprecated in quite a while now...

Maybe it even won't be an API breaking change?

no longer seeing how we could pull that off, very tricky

B)

I think that would also solve the #7615 issue.

It can be part of it but it won't solve it single handed. The problem is that we want users to pick what wgpu backends they use but come with nice defaults. See also #7344

Also that means exposing all the possible backends of wgpu as well there.

We should absolutely not mirror all wgpu feature flags if we can avoid it. Clutters the feature list with things that are perfectly fine for wgpu to own & document. Also, don't get why we'd have to - the thing we want to potentially feature gate is the special metal-resize behavior.... which is why I think the C solution is preferable ;-)

she can enable metal on wgpu crate, but not enable it on eframe crate. That combination has to lead to a compilation error.

don't follow. If someone enables wgpu but doesn't enable it on any egui crate then egui doesn't know about metal, i.e. the resize fix here won't be enabled. But that shouldn't cause any complication issues per se?

C)

But I find it to be a dirty solution.

🤷 😉
I mean the trick would be that we enable it by default, the only reason we have it is to retain the ability to opt out of metal if you really have to. Which is unlikely to happen.

D)

What if the user does not want to metal around? Rare, but for games it makes sense.

Agree that it's rare but just as a sidenotes I think for games it makes the least sense. The main scenario I can think of is that you have a non-game application and therefore don't care about the metal backend and rather run OpenGL on all platforms to avoid surprises.
OpenGL in wgpu is more of a fallback backend and the least well working in general (disregarding WebGL which is a bad subset of something bad in this context ;))

---

Why on earth the winit 0.29 just works out of the box, what have happened there?

Yeah that would be good to investigate :/. No clue! Would be indeed awesome if you could unearth some insight there!

[aspcartman]

I am happy to announce my investigation of the issue in winit has failed in an exceptionally miserable way and I continued with the C) option I least adore.

The PR is prepared for review.
I have collapsed the window resize lifecycle methods down to one for clarity, added comments, and allowed rustfmt to juggle the import statements order as if it asked me in the first place.
I have added new feature flag macos-window-resize-jitter-fix to egui-wgpu that pulls wgpu/metal and enabled it by default.

Please note:

1. That means that now wgpu/metal will be enabled by default if user has wgpu enabled in, say, eframe.
2. I have not collected enough mental capabilities to imagine how exactly could user opt-out. If user enables eframe/wgpu he'll get the default feature set of it. An attempt to specify egui-wgpu = { default-features = false } in one's Cargo.toml achieves outstandingly nothing.

Other than that - seems it works. ^__^

[aspcartman]

By the way, how is this handsome FPS stats overlay called, how to enable it? : )

[Wumpf]

I have not collected enough mental capabilities to imagine how exactly could user opt-out. If user enables eframe/wgpu he'll get the default feature set of it. An attempt to specify egui-wgpu = { default-features = false } in one's Cargo.toml achieves outstandingly nothing.

huh yeah right now you could only opt out if you depend on egui-wgpu but for some reason not on eframe. Kinda limited. Ah well we can take care if that if someone complains. Very unlikely...

[Wumpf]

By the way, how is this handsome FPS stats overlay called, how to enable it? : )

Learned it from one of the articles you linked :). It's MTL_HUD_ENABLED=1

[Wumpf]

really nicely done, thank you!

[Wumpf]

this part can go outside of the unsafe scope, no need to have that longer than necessary

[Wumpf]

.. but it's actually annoying to do so with the condition an all. Whatever (:

[aspcartman]

That's exactly what I have been moving-back-n-forth for 15minutes w/o any even remote satisfaction, agree.