From bc14a8299fa4c3eca2bf26a133ad7b54c64b1a29 Mon Sep 17 00:00:00 2001 From: Carlos Garcia Campos Date: Wed, 25 Jun 2025 14:28:44 +0200 Subject: [PATCH] Add initial documentation about graphics in GTK and WPE ports --- .../WebKitGTK and WPE WebKit/Graphics.md | 142 ++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 docs/Ports/WebKitGTK and WPE WebKit/Graphics.md diff --git a/docs/Ports/WebKitGTK and WPE WebKit/Graphics.md b/docs/Ports/WebKitGTK and WPE WebKit/Graphics.md new file mode 100644 index 00000000..9ae306e5 --- /dev/null +++ b/docs/Ports/WebKitGTK and WPE WebKit/Graphics.md @@ -0,0 +1,142 @@ +# Graphics + +This document describes how the web contents are painted into the final output in GTK and WPE ports of WebKit. The goal is to explain how things currently work, however to understand some of the concepts it's important to know a bit of history and how we ended up with our current model. + +1. Non-accelerated single process model: At the beginning the graphics model was quite simple, all web contents were rendered into a graphics context in a single process. +2. Multi-process model: Then, the multi-process model was introduced ([WebKit2](../../Deep Dive/Architecture/WebKit2.md)) and the rendering model was more or less the same but in a web process. So, we had to find a way to send the rendered contents from the web process to the UI process. The web process rendered into a Cairo image surface and the resulting pixels were copied to a shared memory buffer that the UI process read to create a Cairo surface to pass to the GTK widget. +3. Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split into layers that were rendered with Cairo image surfaces and then upload as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output is now not on main memory but on GPU, and copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget. +4. Wayland: The way to share composited frames with the UI process was X11 specific, so when Wayland support was introduced we had to find a different way. For Wayland we added a nested compositor running in the UI process. The web process connected to the nested compositor to create a surface that was used as the target surface of the GL context where the frames were composited. In the UI process the wayland surface was rendered into the GTK widget by using EGL images when GDK supported it, or downloading the texture to a Cairo image surface otherwise. +5. Threaded compositor: Composition was moved to a dedicated thread to release the main thread while compositing layers. The challenge here was to coordinate the layer changes between the main and composited threads. We reused the CoordinatedGraphics implementation that Qt port was using at the time to do the composition in the UI process, adapting it to our threaded model. Performance improved a lot, but the code became increasingly complex. +6. WPE port: The WPE port was upstreamed, with initial focus on Wayland, and later adding APIs to allow applications to handle rendering in both the web and UI processes. +7. Threaded rendering: Compositing was already in a dedicated thread, but layer flush and rendering was still happening in the main thread. With the idea of releasing more the main thread, the painting was also moved to different threads. The composition still happens after the layer flush, but it waits for all rendering threads to complete before composing the layers. +8. Async scrolling: Scrolling was also moved to a different thread when possible to ensure responsiveness. Scroll events are handled by the scrolling thread triggering a composition if needed without having to go through the main thread. +9. DMA-BUF: The GTK port started using DMA-BUF buffers for the composited frames. In this case composition is rendered into a surfaceless context and buffers are allocated with GBM. This allowed us to remove all the code to create GL context from X11 and Wayland targets. WPE also adopted this approach for its new WPE platform API. +10. Display link: For the synchronization with the actual screen we relied on the X11 server or Wayland compositor, notifying the web process when current frame has already been presented on screen to allow the next one to start. A Display link implementation was added to notify the web process on every vblank signal we get from the screen when animations are running. +11. Skia: Cairo was replaced by Skia as the 2D rendering library used to paint the layer contents. This allowed us to use the GPU also to render the layers. + +## UI process rendering + +The UI process receives the composited frames from the web process to be presented on the final output. Once the frame is painted the UI process notifies the web process. The way in which the composited frame is received and painted depends on every port. + +### Rendering frames in WebKitGTK + +The GTK port supports two rendering modes: the accelerated and non-accelerated mode. In non-accelerated mode, the web process produces frames by sharing an image using shared memory buffers. Whenever a new frame is available the web process sends the `Update` message to the UI process including the file descriptor of the shared memory containing the bitmap and a list of updated rectangles. The updated rectangles are then painted into a Skia image and a web view redraw is scheduled to paint the image into the web view widget. +In accelerated mode, the GTK port always uses the DMA-BUF buffers to get the composited frames from the web process. `AcceleratedBackingStoreDMABuf` is the class that handles all accelerated drawing in GTK port. When the web page is created the set of supported DMA-BUF buffer formats and the DRM device to be used to allocate buffers is sent to the web process. The web process tries to find the first combination of format and modifier from the list that is supported by the driver. When the web process allocates a new buffer the message `DidCreateBuffer` is sent to `AcceleratedBackingStoreDMABuf` with an id and all the parameters needed to import the buffer. Depending on the GTK version and GDK options the buffer is imported differently, when GTK version is recent enough and GDK is using EGL or vulkan, the buffer is imported using `GdkDmabufTextureBuilder`. The import happens only once, when the buffer is created. When a composited frame is ready, the message `Frame` is sent to the UI process, just with the id of the buffer, since it has already been imported, and a fence file descriptor. The buffer is stored as pending and the fence is monitored so that when it's triggered, a repaint is scheduled on the web view widget. When the web view is then painted, a new `GdkTexture` is created with `gdk_dmabuf_texture_builder_build`, the previous committed buffer is released (a `ReleaseBuffer` message is sent to the web process to let it know that buffer can be used again) and the pending one is set as committed. The new `GdkTexture` is passed to GTK to be painted in the widget and the `FrameDone` message is sent to the web process to notify that the buffer has already been painted. + +### Rendering frames in WPE new API + +WPE also uses the DMA-BUF renderer when using the new API. As in the GTK port, the list of supported DMA-BUF buffer formats and the DRM device to be used to allocate buffers is sent to the web process when the page is created, but in the case of WPE the format negotiation is dynamic. For example in the Wayland platform, we receive a notification that preferred buffer formats have changed when the surface becomes a candidate for direct scanout (typically the surface is in fullscreen mode with no transparencies). In that case a new list of buffer formats and devices is built and sent to the web process with the message `PreferredBufferFormatsDidChange`. When the next frame is going to be rendered in the web process, the old buffers are destroyed and a new one is allocated for the format and modifiers that better match using the given device. This will make it possible for the wayland compositor to pass the generated DMA-BUF buffer directly to the screen without any composition. When the web process allocates a new buffer the message `DidCreateBuffer` is sent to `AcceleratedBackingStoreDMABuf` with an id and all the parameters needed to import the buffer and a `WPEBufferDmaBuf` is created. When a composited frame is ready, the message `Frame` is sent to the UI process, just with the id of the buffer and a fence file descriptor. If the `WPEDisplay` supports explicit sync, the fence file descriptor is set on the `WPEBufferDmaBuf` to be handled by the platform and the buffer is passed to the display to be rendered, otherwise the fence is monitored by polling the file descriptor like in the GTK port and the buffer is rendered once the fence is triggered. The WPE platform then renders the buffer and notifies back when it's done. At this moment the pending buffer becomes the committed one and `FrameDone` message is sent to the web process. The platform also notifies when a buffer can be released, optionally including a fence, so that a `ReleaseBuffer` message is sent to the web process passing the buffer id and the optional fence file descriptor. The new WPE is not limited to DMA-BUF buffers but at the moment they are the only ones supported. + +### Rendering frames in WPE old API + +WPE uses libwpe for frame rendering, so depending on the backend the rendering will be done differently. libwpe is used from both the UI and web processes and it uses its own way to communicate between both processes. When the web process is created a new renderer host client is created with `wpe_renderer_host_create_client()` which returns a file descriptor that is passed to the web process. The library name of the current libwpe backend is also passed to the web process. When the web process is initialized the backend is loaded passing the given library name to `wpe_loader_init()` and a global `PlatformDisplay` is created for the given render host client file descriptor. The `PlatformDisplay` creates an EGL render backend using `wpe_renderer_backend_egl_create()` that receives the render host file descriptor. Then the EGL display is initialized for the platform and native display provided by the EGL render backend. Now every web view needs also a way to communicate with the web process to get its contents rendered. In the old API web views are created with a mandatory construct parameter to provide the wpe view backend. When a web page is created `wpe_view_backend_get_renderer_host_fd()` is called to get a file descriptor to be passed to the web process. On the web process side, the WebKit compositor creates a surface that creates an EGL render backend target calling `wpe_renderer_backend_egl_target_create()` that receives the render host file descriptor. Then an EGL context is created for the native window returned by `wpe_renderer_backend_egl_target_get_native_window()`. For every frame the compositor notifies the EGL render backend target before and after the composition by calling `wpe_renderer_backend_egl_target_frame_will_render()` and `wpe_renderer_backend_egl_target_frame_rendered()`. The composition rendering happens directly in the native window provided the WPE backend, so buffer handling depends on the backend and it's all handled there. + +## Web process + +The web process renders the page contents producing a rendered frame to be consumed by the UI process to be displayed into the output. The non-accelerated mode is quite simple, whenever a repaint is requested the drawing area schedules a display using a timer. When the display timer fires a rendering update is done and a shared memory for the frame is allocated. The updated rectangles are painted directly into the shared memory buffer and the `Update` message is sent to the UI process with the list of updated rectangles and the shared memory file descriptor. +The accelerated mode is a lot more complex, the web contents are split into layers depending on the properties of each element. So, in addition to the render tree, in accelerated mode we also have a layer tree. Every layer handles its contents separately producing a GL texture. The layer tree is then composed to produce the final frame that is shared with the UI process. + +### The layer tree + +Every `RenderView` has a `RenderLayerCompositor` to create and handle the layers for the `RenderView` contents. When entering in accelerated compositing mode the `RenderLayerCompositor` creates the root layer. `GraphicsLayer` is the base class used to represent the layers, and requires a derived class for every platform implementation (CoordinatedGraphics, CoreAnimation, etc.). `GraphicsLayer::create()` receives a `GraphicsLayerFactory` to create the derived class for the current platform, and a `GraphicsLayerClient` that is the `RenderLayerCompositor` for the layers it creates. `GraphicsLayerFactory` is an abstract class with a single pure virtual method `createGraphicsLayer()` that derived classes must implement to create the platform specific `GraphicsLayer`. The `RenderLayerCompositor` gets the `GraphicsLayerFactory` from its page `ChromeClient`, that in the WebKit implementation uses the `DrawingArea` to get it. `LayerTreeHost` is the class implementing the `GraphicsLayerFactory` interface for ports using coordinated graphics. A `LayerTreeHost` is created by the `DrawingAreaCoordinatedGraphics` when entering accelerated compositing mode. `LayerTreeHost::createGraphicsLayer()` simply creates a new `GraphicsLayerCoordinated` that is the coordinated graphics implementation of `GraphicsLayer`. A `GraphicsLayer` has a set of properties like size, position, opacity, etc. and contents that can change at any time due to a input event, a running animation, a JavaScript call, etc. A change in a `GraphicsLayer` usually requires a repaint, so whenever a property changes the property is updated and a repaint is scheduled if needed. `RenderingUpdateScheduler` class triggers a rendering update using display link to schedule repaint on display refresh that ends up calling `DrawingAreaCoordinatedGraphics::triggerRenderingUpdate`. In accelerated mode a layer flush is scheduled on the `LayerTreeHost`, that uses a timer to schedule it. `LayerTreeHost::flushLayers()` calls `WebPage::updateRendering()` that runs all the steps to render a page, and then `WebPage::finalizeRenderingUpdate()` that runs the actual layer flush and applies the scrolling tree layers position if async scrolling is enabled. `Page::finalizeRenderingUpdate()` iterates all its root frames calling `Page::finalizeRenderingUpdateForRootFrame()` that calls `LocalFrameView::flushCompositingStateForThisFrame()` that calls `RenderLayerCompositor::flushPendingLayerChanges()` on its `RenderView` compositor. The `RenderLayerCompositor` calls `GraphicsLayer::flushCompositingState` on the root layer and finally `GraphicsLayerCoordinated::flushCompositingState()` iterates the layer tree recursively to commit all pending changes in each layer. Coordinated graphics composes the layer tree in a secondary thread, so `GraphicsLayer` objects can't be used directly by the compositor, since they are not thread safe. An intermediate thread safe object `CoordinatedPlatformLayer` is used to accumulate the `GraphicsLayer` state until it's used by the compositor thread to produce the final frame. The `CoordinatedPlatformLayer` object has an owner that is the `GraphicsLayer`, a client that is the `LayerTreeHost` and a target layer that is a `TextureMapperLayer`. On layer flush, the properties that changed in `GraphicsLasyer` are set to its `CoordinatedPlatformLayer`. Whenever a property changes in `CoordinatedPlatformLayer` a composition is requested. `CoordinatedPlatformLayer` is also responsible for producing the layer contents as a texture. When the compositor executes a request, it iterates the `CoordinatedPlatformLayer` list running another flush that calls `CoordinatedPlatformLayer::flushCompositingState()` to set the current state into the final `TextureMapperLayer` that will be used by `TextureMapper` to compose the frame. + +But not only existing layer properties can change at any time, the layer tree itself can change too due to added or removed layers. This means that the layer tree at the time a composition request might have changed once the composition request is executed in the compositor thread. To make sure that the compositor always uses the current tree `LayerTreeHost` uses `CoordinatedSceneState`. It's a thread safe object that keeps track of the `CoordinatedPlatformLayers` managed by the `LayerTreeHost`. The scene state object is notified whenever a layer is attached to or detached from the `LayerTreeHost`. `CoordinatedSceneState` keeps its own list of `CoordinatedPlatformLayers` with a flag to indicate that the tree has changed. The compositor is created with a reference to the `CoordinatedSceneState` from its `LayerTreeHost`. The compositor flushes the `CoordinatedPlatformLayers` by getting a copy of the current list of layers from `CoordinatedSceneState`. + + +#### BackingStore + +A layer that needs to paint its contents uses a backing store object. During the layer flush, once all properties have been committed (set on the `CoordinatedPlatformLayer`), `CoordinatedPlatformLayer::updateContents()` is called. It checks if a backing store is needed, the condition is that the layer has the draws content flag enabled, it's visible and size is not empty. If it's needed then a `CoordinatedBackingStoreProxy` is created if it wasn't already created and the layer is marked as needing tile updates. Once the root `GraphicsLayer` has committed all properties of all its children recursively it calls `GraphicsLayerCoordinated::updateBackingStoresIfNeeded()`. This method iterates the tree again to call `updateBackingStore()` on all `CoordinatedPlatformLayers`. `CoordinatedPlatformLayers::updateBackingStore()` checks if there's a backing store that needs an update because there's a dirty region, tiles needs an update or there are more tiles pending to be created. When a layer needs to be repainted for whatever reason `setNeedsDisplay()` or `setNeedsDisplayInRect()` are called on `GraphicsLayer` generating a dirty region to be repainted. A `GraphicsLayer` also needs to update the tiles when a new backing store is created or when the layer transformation changes. When a repaint is needed `CoordinatedBackingStoreProxy::updateIfNeeded()` is called. The backing store splits the layer into tiles of 512x512. During the update it checks if tiles need to be created or removed. It also checks if tiles need to be updated according to the passed dirty region. The tiles are iterated checking if they intersect with the dirty region. Every tile produces a single dirty rectangle (the union of all rectangles inside the tile that intersect with the dirty region), for which a paint task is scheduled calling `CoordinatedPlatformLayer::paint()`. The `SkiaPaintingEngine` then sends the painting task to the main, CPU or GPU thread depending on the configuration. The result is a `CoordinatedTileBuffer` where the tile dirty rectangle is rendered into. At the end of `CoordinatedBackingStoreProxy::updateIfNeeded()` we have a list of tiles to be added, removed and updated that are merged with a previous pending update if any. The `CoordinatedBackingStoreProxy` also pre-renders more tiles to ensure they are ready in case of scrolling, but those are follow up requests. So, the result of `CoordinatedBackingStoreProxy::updateIfNeeded()` is an `OptionSet` with flags to indicate whether more tiles need to be painted, whether any tile has been scheduled to be painted and whether there's any tile change (added, removed or updated). If more tiles need to be painted the `GraphicsLayer` notifies its client (the `RenderLayerCompositor`) calling `notifySubsequentFlushRequired()` to ensure a new flayer flush is scheduled right after this one (note that this subsequent flush will not cause a composition if nothing changed). On the compositor side the `CoordinatedBackingStoreProxy` pending request is consumed by `CoordinatedPlatformLayer` in `CoordinatedPlatformLayer::flushCompositingState()`. A `CoordinatedBackingStore` is created if needed and the list of tiles to create, remove or update are set to the backing store and then `CoordinatedBackingStore::processPendingUpdates()` is called. Tiles are iterated calling `CoordinatedBackingStoreTile::processPendingUpdates()` that iterates the list of pending updates to get the buffer of the tile dirty region. Every `CoordinatedBackingStoreTile` has its own texture to cover the whole tile rectangle, but the updates can contain a dirty region that is smaller than the tile size. Depending on the dirty rectangle and the type of the dirty region buffer the possible actions could be: + +- If the buffer is not accelerated: contents need to be submitted to the tile texture. `BitmapTexture::updateContents()` is called to upload the buffer to the texture. In this case there's no difference between having a dirty region covering the whole tile area or not. +- If the buffer is accelerated and dirty region is smaller than tile area: the dirty region texture needs to be copied into the tile texture. `BitmapTexture::copyFromExternalTexture()` is called to perform a GPU to GPU copy. +- If the buffer is accelerated and dirty region is the same as the tile area: in this case we can just use the dirty region texture as the tile texture without any copies. `BitmapTexture::swapTexture()` is called in this case. + +Later, when the texture mapper paints the layer `CoordinatedBackingStore::paintToTextureMapper()` is called. This iterates the tiles calling `TextureMapper::drawTexture()` with each tile texture. + +For layers that don't draw their contents, the contents are provided in different ways using `CoordinatedPlatformLayerBuffer`. + +#### Accelerated images + +Images are usually painted into layers as part of the backing store painting. But in some cases, an image has its own layer, and a different backing store is used, the `CoordinatedImageBackingStore`. When the contents of a layer is a single image, `GraphicsLayer::setContentsToImage()` is called to set the image. If the given image is different to the current one, it's saved and the layer noted as changed. During the layer flush `CoordinatedPlatformLayer::setContentsImage()` is called, which uses a double-buffer like `CoordinatedImageBackingStore`. If there's no current image backing store or it contains a different image, the client (`LayerTreeHost`) is asked for a new `CoordinatedImageBackingStore` for the given image. `LayerTreeHost` handles the image backing stores because it keeps a cache of images, so that if there are multiple layers with the same image as contents, the same `CoordinatedImageBackingStore` is used for all the layers. `CoordinatedImageBackingStore` is just a thread safe ref counted wrapper around a `CoordinatedPlatformLayerBufferNativeImage` that contains the platform image. The `CoordinatedPlatformLayerBufferNativeImage` contains another `CoordinatedPlatformLayerBuffer` to store the native image texture. If the native image is accelerated the buffer is created on construction in the main thread as a `CoordinatedPlatformLayerBufferRGB` that receives the image texture and a fence. If the native image is not accelerated the buffer is created on composition in the compositor thread when it's requested to be painted into the texture mapper. The image contents are then uploaded to a newly created texture (or reused from texture mapper pool) that is passed to `CoordinatedPlatformLayerBufferRGB::create()`. For subsequent paint requests the internal buffer already exists and it's reused. + +#### Delegated content + +Delegated content is used for WebGL, accelerated 2D canvas and Offscreen canvas. When `CanvasRenderingContext::setContentsToLayer()` is called to set the layer, `GraphicsLayer::setContentsDisplayDelegate()` is called for the given layer using the `GraphicsLayerContentsDisplayDelegate` returned by virtual method `GraphicsLayerContentsDisplayDelegate::layerContentsDisplayDelegate()`. The base class `GraphicsLayerContentsDisplayDelegate` contains two pure virtual methods: `setDisplayBuffer()`, that receives a `CoordinatedPlatformLayerBuffer`, and `display()` that receives the `CoordinatedPlatformLayer` that should use the buffer as contents. `GraphicsLayer::setContentsDisplayDelegate()` just sets the contents delegate and marks the layer as changed (both contents buffer and contents buffer needs display). Whenever a repaint is required for a layer using delegated content, `GraphicsLayerContentsDisplayDelegate::setDisplayBuffer()` is called to set the buffer and `GraphicsLayer::setContentsNeedsDisplay()` is called marking the layer as changed. On layer flush if contents need display `GraphicsLayerContentsDisplayDelegate::display()` is called. The implementation of `GraphicsLayerContentsDisplayDelegate::display()` then should call `CoordinatedPlatformLayer::setContentsBuffer()` to set the `CoordinatedPlatformLayerBuffer` for the current delegated content. `CoordinatedPlatformLayerBuffer` also uses a double-buffer for its contents buffers, so the given buffer is set as pending. On composition flush if layer contents buffer changed, the pending is set as committed and the committed one is set in the `TextureMapperLayer` calling `TextureMapperLayer::setContentsLayer()`. `GraphicsLayerContentsDisplayDelegateCoordinated` is the coordinated graphics implementation of `GraphicsLayerContentsDisplayDelegate`, that keeps a `CoordinatedPlatformLayerBuffer` set in `setDisplayBuffer()` and passed to the `CoordinatedPlatformLayerBuffer` in `display()`. The main difference of the `GraphicsLayerContentsDisplayDelegateCoordinated` users is the type of `CoordinatedPlatformLayerBuffer` that they pass to `setDisplayBuffer()`. + + - Accelerated 2D canvas: in `prepareForDisplay()` the canvas contents are extracted as an accelerated image that is used to create a `CoordinatedPlatformLayerBufferNativeImage`. + - WebGL without GBM: in `prepareForDisplay()` a `CoordinatedPlatformLayerBufferRGB` is created for the current display texture. + - WebGL with GBM: in `prepareForDisplay()` a `CoordinatedPlatformLayerBufferDMABuf` is created for the current display buffer. + - Offscreen canvas: this case is a bit different because it uses a `GraphicsLayerAsyncContentsDisplayDelegate` to copy the canvas from different threads, which contains a pure virtual method `tryCopyToLayer()` that receives an `ImageBuffer`. `GraphicsLayerAsyncContentsDisplayDelegateCoordinated` is the coordinated graphics implementation of `GraphicsLayerAsyncContentsDisplayDelegate`. It creates a non-async `GraphicsLayerContentsDisplayDelegate` that is set on the given layer on construction. In `tryCopyToLayer()` the given `ImageBuffer` is cloned and its native image is used to create a `CoordinatedPlatformLayerBufferNativeImage` that is set to the non-async delegate calling `setDisplayBuffer()`. + +#### Media + +In the case of media, the video frames are provided by the media player at its own pace and updates are not handled by layer flush, but passed directly to the compositor. The class used to handle video frames is `CoordinatedPlatformLayerBufferProxy`. The media player creates a `CoordinatedPlatformLayerBufferProxy` on construction that is returned as its platform layer. `GraphicsLayer::setContentsToPlatformLayer()` is called to set the `CoordinatedPlatformLayerBufferProxy` on the graphics layer. `GraphicsLayer::setContentsToPlatformLayer()` calls `CoordinatedPlatformLayerBufferProxy::setTargetLayer()` passing its `CoordinatedPlatformLayer` to associate the proxy with the platform layer. The first time the proxy is set, the layer is marked as changed as usual and if at the time of layer flush there was a pending buffer, it's set to the platform layer calling `CoordinatedPlatformLayer::setContentsBuffer()` the same way it's done for delegated content. The next frames are handled directly by the proxy and sent to the compositor without a layer flush. Whenever a new frame is available (usually from a GStreamer thread) the media player creates a `CoordinatedPlatformLayerBufferVideo` for the current frame and calls `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()`. `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()` saves the buffer as pending if the proxy hasn't been attached to a platform layer or calls `CoordinatedPlatformLayer::setContentsBuffer()` on the attached platform layer and asks the layer to request a composition to its client (`LayerTreeHost`). In `CoordinatedPlatformLayer` the media buffers are handled exactly the same way as the delegated content buffers. `CoordinatedPlatformLayerBufferVideo` is a class that contains another `CoordinatedPlatformLayerBuffer` that depends on the type of video frame and whether the frame contains a DMABuf buffer, GL texture or main memory. + +- If the frame contains a DMABuf buffer, a `CoordinatedPlatformLayerBufferDMABuf` is created for the imported DMABuf. +- If the frame contains a GL texture: + - A `CoordinatedPlatformLayerBufferExternalOES` is created if the frame contains an external EOS texture. + - A `CoordinatedPlatformLayerBufferRGB` is created if the frame contains a single plane texture. + - A `CoordinatedPlatformLayerBufferYUV` is created if the frame contains a supported YUV multiplane texture. +- If the frame contains main memory the frame is mapped on construction and a `CoordinatedPlatformLayerBufferRGB` is created for a newly created texture (or taken from the texture mapper pool), for which the frame contents are uploaded, from the compositor thread when the buffer is requested to be painted into the texture mapper. + +### The compositor + +The compositor runs in a separate thread in the web process. It's created by the `LayerTreeHost` on construction. When the compositor is created it performs the following actions: + +- Creates an `AcceleratedSurface` to render the composited frames that will be shared with the UI process. +- Takes a reference to the `CoordinatedSceneState` of the `LayerTreeHost` to keep track of the layers. +- Creates a `CompositingRunLoop` that will drive the compositor tasks. +- Notifies the surface that the compositing run loop has been created, which is implemented by `AcceleratedSurfaceDMABuf` to initialize the IPC message receiver to receive messages from UI process in the compositor thread. +- Synchronously creates the GL context in the compositor thread by scheduling a sync task. + +After that the compositor is idle waiting for requests. It can receive requests in different ways: + +- `ThreadedCompositor::requestComposition()`: This is called by `LayerTreeHost` on layer flush when a composition is required. It returns a request ID that the `LayerTreeHost` uses to know when this particular request has been completed. +- `ThreadedCompositor::scheduleUpdate()`: This is called by `LayerTreeHost` when a new composition is required for async scrolling or media player, or by the compositor itself when there are active animations. In this case there's no request identifier so the last one set by `requestComposition()` is used. + +When a new composition is requested either by `requestComposition()` or `scheduleUpdate()` an update is scheduled in the `CompositingRunLoop`. The `CompositingRunLoop` only allows one composition at a time per frame so the following can happen: + +- The loop is idle: state is set as scheduled and a 0 timer is started to run the update function (`renderLayerTree`). +- The loop is scheduled: does nothing, update function will be called when the timer fires. +- The loop is "in progress": the state is marked as having a pending update to be run once the loop is idle again. + +When the update timer fires, the state is set to "in progress" and the update function is called (`renderLayerTree`). It will stay in "in progress" state until UI notifies that the current frame has been presented on screen (the `FrameDone` message). Then `CompositingRunLoop::updateCompleted()` is called and the following can happen in this case: + +- The loop is idle or scheduled: does nothing. +- The loop is "in progress": if the state has a pending update, the state is changed as scheduled and the update timer is started, otherwise the state is set to idle. + +The compositor uses texture mapper to render all layers into the final frame. This happens in `ThreadedCompositor::renderLayerTree()` that is called by the `CompositingRunLoop` when the update timer fires. The following steps are run: + +1. Scene attributes (size and device scale factor) are used to create a `TransformationMatrix` and resize the surface if needed. +2. The surface is notified that a new frame is about to be rendered. The surface prepares the next target allocating new buffers if needed and preparing the framebuffer. +3. `LayerTreeHost` is also notified scheduling a main thread call. +4. If the surface was resized, `glViewport()` is called with the new size. +5. Surface is asked to clear if needed to call `glClear()` with transparent color if the surface is not opaque. +6. Scene is updated: `flushCompositingState()` is called for all the currently active layers in `CoordinatedSceneState`. +7. Layers are now painted into the texture mapper. +8. If there are running animations another composition is requested. +9. The composition response identifier is set to the current request identifier and `LayerTreeHost` is notified calling `didComposite()` from a main thread timer passing the composition response identifier. +10. The surface is notified that a frame has been rendered. In the case of `AcceleratedSurfaceDMABuf` the `Frame` message is sent to the UI process. +11. `LayerTreeHost` is also notified scheduling a main thread call. + +The `LayerTreeHost` ensures layer flushes can't be done while there's an ongoing composition by setting a variable (`m_isWaitingForRenderer`) before calling `ThreadedCompositor::requestComposition()` and unsetting it when the composition is done in `LayerTreeHost::didComposite()` if the given response identifier matches the current request identifier. If a layer flush is requested while `m_isWaitingForRenderer` is enabled, another variable is used to indicate that a layer flush was requested but couldn't be done. When current composition finishes, if a layer flush was scheduled while `m_isWaitingForRenderer` was true, the pending layer flush is performed immediately. + +### Async scrolling + +Async scrolling handles wheel events in a secondary thread in the web process to try to ensure we respond quickly to scrolling requests with visual feedback. The idea is that the scrolling thread can schedule a composition without going through the main thread if rendering layers is taking too long since the scrolling was requested. Wheel events are not sent from `WebPageProxy` to `WebPage` like other events, they are sent from `WebPageProxy` to `EventDispatcher` to be processed by the web process in a separate thread. The `EventDispatcher` is a singleton running in the web process that uses a work queue to handle IPC messages off the main thread. When async scrolling is enabled `DrawingAreaCoordinatedGraphics` registers the scrolling tree in the event dispatcher by calling `EventDispatcher::addScrollingTreeForPage()`. `EventDispatcher::wheelEvent()` handles the IPC message from the work queue thread. If there isn't a scrolling tree registered for the given page identifier the event is scheduled to be handled in the main thread. Otherwise the event is dispatched by the scrolling thread (from the `EventDispatcher` thread). When executed in the scrolling thread `ScrollingTree::handleWheelEvent()` is called, which determines the scrolling tree node that should handle the event for the given position and calls `ScrollingTree::handleWheelEventWithNode()`. The scrolling tree is iterated from the given node calling `ScrollingTreeNode::handleWheelEvent()` until the event is handled by a node. `handleWheelEvent()` is a virtual method that just returns "unhandled" by default, so it needs to be implemented by derived classes. Events are handled by a delegate, so derived classes usually just forward the event handling to their delegate. `ScrollingTreeScrollingNodeDelegateCoordinated` is the coordinated graphics implementation of `ThreadedScrollingTreeScrollingNodeDelegate` used by the coordinated graphics scrolling nodes to handle scrolling events. The delegate uses a `ScrollingEffectsController` to process the events, so `ScrollingEffectsController::handleWheelEvent()` is what actually processes the event from the scrolling thread (the same way it's done in the main thread by `ScrollAnimator` class). The scroll deltas are calculated and then an immediate or animated scrolling is started by asking the `ScrollingEffectsController` client (in this case is the delegate). The delegate calls `scrollBy()` on its scrolling tree node passing the given scroll deltas. The node calculates its new position for the scrolling deltas and the scrolling tree is notified that the node has been scrolled. `ThreadedScrollingTree::scrollingTreeNodeDidScroll()` generates a `ScrollUpdate` for the node and notifies the scrolling coordinator from the main thread, which schedules a rendering update. When `RenderingUpdateScheduler` schedules the update the scrolling coordinator is also notified. After iterating the nodes `EventDispatcher` notifies the UI process by sending the message `DidReceiveEvent` to the `WebPageProxy` indicating whether it was handled or not. At this point the scrolling tree nodes have updated their position for the scrolling event, and the main thread has scheduled a rendering update. When the layer flush is triggered the following happens: + +- From `Page::updateRendering()`, which is called right before the layer properties are committed, `willStartRenderingUpdate()` is called on the scrolling coordinator, which notifies its tree and calls `synchronizeStateFromScrollingTree()` to update. + - `ThreadedScrollingTree::willStartRenderingUpdate()`: sends a sync task to the scrolling thread to block until current rendering update is completed or until a timeout of at most half a frame duration fires. The tree state is the set as `InRenderingUpdate`. If the scrolling thread is unblocked because of the timeout, it means the rendering update hasn't completed yet and the tree state is set as `Desynchronized`. If layer positions can be updated from the scrolling thread `applyLayerPositions()` is called in a follow up run loop iteration inside the scrolling thread. + - `AsyncScrollingCoordinator::synchronizeStateFromScrollingTree()`: First calls `applyPendingScrollUpdates()` that iterates all pending updates calling `applyScrollPositionUpdate()`. For position updates `updateScrollPositionAfterAsyncScroll()` is called which calls `reconcileScrollingState()` if the given node id is the frame view one. This ends up calling `GraphicsLayer::syncPosition()` for the frame view scroll container layer and all other scroll related layers if present. `GraphicsLayer::syncPosition()` is like `GraphicsLayer::setPosition()`, but the new value is not set on the `CoordinatedPlatformLayer` when properties are committed during the layer flush, since the platform layer is expected to be updated directly by the scrolling tree node. +- From `Page::finalizeRenderingUpdate()`, right after all layer properties have been committed, `applyScrollingTreeLayerPositions()` and `didCompleteRenderingUpdate()` are called for the page scrolling coordinator. + - `AsyncScrollingCoordinator::applyScrollingTreeLayerPositions()` just calls `ScrollingTree::applyLayerPositions()` which recursively iterates the scrolling tree to call `applyLayerPositions()` for every node. All the nodes end up setting their updated position to their associated `CoordinatedPlatformLayer` that will request a composition if the position changed. At this point the `GraphicsLayer` and its `CoordinatedPlatformLayer` are in sync again. + - `ScrollingTreeCoordinated::didCompleteRenderingUpdate()`: layers are now committed, but rendering is asynchronous with coordinated graphics, so at this point the layers are being painted if needed and a composition was requested if any property changed during the layer flush. So, this checks if a composition is ongoing or has been requested and returns early in that case to wait for the composition to complete. Otherwise `renderingUpdateComplete()` is called to unblock the scrolling thread if it's still blocked waiting for the rendering. + +If the layer flush triggered a composition, `LayerTreeHost` notifies the page when the composition is done by calling `Page::didCompleteRenderingUpdateDisplay()` which calls `didCompletePlatformRenderingUpdate()` for the scrolling coordinator. `ScrollingTreeCoordinated::didCompletePlatformRenderingUpdate()` just calls `renderingUpdateComplete()` to unblock the scrolling thread if it's still blocked waiting for the rendering. + +`ThreadedScrollingTree` is also synchronized with the display link. `EventDispatcher` is also notified from the UI process when display has been refreshed, and notifies all the registered scrolling trees calling `displayDidRefresh()`. As with the events, this is done from work queue thread to scrolling thread without passing through the main thread. On display refresh the scrolling thread does the following: + +- Calls `serviceScrollAnimations()` which calls `serviceScrollAnimation()` on every scrolling node with an active scroll animation. This calls the animation callback of the `ScrollingEffectsController`. +- If state is not idle and layers can be updated from the scrolling thread, `applyLayerPositions()` is called. This ends up updating the `CoordinatedPlatformLayer` position like during the rendering update but now from the scrolling thread instead of the main thread. In this case a composition is requested directly from the scrolling thread after positions have been applied. +- If state is still idle and a rendering update was scheduled then state is set as `WaitingForRenderingUpdate` and a 1ms timer is started. The timer is stopped right before the scrolling thread is blocked to wait for the rendering update. If the timer fires before the next rendering update starts, `applyLayerPositions()` is called, if layers can be updated from the scrolling thread, and state is set to `Desynchronized`. +