perf(trickplay): retain sprite textures and prefetch directional previews

Author: sachk

CMakeLists.txt

@@ -206,6 +206,10 @@ set(SPOOL_SHELL_SOURCES
     src/app/ArtworkPrefetcher.h
     src/app/ArtworkService.cpp
     src/app/ArtworkService.h
+    src/app/TrickplayDecoder.cpp
+    src/app/TrickplayDecoder.h
+    src/app/TrickplayService.cpp
+    src/app/TrickplayService.h
     src/app/BrowseSessionController.cpp
     src/app/BrowseSessionController.h
     src/app/ContentModelController.cpp
@@ -789,6 +793,7 @@ if(BUILD_TESTING AND NOT CMAKE_CROSSCOMPILING)
         group-playback-policy tests/app/GroupPlaybackPolicyTest.cpp
         update-manifest tests/app/UpdateManifestTest.cpp
         artwork-url tests/app/ArtworkUrlTest.cpp
+        trickplay tests/app/TrickplayTest.cpp
     )
     set(SPOOL_TESTS_cache
         database-manager tests/cache/DatabaseManagerTest.cpp

README.md

@@ -186,8 +186,48 @@ being mistaken for good performance.
 Provider form layouts are precompiled in the host. Settings creates only visible
 rows and opens its native folder chooser on demand; initial viewport readiness is
 checked at the end of the current event-loop turn, with timed retries only when
-delegates are still missing. Playback no longer creates hidden image objects for
-every trickplay sheet: previews load the sheet needed by the current seek position.
+delegates are still missing. Playback previews prime the resume-position
+texture on a dedicated network/decode path, independent of poster queues.
+Sprite sheets up to 50 MB decoded RGB32 (12.5 megapixels) stay resident:
+moving between tiles changes the displayed crop without another image load,
+CPU crop, or texture upload. Larger JPEG sheets use codec-clipped frames.
+Whole BIF sequences use their native timestamp indexes, not an assumed
+uniform interval. Qt's existing JPEG plugin supplies its libjpeg/libjpeg-turbo
+decoder; no duplicate JPEG dependency is needed.
+
+The server-free native fixture smoke exercises real JPEG sprite and BIF pixels,
+malformed BIF boundaries, resident-sheet crops, the 50 MB boundary, directional
+prefetch, and stale-session/late-request cancellation:
+
+```sh
+nix develop .#native -c cmake --build build/linux-dev/app --target app-tests providers-tests
+nix develop .#native -c env SPOOL_TRICKPLAY_SMOKE_OUTPUT=/tmp/spool-preview-frame.png \
+  build/linux-dev/app/app-tests trickplay
+nix develop .#native -c ctest --test-dir build/linux-dev/app \
+  -R '^(trickplay|artwork-authentication|playback-timeline)$' --output-on-failure
+```
+
+The smoke prints cold response time, 24 explicit warm image-provider response
+times, full-sheet decode time, and texture byte counts. Normal hover within a
+resident sheet does not issue those additional image-provider requests.
+These measurements exclude actual GPU transfer and presentation duration.
+Each local/remote session caps retained decoded textures at 50,000,000 bytes
+and encoded sheets (or one BIF sequence) at 32 MiB. The decoded budget is not
+a total RAM/VRAM limit: decoder workspace, active image delivery and GPU
+textures add to it. Sequences over 32 MiB are rejected.
+
+Hover image selection consumes the latest input once per rendered frame;
+the timestamp and preview position update immediately. Prefetch follows hover
+direction to one neighbouring sheet or at most two BIF frames. Foreground
+requests preempt speculative work; a neighbouring sheet is decoded only when
+it fits beside the current retained textures, otherwise only its encoded bytes
+are prefetched. Qt image caching is disabled for previews so historical sheets
+do not accumulate in its image cache. Switching sessions cancels outstanding
+delivery and drops caches.
+Remote resource revisions also change when a target switches preview URL or
+credentials for the same item. Retired revisions cannot reuse cached frames;
+unchanged resources keep stable identities across state polling.
+
 
 
 ## Android development

docs/providers.md

@@ -332,14 +332,19 @@ Advanced navigation/text commands remain provider-owned operations; their thin
 picker adapters configure the shared, precompiled `ProviderRemoteControls` surface.
 
 Protected playback previews use account-scoped `spool-artwork://` resources,
-resolved by `ArtworkService` on the GUI thread. Local previews use the owning
-account's playback headers, not the currently active player's credentials.
-Remote descriptors may supply `headers`; SourceHub retains them privately and
-exposes only an opaque numeric-index template to QML. Remote previews do not
-require starting local playback. Requests stay on the approved origin, reject
-foreign-origin redirects, disable cookies and bypass Qt's URL-only disk cache.
-The memory cache is partitioned by account, resolved URL and credential hash;
-removed accounts cannot resolve old resources, even from memory.
+resolved privately by `SourceHub`. `TrickplayService` loads local and remote
+sprite sheets or whole BIF sequences on a dedicated worker. Sheets fitting
+the 50,000,000-byte decoded RGB32 budget expose stable sheet `image://` URLs
+and per-thumbnail crop offsets; larger JPEGs and BIF use single-frame URLs.
+Local descriptors use the owning account's playback headers; remote descriptors
+may supply their account/device `headers`.
+Remote previews do not require starting local playback. Requests stay on the
+approved origin, reject foreign-origin redirects, disable cookies and bypass
+Qt's URL-only disk cache. Encoded and decoded texture caches are bounded and
+session-isolated; removed accounts cannot resolve old resources, even from memory.
+Directional prefetch is limited to one neighbouring sheet or two BIF frames
+and yields to foreground requests. Both preview surfaces consume the latest
+selection once per rendered frame and clip resident sheets to the selected tile.
 
 These adapters do not implement Plex watch-together or native SpoolLink peer
 enhancements. Protocol/loopback verification does not imply live-device support

providers/bundled/spool.emby-0.1.4.tar.zst

@@ -5,19 +5,19 @@
       "id": "spool.jellyfin",
       "version": "0.2.8",
       "archive": "bundled/spool.jellyfin-0.2.8.tar.zst",
-      "sha256": "d64e4efbea49e757361a8b5a2d56701768489d3b78bc990aad57687f87bfd65f"
+      "sha256": "17913a9c887edd29c5a886d5fc9af80d7659e1559c8e06aa86a8ad88aa2e4d08"
     },
     {
       "id": "spool.emby",
       "version": "0.1.4",
       "archive": "bundled/spool.emby-0.1.4.tar.zst",
-      "sha256": "5dec32cd62dafdd4a461e40bb0faa24b51c07ff251d36c85b301cf2740fe13ac"
+      "sha256": "437db9ab69491c8f3b93627afa7e475a6c6cef95d0c134faa580df9237f3e6fb"
     },
     {
       "id": "spool.plex",
       "version": "0.1.5",
       "archive": "bundled/spool.plex-0.1.5.tar.zst",
-      "sha256": "5e426dd3d888d75bfd00a12d37cbab7ab11ab56f02fbd684f4f876ed3ba8c80d"
+      "sha256": "10ffbe71e182177bf07035c3c025ccf14872bacff31c03653c76a4e6dcf166fa"
     }
   ]
 }

providers/bundled/spool.jellyfin-0.2.8.tar.zst

@@ -6,22 +6,56 @@ Item {
     id: root
 
     required property var overlay
+    readonly property var previewPlayer: overlay.hasPlayer ? overlay.player : null
     readonly property bool localHovering: overlay.timelineHovering && overlay.controlsReason !== "remote"
-    readonly property bool active: overlay.hasPlayer && overlay.controlsVisible && overlay.player.trickplayAvailable && (
-                                       overlay.scrubbing || overlay.remoteScrubbing || localHovering
-                                       || overlay.previewing)
+    readonly property bool active: previewPlayer !== null && overlay.controlsVisible
+                                   && previewPlayer.trickplayAvailable && (overlay.scrubbing || overlay.remoteScrubbing
+                                                                           || localHovering || overlay.previewing)
     readonly property double previewSeconds: overlay.scrubbing || overlay.remoteScrubbing ? overlay.scrubSeconds :
                                                                                             localHovering
                                                                                             ? overlay.timelineHoverSeconds :
                                                                                               overlay.positionSeconds()
-    readonly property var trickplayData: active ? overlay.player.trickplayForSeconds(previewSeconds) : ({})
+    property var trickplayData: ({})
+    property bool selectionPending: active
     readonly property bool ready: trickplayData && trickplayData.available === true
     readonly property real scaleFactor: overlay.uiScale * 1.4
     readonly property real previewRatio: overlay.hasPlayer && overlay.player.durationSeconds > 0 ? Math.max(0, Math.min(
                                                                                                                 1, previewSeconds
                                                                                                                 / overlay.player.durationSeconds)) :
                                                                                                    0
 
+    onActiveChanged: {
+        selectionPending = active
+        if (!active)
+            trickplayData = ({})
+    }
+    onPreviewSecondsChanged: if (active)
+                                 selectionPending = true
+    onPreviewPlayerChanged: {
+        trickplayData = ({})
+        selectionPending = active
+    }
+
+    Connections {
+        target: root.previewPlayer
+        ignoreUnknownSignals: true
+        function onTrickplayChanged() {
+            if (!root.active)
+                root.trickplayData = ({})
+            root.selectionPending = root.active
+        }
+    }
+
+    // Keep the scheduler outside the image-ready container so a pending
+    // asynchronous image cannot prevent the first selection from running.
+    FrameAnimation {
+        running: root.active && root.selectionPending
+        onTriggered: {
+            root.selectionPending = false
+            root.trickplayData = root.previewPlayer.trickplayForSeconds(root.previewSeconds)
+        }
+    }
+
     function dp(value) {
         return overlay.dp(value)
     }
@@ -38,10 +72,11 @@ Item {
     anchors.bottom: parent.bottom
     anchors.bottomMargin: dp(200)
     height: ready ? Math.round((trickplayData.height || 0) * scaleFactor) + dp(32) : 0
-    visible: ready
+    visible: active
     z: 22
 
     Item {
+        visible: root.ready && previewImage.status === Image.Ready
         readonly property real imageWidth: root.ready ? root.trickplayData.width * root.scaleFactor : 0
         readonly property real imageHeight: root.ready ? root.trickplayData.height * root.scaleFactor : 0
         x: Math.max(root.dp(52), Math.min(parent.width - imageWidth - root.dp(52), root.previewRatio * parent.width
@@ -53,20 +88,21 @@ Item {
             id: frame
             width: parent.imageWidth
             height: parent.imageHeight
-            color: "black"
+            color: "transparent"
             border.color: Theme.borderStrong
             border.width: 1
             radius: Theme.radiusLarge
             clip: true
 
             Image {
+                id: previewImage
                 source: root.ready ? root.artworkSource(root.trickplayData.url) : ""
                 x: root.ready ? root.trickplayData.offsetX * root.scaleFactor : 0
                 y: root.ready ? root.trickplayData.offsetY * root.scaleFactor : 0
                 width: root.ready ? root.trickplayData.sheetWidth * root.scaleFactor : 0
                 height: root.ready ? root.trickplayData.sheetHeight * root.scaleFactor : 0
                 fillMode: Image.Stretch
-                cache: true
+                cache: false
                 asynchronous: true
             }
         }

providers/bundled/spool.plex-0.1.5.tar.zst

@@ -35,11 +35,30 @@ FocusScope {
     onFocusedIndexChanged: if (focusedIndex >= 0)
                                lastFocusedIndex = focusedIndex
     readonly property var preview: snapshot.preview || null
-    readonly property int previewFrame: preview ? Math.min(preview.count - 1, Math.max(0, Math.floor(editedValue * 1000
-                                                                                                     / preview.intervalMs))) :
-                                                  0
-    readonly property int previewSheet: preview ? Math.floor(previewFrame / (preview.columns * preview.rows)) : 0
-    readonly property int previewTile: preview ? previewFrame % (preview.columns * preview.rows) : 0
+    readonly property bool previewActive: visible && !customContext && editingKey === "seek" && preview !== null
+                                          && remote.trickplayAvailable
+    property var previewFrame: ({})
+    property bool previewSelectionPending: previewActive
+    onPreviewActiveChanged: {
+        previewSelectionPending = previewActive
+        if (!previewActive)
+            previewFrame = ({})
+    }
+    onEditedValueChanged: if (previewActive)
+                              previewSelectionPending = true
+    onRemoteChanged: {
+        previewFrame = ({})
+        previewSelectionPending = previewActive
+    }
+
+    // Selection belongs to the page, not the image-ready item below.
+    FrameAnimation {
+        running: root.previewActive && root.previewSelectionPending
+        onTriggered: {
+            root.previewSelectionPending = false
+            root.previewFrame = root.remote.trickplayForSeconds(root.editedValue)
+        }
+    }
 
     function supports(action) {
         return attached && commands.indexOf(action) >= 0
@@ -237,6 +256,12 @@ FocusScope {
     }
     Connections {
         target: root.remote
+        ignoreUnknownSignals: true
+        function onTrickplayChanged() {
+            if (!root.previewActive)
+                root.previewFrame = ({})
+            root.previewSelectionPending = root.previewActive
+        }
         function onSelectionChanged() {
             root.editingKey = ""
             root.customContext = null
@@ -462,20 +487,29 @@ FocusScope {
             color: Theme.textSecondary
             wrapMode: Text.WordWrap
         }
-        Image {
-            visible: root.editingKey === "seek" && root.preview !== null
+        Item {
+            id: seekPreview
+            readonly property bool ready: root.previewFrame.available === true
+            readonly property real imageScale: ready && root.previewFrame.width > 0 ? width / root.previewFrame.width :
+                                                                                      0
+
+            visible: ready && seekPreviewImage.status === Image.Ready
             Layout.alignment: Qt.AlignHCenter
             Layout.preferredWidth: Metrics.scaled(240)
-            Layout.preferredHeight: root.preview ? width * root.preview.height / root.preview.width : 0
-            source: visible ? "image://artwork/" + encodeURIComponent(root.preview.urlTemplate.replace("{index}", String(
-                                                                                                           root.previewSheet))) :
-                              ""
-            sourceClipRect: root.preview ? Qt.rect((root.previewTile % root.preview.columns) * root.preview.width,
-                                                   Math.floor(root.previewTile / root.preview.columns)
-                                                   * root.preview.height, root.preview.width, root.preview.height) :
-                                           Qt.rect(0, 0, 0, 0)
-            fillMode: Image.PreserveAspectFit
-            asynchronous: true
+            Layout.preferredHeight: ready ? root.previewFrame.height * imageScale : 0
+            clip: true
+
+            Image {
+                id: seekPreviewImage
+                source: seekPreview.ready ? root.previewFrame.url : ""
+                x: seekPreview.ready ? root.previewFrame.offsetX * seekPreview.imageScale : 0
+                y: seekPreview.ready ? root.previewFrame.offsetY * seekPreview.imageScale : 0
+                width: seekPreview.ready ? root.previewFrame.sheetWidth * seekPreview.imageScale : 0
+                height: seekPreview.ready ? root.previewFrame.sheetHeight * seekPreview.imageScale : 0
+                fillMode: Image.Stretch
+                cache: false
+                asynchronous: true
+            }
         }
         ListView {
             id: list

providers/lock.json

@@ -162,6 +162,21 @@ media rows; home cards and details request the image from that owner.
 Do not attach a parent's tag to a child without its owner ID. Series posters
 and album covers retain their existing `seriesId`/`albumId` ownership.
 
+## Seek previews
+
+Return `resolve().trickplay` for the selected media variant, not a global URL
+template in `describe()`. Sprite sheets use
+`{width, height, columns, rows, count, intervalMs, urlTemplate}`; the absolute
+HTTP(S) template has one `{index}` substitution. BIF sequences use
+`{format: "bif", url}` with optional `width`/`height`. Pass the whole sequence
+URL, not individual JPEGs: C++ parses its timestamp/offset index and decodes
+the selected frame. Missing server-generated previews mean omit `trickplay`.
+
+Both formats use the account's `resolve().headers`; keep tokens out of URLs.
+The native loader prefetches the resume preview, caches bounded preview data
+separately from posters, and uploads only the requested frame. While loading
+or on failure, the player shows no preview frame or black placeholder.
+
 ## Screens
 
 A screen is mounted with a `provider` property (`ScreenContext` in `provider.d.ts`) and may
@@ -283,14 +298,13 @@ is verified against count/order and occurrence IDs, with read-back before
 reconciliation after uncertain mutations; fixture success is not a live-server
 compatibility guarantee.
 
-Protected `spool.remote-targets` preview descriptors may include an optional
-`headers` map alongside `urlTemplate`. Use the same account/device authorization
-as media requests; never put account tokens in preview query strings. The host
-keeps headers out of QML, validates the approved HTTP(S) origin and numeric
-`{index}` substitution, and loads only requested sheets through its isolated
-artwork pipeline. Local trickplay sheets use `resolve().headers` from their own
-account. Authenticated sheets bypass the URL-only disk cache and cookies, reject
-foreign-origin redirects, and reuse only their credential-partitioned memory cache.
+Protected `spool.remote-targets` state uses the same sheet/BIF descriptor in
+`preview`, with an optional `headers` map. Use the same account/device
+authorization as media requests; never put tokens in preview query strings.
+The host keeps headers out of QML, validates the approved HTTP(S) origin and
+numeric `{index}` substitution, and decodes previews through the dedicated
+native loader. Authenticated previews bypass URL-only disk caching and cookies,
+reject foreign-origin redirects, and keep cached data isolated by session.
 
 ### Native preferences and application data