Repository navigation
ci: compositor black outside the camera picture (CI trigger only) - #2
Closed
christian-wr wants to merge 357 commits into
Closed
christian-wr wants to merge 357 commits into
christian-wr wants to merge 357 commits into
Conversation
…nged screen The video writer read the full GPU frame back on every tick, even when WGC delivered nothing new: a CopyResource, a Map that waits on the GPU, and the BGRA to NV12 conversion, for a picture it already had. On a static screen, most of a demo, that was every tick. When no new WGC frame arrived and no new camera frame is drawn into the picture, the encoder now hands over a new sample on the last buffer. Measured at 1080p: 5.0 ms per readback, under 0.001 ms per repeat. The DXGI path and the legacy callback path read back as before. Fixes getopenscreen#925
whisper.cpp's DTW pass teacher-forces the decoded BPE tokens; ours feeds the text back one character per decoder row (arXiv 2509.09987), averages the alignment heads with per-frame L2 normalisation, and drops one silent final consonant per French word. It is spliced into a build-tree copy of whisper.cpp at configure time, so the fetched source stays untouched. Word-timing harness, inner starts: median 31 -> 16 ms, within 50 ms 64% -> 86% (FR 83%, EN 89%); clean single-word cuts 19% -> 39%. Refs getopenscreen#948
…read the harness port from the helper env
A second alignment pass (issue getopenscreen#948, phase 3): a wav2vec2 CTC model scores every 20 ms frame of the speech against every letter, and a Viterbi pass forces whisper's own words through those scores. English uses wav2vec2-base-960h (109 MB), French wav2vec2-large-xlsr-53-french (348 MB), both Apache-2.0, converted to Q8_0 GGUF and run on ggml inside whisper-stt-server, on the GPU whisper uses. The helper only scores (POST /emissions); spelling, Viterbi, calibration and fallback live in electron/stt/ctcAlign.ts. Other languages, a failed download or an older helper keep the phase 1 times. TTS corpus, clean: inner start median 31 -> 14 ms, P90 125 -> 35 ms, within 50 ms 64% -> 96%; clean single-word cuts 19% -> 50%; phrase delete 89% -> 96%. LibriSpeech vs MFA: inner median 40 -> 15 ms, within 50 ms 55% -> 92%, clean cuts 14% -> 44%. Extra runtime about +10% on Vulkan.
Address the review of getopenscreen#954: - the aligner downloads in the background; chunks before it lands keep whisper's times, a cached copy is verified in place; a 30 s stall, Cancel and quit abort it, and a failed one is retried on the next transcription - accept an emissions region ending within two frames of a stretch that runs to the end of the upload - the harness skips (evaluate) or names (real-check) unreadable emissions - test-whisper-stt says which aligner file it looked for; the doc lists the aligner checks it runs
…se 2 On top of character-level DTW, the CTC aligner still moves TTS inner starts from 17/60 ms (median/P90) to 14/35 ms and LibriSpeech from 20/65 to 15/45 ms, for +9% (English) to +14% (French) on Vulkan. On the CPU it would cost +29% to +58%, so a chunk whisper ran on the CPU keeps the DTW times, and a CPU-only machine never downloads the model.
…ter trial The energy at 10 ms puts "quoi" on the start of its /k/ closure, where the aligner has it; the earlier spectrogram reading had it early. Dropping French silent final consonants before aligning, as char-dtw does, moved none of the take's boundaries by more than 15 ms and cost 1 to 2 points of French phrase deletes, so it is left out.
Switching the bar to vertical and back left it about 130 DIP higher each round trip (Windows 11, 125 %). Right after hud-overlay-set-size the ResizeObserver fires again while the old viewport is still laid out, so the hud-overlay-content that follows carries a rect measured in the window the main process has just replaced. The main process stores it as the anchor, and the next flip re-anchors the window on it: the window keeps its top-left instead of the bar's bottom-centre. The stack is always centred and pinned HUD_BAR_BOTTOM above the bottom edge, which is how grantedContent is already computed. The reported rect now uses the same formula on the allocated size, and only its size comes from the measurement. Fixes getopenscreen#951
…s rejected The staging script took the first 50 runs `gh run list --status success` returned and trusted their order. That order is not stable: during the v2.0.0-rc.5 build the same query listed this morning's runs first in one job and August's in another, so three jobs found the matching helper and two failed with "no successful build-whisper-stt run". It now reads every successful run, sorts them by creation date itself, and reads the list a second time before giving up. A rejected run now says which source differs and how, a commit that no longer exists is skipped quietly instead of printing a 404, and any other API error is retried, then fatal, instead of passing for a mismatch.
…a broken listing GitHub caps status-filtered run queries at 1,000 results, so the success filter now runs locally. The list is read into a file with its exit status checked and retried: from a process substitution, a failed or partial listing passed for a complete one.
… own proportions The picture-in-picture camera now starts square. The other shape, which follows the camera's own aspect (a portrait camera stays portrait), is labelled "Original" instead of "Rectangle". Each shape has its own default roundness, 70% for Square and 40% for Original. An untouched roundness follows the shape when it changes; a tuned one is kept. Projects saved with a shape keep the look they had. The editor's drag box was laid out from a fixed 4:3 camera while the native canvas drew the real one. Both now read webcamBoxSourceSize, so the box matches the picture.
DTW can anchor a sentence's last word in the pause after it, past the kept tail. The next stretch then owned it, put it on its onset after the words that follow it, and the aligner looked for it in the wrong audio (issue getopenscreen#948). Ownership is now one function, read by both passes; a sentence-final word anchored before the next stretch's speech closes the stretch before it. Word starts also come out monotonic and never inverted.
A separate "(" after a pause-anchored "b." made the scan read "b.(", no
sentence end, so "b." opened the next stretch again.
The menu is absolutely positioned under an anchor only as wide as the wordmark, so shrink-to-fit settled on its 232px min-width and French wrapped "Changer de langue" and split the version as "2.0.0-" / "rc.8". Size it to its widest row with width: max-content (capped to the viewport), keep rows on one line and never shrink or break the trailing value.
A screen picked in Apple's system picker only leaves out the windows the picker was told about when it was shown: their ids are baked into the picked filter, and a take cannot add one without the Screen Recording grant the picker exists to avoid. The HUD is a new window after every trip through the editor, so every take after the first one reused a pick that did not exclude it, and recorded the HUD and its timer. The picker session now remembers which windows each pick excludes, and the main process forgets a screen pick once the current HUD or notes window is not among them. `get-selected-source` then answers null, so the next Record presents the picker again with the current ids. Record start applies the same check as a last guard. Window picks and a take already running are left alone. Fixes getopenscreen#965
The empty-lane hints wrote the default letter into the string ("Press Z to
add zoom"), so after a rebind in Keyboard Shortcuts the hint kept teaching a
key that did nothing while the toolbar chip showed the right one.
The hints now take the key through interpolation ({{key}}, and
{{audioKey}} / {{voiceoverKey}} for the audio lane), formatted with
formatBinding from the live bindings, the same way the toolbar tooltip chip
is. All 15 locales keep their wording; only the letter became a placeholder.
Fixes getopenscreen#966
On a Mac with no camera, camera access is still granted, so the HUD toggle reported success and stored camEnabled: true before the stream failed with NotFoundError. The HUD then showed the camera off while Record mode read On from the stored preference. The toggle now checks that the OS lists a camera before switching on, and a failed acquire stores camEnabled: false so a stale On is cleared too. Fixes getopenscreen#967.
…k taken for camera 1 by name
…ers read the document at write time
…e lock hint on the switches
The lane read the layout list through the role filter of the current main camera, so a camera-full row whose camera became the main camera vanished from the lane while the scene still drew it and the lane still counted it as occupied. The role rule now decides storage only; the displayed list keeps every valid stored row.
The agent's overlap check and listing read the layout list through the role filter of camera 1, so while another camera was main a camera-full section of camera 1 neither blocked a Full Camera add nor showed in getCurrentDocument. Both now read every valid stored row, as the editor lane does.
…d log the re-runs
The camera list matched a saved pick only by its id, while the recording request resolves a stale id by name. Both now go through resolveAdditionalCameraPicks, so the checked rows, the cap and the toggles follow the cameras the request carries.
…a picks The desk pick matched by id alone, so a camera whose id changed showed as None and was not marked in the take. deskCameraIndex now falls back to the name once the id is gone, recorded extras before camera 1, and the HUD desk list checks the camera it returns.
A stale desk pick fell back to the first recorded camera of its name, which is not always where the resolver put that pick. deskCameraIndex now takes the resolver's pick/device pairs: it uses the extra the desk pick's own saved pick resolved to, else the first recorded camera of that name in enumeration order. Camera 1 is matched by id only while a plugged-in camera carries the pick's id, and present is now required.
…es past its picture A camera layer with a homography multiplied its alpha by the in-picture flag of `persp_camera`, so where the corrected rectangle reached past the camera image the layer was transparent and the wallpaper showed through. The spec wants that edge black. The flag now blacks out the colour (and drops the cutout mask there) instead of the alpha, so the edge stays as opaque as the layer and still follows its fades and glides. Same change in HLSL, WGSL and MSL; layers without a homography are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI trigger for Linux/macOS compositor tests and MSL. Not for merge; will be closed.