Skip to content

ci: compositor black outside the camera picture (CI trigger only) - #2

Closed
christian-wr wants to merge 357 commits into
mainfrom
ci/compositor-black
Closed

christian-wr wants to merge 357 commits into
mainfrom
ci/compositor-black

Conversation

@christian-wr

Copy link
Copy Markdown
Owner

CI trigger for Linux/macOS compositor tests and MSL. Not for merge; will be closed.

…nged screen

The video writer read the full GPU frame back on every tick, even when
WGC delivered nothing new: a CopyResource, a Map that waits on the GPU,
and the BGRA to NV12 conversion, for a picture it already had. On a
static screen, most of a demo, that was every tick.

When no new WGC frame arrived and no new camera frame is drawn into the
picture, the encoder now hands over a new sample on the last buffer.
Measured at 1080p: 5.0 ms per readback, under 0.001 ms per repeat. The
DXGI path and the legacy callback path read back as before.

Fixes getopenscreen#925
whisper.cpp's DTW pass teacher-forces the decoded BPE tokens; ours feeds the
text back one character per decoder row (arXiv 2509.09987), averages the
alignment heads with per-frame L2 normalisation, and drops one silent final
consonant per French word. It is spliced into a build-tree copy of
whisper.cpp at configure time, so the fetched source stays untouched.

Word-timing harness, inner starts: median 31 -> 16 ms, within 50 ms
64% -> 86% (FR 83%, EN 89%); clean single-word cuts 19% -> 39%.

Refs getopenscreen#948
A second alignment pass (issue getopenscreen#948, phase 3): a wav2vec2 CTC model scores
every 20 ms frame of the speech against every letter, and a Viterbi pass forces
whisper's own words through those scores. English uses wav2vec2-base-960h
(109 MB), French wav2vec2-large-xlsr-53-french (348 MB), both Apache-2.0,
converted to Q8_0 GGUF and run on ggml inside whisper-stt-server, on the GPU
whisper uses. The helper only scores (POST /emissions); spelling, Viterbi,
calibration and fallback live in electron/stt/ctcAlign.ts. Other languages,
a failed download or an older helper keep the phase 1 times.

TTS corpus, clean: inner start median 31 -> 14 ms, P90 125 -> 35 ms, within
50 ms 64% -> 96%; clean single-word cuts 19% -> 50%; phrase delete 89% -> 96%.
LibriSpeech vs MFA: inner median 40 -> 15 ms, within 50 ms 55% -> 92%, clean
cuts 14% -> 44%. Extra runtime about +10% on Vulkan.
Address the review of getopenscreen#954:
- the aligner downloads in the background; chunks before it lands keep
  whisper's times, a cached copy is verified in place; a 30 s stall, Cancel
  and quit abort it, and a failed one is retried on the next transcription
- accept an emissions region ending within two frames of a stretch that runs
  to the end of the upload
- the harness skips (evaluate) or names (real-check) unreadable emissions
- test-whisper-stt says which aligner file it looked for; the doc lists the
  aligner checks it runs
…se 2

On top of character-level DTW, the CTC aligner still moves TTS inner starts
from 17/60 ms (median/P90) to 14/35 ms and LibriSpeech from 20/65 to 15/45 ms,
for +9% (English) to +14% (French) on Vulkan. On the CPU it would cost +29% to
+58%, so a chunk whisper ran on the CPU keeps the DTW times, and a CPU-only
machine never downloads the model.
…ter trial

The energy at 10 ms puts "quoi" on the start of its /k/ closure, where the
aligner has it; the earlier spectrogram reading had it early. Dropping French
silent final consonants before aligning, as char-dtw does, moved none of the
take's boundaries by more than 15 ms and cost 1 to 2 points of French phrase
deletes, so it is left out.
Switching the bar to vertical and back left it about 130 DIP higher each
round trip (Windows 11, 125 %). Right after hud-overlay-set-size the
ResizeObserver fires again while the old viewport is still laid out, so
the hud-overlay-content that follows carries a rect measured in the
window the main process has just replaced. The main process stores it
as the anchor, and the next flip re-anchors the window on it: the window
keeps its top-left instead of the bar's bottom-centre.

The stack is always centred and pinned HUD_BAR_BOTTOM above the bottom
edge, which is how grantedContent is already computed. The reported rect
now uses the same formula on the allocated size, and only its size comes
from the measurement.

Fixes getopenscreen#951
…s rejected

The staging script took the first 50 runs `gh run list --status success`
returned and trusted their order. That order is not stable: during the
v2.0.0-rc.5 build the same query listed this morning's runs first in one
job and August's in another, so three jobs found the matching helper and
two failed with "no successful build-whisper-stt run". It now reads every
successful run, sorts them by creation date itself, and reads the list a
second time before giving up.

A rejected run now says which source differs and how, a commit that no
longer exists is skipped quietly instead of printing a 404, and any other
API error is retried, then fatal, instead of passing for a mismatch.
…a broken listing

GitHub caps status-filtered run queries at 1,000 results, so the success
filter now runs locally. The list is read into a file with its exit status
checked and retried: from a process substitution, a failed or partial
listing passed for a complete one.
… own proportions

The picture-in-picture camera now starts square. The other shape, which follows the
camera's own aspect (a portrait camera stays portrait), is labelled "Original" instead
of "Rectangle".

Each shape has its own default roundness, 70% for Square and 40% for Original. An
untouched roundness follows the shape when it changes; a tuned one is kept. Projects
saved with a shape keep the look they had.

The editor's drag box was laid out from a fixed 4:3 camera while the native canvas drew
the real one. Both now read webcamBoxSourceSize, so the box matches the picture.
DTW can anchor a sentence's last word in the pause after it, past the kept
tail. The next stretch then owned it, put it on its onset after the words
that follow it, and the aligner looked for it in the wrong audio (issue getopenscreen#948).
Ownership is now one function, read by both passes; a sentence-final word
anchored before the next stretch's speech closes the stretch before it.
Word starts also come out monotonic and never inverted.
A separate "(" after a pause-anchored "b." made the scan read "b.(", no
sentence end, so "b." opened the next stretch again.
The menu is absolutely positioned under an anchor only as wide as the wordmark, so shrink-to-fit settled on its 232px min-width and French wrapped "Changer de langue" and split the version as "2.0.0-" / "rc.8". Size it to its widest row with width: max-content (capped to the viewport), keep rows on one line and never shrink or break the trailing value.
A screen picked in Apple's system picker only leaves out the windows the
picker was told about when it was shown: their ids are baked into the
picked filter, and a take cannot add one without the Screen Recording
grant the picker exists to avoid. The HUD is a new window after every
trip through the editor, so every take after the first one reused a pick
that did not exclude it, and recorded the HUD and its timer.

The picker session now remembers which windows each pick excludes, and
the main process forgets a screen pick once the current HUD or notes
window is not among them. `get-selected-source` then answers null, so
the next Record presents the picker again with the current ids. Record
start applies the same check as a last guard. Window picks and a take
already running are left alone.

Fixes getopenscreen#965
The empty-lane hints wrote the default letter into the string ("Press Z to
add zoom"), so after a rebind in Keyboard Shortcuts the hint kept teaching a
key that did nothing while the toolbar chip showed the right one.

The hints now take the key through interpolation ({{key}}, and
{{audioKey}} / {{voiceoverKey}} for the audio lane), formatted with
formatBinding from the live bindings, the same way the toolbar tooltip chip
is. All 15 locales keep their wording; only the letter became a placeholder.

Fixes getopenscreen#966
On a Mac with no camera, camera access is still granted, so the HUD toggle reported success and stored camEnabled: true before the stream failed with NotFoundError. The HUD then showed the camera off while Record mode read On from the stored preference.

The toggle now checks that the OS lists a camera before switching on, and a failed acquire stores camEnabled: false so a stale On is cleared too. Fixes getopenscreen#967.
The lane read the layout list through the role filter of the current main camera, so a camera-full row whose camera became the main camera vanished from the lane while the scene still drew it and the lane still counted it as occupied. The role rule now decides storage only; the displayed list keeps every valid stored row.
The agent's overlap check and listing read the layout list through the role filter of camera 1, so while another camera was main a camera-full section of camera 1 neither blocked a Full Camera add nor showed in getCurrentDocument. Both now read every valid stored row, as the editor lane does.
The camera list matched a saved pick only by its id, while the recording request resolves a stale id by name. Both now go through resolveAdditionalCameraPicks, so the checked rows, the cap and the toggles follow the cameras the request carries.
…a picks

The desk pick matched by id alone, so a camera whose id changed showed as None and was not marked in the take. deskCameraIndex now falls back to the name once the id is gone, recorded extras before camera 1, and the HUD desk list checks the camera it returns.
A stale desk pick fell back to the first recorded camera of its name, which is not always where the resolver put that pick. deskCameraIndex now takes the resolver's pick/device pairs: it uses the extra the desk pick's own saved pick resolved to, else the first recorded camera of that name in enumeration order. Camera 1 is matched by id only while a plugged-in camera carries the pick's id, and present is now required.
…es past its picture

A camera layer with a homography multiplied its alpha by the in-picture
flag of `persp_camera`, so where the corrected rectangle reached past the
camera image the layer was transparent and the wallpaper showed through.
The spec wants that edge black. The flag now blacks out the colour (and
drops the cutout mask there) instead of the alpha, so the edge stays as
opaque as the layer and still follows its fades and glides. Same change
in HLSL, WGSL and MSL; layers without a homography are untouched.
@christian-wr
christian-wr deleted the ci/compositor-black branch October 7, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants