Skip to content

feat(linux): auto-sample X11 cursor telemetry when portal metadata is missing - #1

Open
Jasonsu22 wants to merge 326 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6
Open

Jasonsu22 wants to merge 326 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6

Conversation

@Jasonsu22

@Jasonsu22 Jasonsu22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

On X11 + Xvfb hosts the ScreenCast portal often has no METADATA cursor mode, so editable-cursor takes used to write no sidecar. This PR samples the pointer automatically during record (Electron screen.getCursorScreenPoint(), then xdotool / xinput / xev / libinput) and writes version-2 <video>.cursor.json next to the take. openscreen export --auto-zoom then works with no extra import step.

Also adds openscreen cursor import for an external ffmpeg/x11grab CSV track.

Follow-up on this branch addresses CodeRabbit’s review of the first revision: discard stops the X11 sampler, fallback samples are compacted by pause ranges, the sampler clock starts at capture start, cursor import CSV/write guards are stricter, XI2 rest no longer accumulates, click watchers fail over on child exit, and the browser fallback keeps the source it just picked.

Related issue

Fixes getopenscreen#1415

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Repro host (from getopenscreen#1415)

Item Value
Host Grok Bot cloud Linux (grok-bot-vm-316768464)
OS Debian 13 (trixie), kernel 6.12.94+ x86_64
Session X11 DISPLAY=:2, Xvfb 1920×1200×24
Portal ScreenCast METADATA unavailable (AvailableSourceTypes=0)
GPU Mesa llvmpipe (CPU export fallback)
OpenScreen tested Linux AppImage 2.1.2

Behavior

  • Auto path: X11 (incl. Xvfb) — PipeWire fail → Electron/xdotool/xinput/xev/libinput; native Linux capture with empty cursor track fills sidecar via parallel X11 sampler
  • CLI: openscreen cursor import --csv mouse.csv --clicks clicks.txt --video take.mp4
  • Env: OPENSCREEN_DISABLE_X11_CURSOR=1 / OPENSCREEN_FORCE_X11_CURSOR=1

Testing

  • Unit tests for sampler / sidecar import / CLI parser / factory (vitest)
  • tsc --noEmit and tsc -p tsconfig.test.json --noEmit
  • Targeted vitest after the CodeRabbit follow-up
  • Maintainer smoke on a normal Linux desktop (should be no-op when METADATA works)
  • Optional: X11/Xvfb agent host record → sidecar present → export --auto-zoom

Notes

Upstream PR (same branch): getopenscreen#1416

Open in Web Open in Cursor 

juliogargue and others added 28 commits November 4, 2026 22:28
xdg-desktop-portal-cosmic picks the modifier-carrying EnumFormat even in
second place, but only offers MemFd buffers for it, so the DmaBuf-only
Buffers param fails with "error alloc buffers". The shared-memory retry
offered the same two objects and failed the same way. It now offers the
shm EnumFormat alone, and is armed whether or not the DMA-BUF import is
preferred.

Refs getopenscreen#1445
ffmpeg 8.1 segfaults while cleaning up a h264_vulkan encoder that failed
with "Encoding of h264 is not supported by this device", so the ladder
never reached libopenh264 (getopenscreen#1340). Ask Vulkan for an H.264 encode queue
family before avcodec_open2 and skip the rung when there is none.
A hidden Openscreen mcp process answered every reopen Apple Event by starting the GUI, so OpenScreen came to the front out of nowhere for each agent session still open. macOS only routes a real launch to a CLI process that is in the Dock (getopenscreen#1214); a hidden one now ignores activate. Refs getopenscreen#1488.
On X11, KDE's portal has no ScreenCast, so every take died on
"CreateSession: Portal request didn't succeed: Other". Treat the portal
helper as absent outside Wayland: capture falls back to Chromium, which
X11 serves without a portal, and the cursor to screen telemetry.

Refs getopenscreen#1491
A cutout camera has no bubble, so the compositor draws no shadow there.
A disabled row named a setting with no effect; hide it instead.
…rame rate

The pool had two buffers and the writer held one until the next tick, so
under GPU load WGC dropped every other frame. Pool of three, drain the
queue to the newest frame, hand the buffer back right after the copy, and
log unique frames per second in [pacing].

Refs getopenscreen#510
Every job that reads a release credential now declares environment:
release. Once the credentials move from repository secrets into that
environment, only its approver can run them: write access no longer
reaches the signing certificates or the release token from a branch
workflow, nor cuts a release through workflow_dispatch.
…ove the secrets

Write access can still publish a release by hand; variables stay at
repository level; old release branches need the environment lines
before the credentials move; repository copies must be deleted; the
token is read from a prompt, not the command line.
The rules Rusuban's agents follow on this repo, read from the default branch: what they do alone
and what comes to the maintainer, what merges alone and how this repo merges, review rules, and
how they write on GitHub. locks.json holds what the server checks without the model: supply-chain
paths, size limits, the zones that need a rendering and performance proof, CodeRabbit required, and
the Nix build and notification workflows ignored.
…en, contributor PRs kept

From the review: *.spec.* files are tests; electron/native-bridge/ is capture code and needs the
rendering proof; a fork that refuses maintainer edits gets a fallback that keeps the contributor's
commits; a worker finishes a contributor's PR on their branch, never in a competing one.
EtienneLescot and others added 29 commits November 17, 2026 01:03
… missing

On X11 (including Xvfb cloud agents) the ScreenCast portal often has no
METADATA cursor mode, so editable-cursor takes used to write no sidecar.
Sample the pointer via Electron/xdotool/xev during record, write the
version-2 <video>.cursor.json next to the take, and add
`openscreen cursor import` for an external CSV. export --auto-zoom then
works with no extra step.
Stop the X11 sampler on discard, compact pause ranges into the fallback
sidecar, start the sampler clock at capture start, harden cursor import
CSV parsing/writes, trim XI2 rest, fail over click watchers on exit,
and pass the browser-fallback source through instead of re-reading
getSelectedSource().
Drop fallbackToBrowser so cancelling the portal picker cannot record
the first screen, move cursor import out of this change, and pass the
helper's granted capture bounds to the X11 pointer sampler.
@cursor
cursor Bot force-pushed the cursor/x11-cursor-telemetry-fallback-19b6 branch from 4d76cd5 to 5fd254a Compare November 17, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linux X11/Xvfb: auto-sample cursor telemetry when ScreenCast METADATA is unavailable