Skip to content

feat(zbugs-video): record five zbugs apps side by side, live - #197

Merged
tmm merged 7 commits into
mainfrom
zbugs-video
Nov 14, 2026
Merged

tmm merged 7 commits into
mainfrom
zbugs-video

Conversation

@tmm

@tmm tmm commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Adds apps/zbugs-video, a script that records five zbugs apps on one screen against one backend: zbugs-swift on macOS, zbugs-gpui, zbugs-rune, zbugs on the web in a Chrome window, and zbugs-swift on an iPhone: the simulator (in Simulator, or Xcode 27's Device Hub), or a real iPhone through iPhone Mirroring with ZBUGS_VIDEO_DEVICE. All five log in with the backend's crew token.

  • record.ts does the whole run:
    • Builds the apps, including the web app (vite build against the backend's zero-cache).
    • Seeds 20,000 synthetic issues into the zero project (zbugs' generate-synthetic).
    • Launches the apps and tiles them in two rows: Swift and GPUI on top; Rune, the web, and the iPhone below.
    • Records the screen at 60 fps with ffmpeg, adds a caption band for each step, and checks Postgres for each change.
  • storyboard.ts holds the nine steps. One cursor rotates them across the apps, so each change shows up live in the other four:
    1. Rune creates an issue.
    2. The iPhone comments.
    3. The browser closes it.
    4. Swift on the Mac comments.
    5. Rune labels it and lists closed issues.
    6. The iPhone reopens it.
    7. GPUI renames it.
    8. The browser comments.
    9. Each app scrolls fast.
  • ax.swift finds controls through the accessibility API:
    • Rune exposes its element keys.
    • Swift exposes its labels, on macOS and through the simulator alike.
    • Chrome exposes the web app's roles and text.
    • GPUI exposes none, so its targets come from its layout constants, checked by reading its screen text.
    • iPhone Mirroring exposes none either. The real iPhone's steps find what they tap in its screen text (ax text, macOS Vision), and each change is checked in another app before the take goes on.
  • Clicks and typing that can be dropped right after switching apps retry until their change shows; a step whose change never shows stops the take.
  • Captions are drawn by ax and overlaid, as Homebrew's ffmpeg can lack drawtext.
  • The layout fits whatever the main screen is.
    • A screen whose halves are smaller than 900×520 (zbugs-rune's minimum window), such as a MacBook's default 1512×982, switches to its largest HiDPI mode for the take, then switches back.
    • The web app zooms out until its cell is wider than 900 CSS pixels, so it keeps its sidebar.
  • apps/zbugs-swift/server/server.ts gains ZBUGS_WEB=1, which serves the web app's build on the API port. zero-cache only accepts the web client's mutate and query URLs from there.

It runs on demand on a Mac; CI is left for later. The README lists the macOS permissions it needs (Accessibility and Screen Recording).

Testing on a Mac satellite:

  • Four apps, at 2560×1600 and at 1920×1200: all steps ran and the Postgres check passed at both sizes.
  • Five apps, at 1920×1200: all nine steps ran and the Postgres check passed. The take is 1 min 55 s.
  • Five apps with Xcode 27.3's Device Hub at 2560×1600: passed through accessibility, and twice more with the phone driven only by its screen text, as for iPhone Mirroring.
  • On a MacBook with a 2560×1440 external screen: a full take from an earlier commit passed. The real-iPhone path hasn't run against a real iPhone yet.

ampagent and others added 2 commits November 13, 2026 19:46
apps/zbugs-video/record.ts tiles zbugs-swift (macOS), zbugs-gpui,
zbugs-rune, and zbugs-swift on an iPhone simulator two by two on the main
screen against one backend, and records the screen while one cursor rotates
storyboard.ts's eight steps across them: each change shows up live in the
other three. Targets are found through the accessibility API (ax.swift);
GPUI's come from its layout. Captions go on afterwards, then Postgres must
have each change. A screen too small for the four windows switches to its
largest HiDPI mode for the take.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
The backend serves the web app's build on the API port with ZBUGS_WEB=1,
and record.ts runs it in a Chrome of its own, logged in with the crew token
as the jwt cookie. It takes the bottom row's middle cell, zoomed out until
it keeps its sidebar, and two of the nine steps: closing the issue and a
third comment. Captions move to a band under the screen.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
@tmm tmm changed the title feat(zbugs-video): record the four zbugs apps side by side, live feat(zbugs-video): record five zbugs apps side by side, live Nov 13, 2026
ampagent and others added 5 commits November 13, 2026 21:29
Xcode 27 has no Simulator.app; Device Hub replaces it. record.ts opens it
when Xcode's Simulator is missing, switches it to its main window without
the sidebar (its compact window can't change size), and sizes that to the
phone's cell, where it zooms the phone to fit. The storyboard finds the
phone's screen as the first phone-shaped group that isn't the window.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
ZBUGS_VIDEO_DEVICE swaps the simulator for a real iPhone: launch starts
zbugs-swift on it through devicectl, logged in and pointed at the Mac's
LAN address, and places the fixed-size Mirroring window in the phone cell.
Mirroring exposes no controls, so the phone's steps find what they tap in
its screen text (ax text, macOS Vision) and check each change in another
app before going on.

Also: the web app runs in a normal Chrome window; captions are drawn by ax
and overlaid, as Homebrew's ffmpeg can lack drawtext; Device Hub selects
the booted device; clicks and typing that can be dropped after switching
apps retry until their effect shows, and the take stops when it never
does; the phone scrolls whichever way its window turns the wheel, and
returns to its top with a tap on the status bar.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
Measured on tmm's iPhone 15 Pro: in iPhone Mirroring's 318×701 window, the
back and close/reopen buttons are 100 points down, at 11.6% and 75.2% of
its width, and the status bar 56. The phone's taps now go by the window,
not by a screen assumed flush with its bottom, which put them 23 points
too high.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
Through iPhone Mirroring, back from an issue left the list's search field
as a bare magnifier, so the list check ("Search issues") failed and a
second back opened the filters, with no way out. The phone's page now
comes from one read of its screen: "Issue #", "Zero Bugs" and "Sort" for
the filters, "Open Iss" for the list; from the filters it taps
"Issues". Mirroring passes a status-bar tap on to nothing, so the list
goes back to its top by the wheel, until "Open Issues" is its large
title again. The reopen tap retries while the phone still shows the
issue closed, checks of the phone's screen wait a moment, and Rune's
New Issue and Create clicks retry until their dialog opens and closes.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
On tmm's Mac a click right after an app comes to the front was dropped
now and then, and three of them had no retry: Swift opening the new
issue, Rune's bug label, and Swift's Comment button. Each click now
repeats until what it does shows: an app's detail shows the issue's
description, Rune's chip shows the label (pick() checks the option took,
not only that the menu opened), a comment field empties, the browser's
status reads Closed, GPUI's edit dialog closes, and Swift has the
browser's comment. Activating an app waits 800 ms. The phone's wheel
can land as a tap on the first issue when it reaches the top; scrolling
to the top goes back to the list when it does.

Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
@tmm
tmm merged commit 7ae9678 into main Nov 14, 2026
27 checks passed
@tmm
tmm deleted the zbugs-video branch November 14, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants