Repository navigation
feat(zbugs-video): record five zbugs apps side by side, live - #197
Merged
Merged
Conversation
apps/zbugs-video/record.ts tiles zbugs-swift (macOS), zbugs-gpui, zbugs-rune, and zbugs-swift on an iPhone simulator two by two on the main screen against one backend, and records the screen while one cursor rotates storyboard.ts's eight steps across them: each change shows up live in the other three. Targets are found through the accessibility API (ax.swift); GPUI's come from its layout. Captions go on afterwards, then Postgres must have each change. A screen too small for the four windows switches to its largest HiDPI mode for the take. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
The backend serves the web app's build on the API port with ZBUGS_WEB=1, and record.ts runs it in a Chrome of its own, logged in with the crew token as the jwt cookie. It takes the bottom row's middle cell, zoomed out until it keeps its sidebar, and two of the nine steps: closing the issue and a third comment. Captions move to a band under the screen. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
Xcode 27 has no Simulator.app; Device Hub replaces it. record.ts opens it when Xcode's Simulator is missing, switches it to its main window without the sidebar (its compact window can't change size), and sizes that to the phone's cell, where it zooms the phone to fit. The storyboard finds the phone's screen as the first phone-shaped group that isn't the window. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
ZBUGS_VIDEO_DEVICE swaps the simulator for a real iPhone: launch starts zbugs-swift on it through devicectl, logged in and pointed at the Mac's LAN address, and places the fixed-size Mirroring window in the phone cell. Mirroring exposes no controls, so the phone's steps find what they tap in its screen text (ax text, macOS Vision) and check each change in another app before going on. Also: the web app runs in a normal Chrome window; captions are drawn by ax and overlaid, as Homebrew's ffmpeg can lack drawtext; Device Hub selects the booted device; clicks and typing that can be dropped after switching apps retry until their effect shows, and the take stops when it never does; the phone scrolls whichever way its window turns the wheel, and returns to its top with a tap on the status bar. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
Measured on tmm's iPhone 15 Pro: in iPhone Mirroring's 318×701 window, the back and close/reopen buttons are 100 points down, at 11.6% and 75.2% of its width, and the status bar 56. The phone's taps now go by the window, not by a screen assumed flush with its bottom, which put them 23 points too high. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
Through iPhone Mirroring, back from an issue left the list's search field
as a bare magnifier, so the list check ("Search issues") failed and a
second back opened the filters, with no way out. The phone's page now
comes from one read of its screen: "Issue #", "Zero Bugs" and "Sort" for
the filters, "Open Iss" for the list; from the filters it taps
"Issues". Mirroring passes a status-bar tap on to nothing, so the list
goes back to its top by the wheel, until "Open Issues" is its large
title again. The reopen tap retries while the phone still shows the
issue closed, checks of the phone's screen wait a moment, and Rune's
New Issue and Create clicks retry until their dialog opens and closes.
Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1
Co-authored-by: tmm <tmm@tmm.dev>
On tmm's Mac a click right after an app comes to the front was dropped now and then, and three of them had no retry: Swift opening the new issue, Rune's bug label, and Swift's Comment button. Each click now repeats until what it does shows: an app's detail shows the issue's description, Rune's chip shows the label (pick() checks the option took, not only that the menu opened), a comment field empties, the browser's status reads Closed, GPUI's edit dialog closes, and Swift has the browser's comment. Activating an app waits 800 ms. The phone's wheel can land as a tap on the first issue when it reaches the top; scrolling to the top goes back to the list when it does. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
apps/zbugs-video, a script that records five zbugs apps on one screen against one backend: zbugs-swift on macOS, zbugs-gpui, zbugs-rune, zbugs on the web in a Chrome window, and zbugs-swift on an iPhone: the simulator (in Simulator, or Xcode 27's Device Hub), or a real iPhone through iPhone Mirroring withZBUGS_VIDEO_DEVICE. All five log in with the backend's crew token.record.tsdoes the whole run:vite buildagainst the backend's zero-cache).generate-synthetic).storyboard.tsholds the nine steps. One cursor rotates them across the apps, so each change shows up live in the other four:ax.swiftfinds controls through the accessibility API:ax text, macOS Vision), and each change is checked in another app before the take goes on.axand overlaid, as Homebrew's ffmpeg can lackdrawtext.apps/zbugs-swift/server/server.tsgainsZBUGS_WEB=1, which serves the web app's build on the API port. zero-cache only accepts the web client's mutate and query URLs from there.It runs on demand on a Mac; CI is left for later. The README lists the macOS permissions it needs (Accessibility and Screen Recording).
Testing on a Mac satellite: