Skip to content

Commit 7ae9678

Browse files
tmmampagent
andauthored
feat(zbugs-video): record five zbugs apps side by side, live (#197)
* feat(zbugs-video): record the four zbugs apps side by side, live apps/zbugs-video/record.ts tiles zbugs-swift (macOS), zbugs-gpui, zbugs-rune, and zbugs-swift on an iPhone simulator two by two on the main screen against one backend, and records the screen while one cursor rotates storyboard.ts's eight steps across them: each change shows up live in the other three. Targets are found through the accessibility API (ax.swift); GPUI's come from its layout. Captions go on afterwards, then Postgres must have each change. A screen too small for the four windows switches to its largest HiDPI mode for the take. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: add zbugs on the web as a fifth app The backend serves the web app's build on the API port with ZBUGS_WEB=1, and record.ts runs it in a Chrome of its own, logged in with the crew token as the jwt cookie. It takes the bottom row's middle cell, zoomed out until it keeps its sidebar, and two of the nine steps: closing the issue and a third comment. Captions move to a band under the screen. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: show the iPhone in Xcode 27's Device Hub Xcode 27 has no Simulator.app; Device Hub replaces it. record.ts opens it when Xcode's Simulator is missing, switches it to its main window without the sidebar (its compact window can't change size), and sizes that to the phone's cell, where it zooms the phone to fit. The storyboard finds the phone's screen as the first phone-shaped group that isn't the window. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: record a real iPhone through iPhone Mirroring ZBUGS_VIDEO_DEVICE swaps the simulator for a real iPhone: launch starts zbugs-swift on it through devicectl, logged in and pointed at the Mac's LAN address, and places the fixed-size Mirroring window in the phone cell. Mirroring exposes no controls, so the phone's steps find what they tap in its screen text (ax text, macOS Vision) and check each change in another app before going on. Also: the web app runs in a normal Chrome window; captions are drawn by ax and overlaid, as Homebrew's ffmpeg can lack drawtext; Device Hub selects the booted device; clicks and typing that can be dropped after switching apps retry until their effect shows, and the take stops when it never does; the phone scrolls whichever way its window turns the wheel, and returns to its top with a tap on the status bar. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: tap Mirroring's nav buttons where they are Measured on tmm's iPhone 15 Pro: in iPhone Mirroring's 318×701 window, the back and close/reopen buttons are 100 points down, at 11.6% and 75.2% of its width, and the status bar 56. The phone's taps now go by the window, not by a screen assumed flush with its bottom, which put them 23 points too high. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: find the mirrored iPhone's pages, and its way back Through iPhone Mirroring, back from an issue left the list's search field as a bare magnifier, so the list check ("Search issues") failed and a second back opened the filters, with no way out. The phone's page now comes from one read of its screen: "Issue #", "Zero Bugs" and "Sort" for the filters, "Open Iss" for the list; from the filters it taps "Issues". Mirroring passes a status-bar tap on to nothing, so the list goes back to its top by the wheel, until "Open Issues" is its large title again. The reopen tap retries while the phone still shows the issue closed, checks of the phone's screen wait a moment, and Rune's New Issue and Create clicks retry until their dialog opens and closes. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> * zbugs-video: retry every click until its change shows On tmm's Mac a click right after an app comes to the front was dropped now and then, and three of them had no retry: Swift opening the new issue, Rune's bug label, and Swift's Comment button. Each click now repeats until what it does shows: an app's detail shows the issue's description, Rune's chip shows the label (pick() checks the option took, not only that the menu opened), a comment field empties, the browser's status reads Closed, GPUI's edit dialog closes, and Swift has the browser's comment. Activating an app waits 800 ms. The phone's wheel can land as a tap on the first issue when it reaches the top; scrolling to the top goes back to the list when it does. Amp-Thread-ID: https://ampcode.com/threads/T-07b9f486-ecbb-4cc4-b74d-6fa9ecb9dca1 Co-authored-by: tmm <tmm@tmm.dev> --------- Co-authored-by: Amp <amp@ampcode.com>
1 parent 2fb54ea commit 7ae9678

5 files changed

Lines changed: 2015 additions & 1 deletion

File tree

‎apps/zbugs-swift/server/server.ts‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,11 @@
3030
* ZBUGS_KEEP 1 to keep the last run's database, and the replica and
3131
* keys in ZBUGS_DIR, which it needs, and to restart
3232
* zero-cache when it exits
33+
* ZBUGS_WEB 1 to serve the web app too, from its build
34+
* (apps/zbugs/dist, `vite build` with VITE_PUBLIC_SERVER
35+
* set to zero-cache's URL), on the API's port: zero-cache
36+
* takes the web client's mutate and query URLs, which
37+
* it makes from its page's origin, from there only
3338
* ZBUGS_PG_START with ZBUGS_KEEP, a shell command that starts Postgres,
3439
* run once Postgres has refused three checks 10 seconds
3540
* apart (for example `docker compose -f
@@ -42,11 +47,12 @@ import {
4247
readFileSync,
4348
readdirSync,
4449
rmSync,
50+
statSync,
4551
writeFileSync,
4652
} from 'node:fs';
4753
import {registerHooks} from 'node:module';
4854
import {networkInterfaces, tmpdir} from 'node:os';
49-
import {join} from 'node:path';
55+
import {extname, join} from 'node:path';
5056
import {fileURLToPath, pathToFileURL} from 'node:url';
5157
import {createJwkPair} from '../../../packages/zero-cache/src/auth/jwt.ts';
5258
import {postgresAnswers, watchPostgres} from './postgres.ts';
@@ -196,8 +202,42 @@ const {fastify} = (await import('../../zbugs/api/index.ts')) as {
196202
fastify: {
197203
listen(o: {port: number; host: string}): Promise<string>;
198204
close(): Promise<void>;
205+
setNotFoundHandler(
206+
handler: (
207+
request: {url: string},
208+
reply: {type(t: string): {send(body: Buffer): void}},
209+
) => void,
210+
): void;
199211
};
200212
};
213+
if (env['ZBUGS_WEB'] === '1') {
214+
// Each path the API doesn't have is the web app's: a file of its build,
215+
// or else its page, which routes in the browser.
216+
const dist = join(zbugs, 'dist');
217+
const types: Record<string, string> = {
218+
'.html': 'text/html; charset=utf-8',
219+
'.js': 'text/javascript',
220+
'.css': 'text/css',
221+
'.svg': 'image/svg+xml',
222+
'.png': 'image/png',
223+
'.ico': 'image/x-icon',
224+
'.woff2': 'font/woff2',
225+
'.json': 'application/json',
226+
};
227+
fastify.setNotFoundHandler((request, reply) => {
228+
let file = join(dist, decodeURIComponent(request.url.split('?')[0]));
229+
if (
230+
!file.startsWith(dist) ||
231+
!existsSync(file) ||
232+
statSync(file).isDirectory()
233+
) {
234+
file = join(dist, 'index.html');
235+
}
236+
reply
237+
.type(types[extname(file)] ?? 'application/octet-stream')
238+
.send(readFileSync(file));
239+
});
240+
}
201241
await fastify.listen({port: apiPort, host});
202242

203243
function startZeroCache() {

‎apps/zbugs-video/README.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# A video of the five zbugs apps
2+
3+
`record.ts` records five zbugs apps side by side on one screen, against one
4+
backend: zbugs-swift on macOS, [zbugs-gpui](../zbugs-gpui),
5+
[zbugs-rune](../zbugs-rune), [zbugs](../zbugs) on the web in Chrome, and
6+
zbugs-swift on an iPhone simulator. All five log in with the backend's crew
7+
token. One cursor and keyboard rotate the storyboard's steps
8+
(`storyboard.ts`) across the apps, so each change shows up live in the other
9+
four:
10+
11+
1. Rune creates an issue; the others open it.
12+
2. The iPhone comments.
13+
3. The browser closes it.
14+
4. Swift on the Mac answers.
15+
5. Rune labels it, then lists closed issues.
16+
6. The iPhone reopens it.
17+
7. GPUI renames it.
18+
8. The browser comments.
19+
9. Each app scrolls fast through its list of 20,000 issues.
20+
21+
The iPhone shows in Xcode 26's Simulator or, with Xcode 27, in Device Hub,
22+
which replaces it. Steps that can drop a click retry until their change
23+
shows, and a step whose change never shows stops the take.
24+
25+
Each step's caption goes on afterwards, in a band under the screen. Then
26+
Postgres must have each change. A take runs about two minutes.
27+
28+
The windows tile the main screen in two rows: Swift and GPUI on top; Rune,
29+
the web, and the iPhone, as tall as the row, below. A screen too small for
30+
that (each half needs zbugs-rune's smallest window, 900×520 points), such as
31+
a MacBook's built-in screen at its default 1512×982, switches to its largest
32+
HiDPI mode for the take (1800×1169 on a 14-inch MacBook Pro) with
33+
`displayplacer`, then back; `ZBUGS_VIDEO_DISPLAY=keep` leaves it alone. The
34+
web app zooms out (Chrome's Cmd-minus) until its cell is wider than 900 CSS
35+
pixels, where it keeps its sidebar. The screen is captured whole at 60 fps
36+
(ffmpeg's AVFoundation input, with the cursor and its clicks) and encoded at
37+
up to 3840 wide.
38+
39+
## Running it
40+
41+
It needs macOS with Xcode (26 or later), Rust, Node 24, `psql`, Google
42+
Chrome, and:
43+
44+
```sh
45+
brew install ffmpeg cliclick xcodegen displayplacer
46+
```
47+
48+
The app that runs it, such as Terminal or the Amp runner's terminal, needs
49+
two permissions in System Settings → Privacy & Security, which macOS asks for
50+
the first time:
51+
52+
- **Accessibility**, for `cliclick` to move the cursor and type, and for
53+
`ax` (`ax.swift`) to find the apps' controls and place their windows.
54+
- **Screen & System Audio Recording**, for ffmpeg to capture the screen.
55+
56+
Quit and reopen that app after granting them; `~/.zbugs-video/ax
57+
permissions` says whether it has them, once `build` has run. Then start
58+
zbugs' backend, as for zbugs-swift (see `apps/zbugs-swift/README.md`), with
59+
`ZBUGS_WEB=1` so it serves the web app's build too; it prints the crew token:
60+
61+
```sh
62+
ZBUGS_WEB=1 ZBUGS_PG=postgresql://user:password@127.0.0.1:6434/postgres \
63+
node apps/zbugs-swift/server/server.ts
64+
```
65+
66+
A backend on other ports needs `ZBUGS_CACHE_URL` and `ZBUGS_API_URL` for
67+
`record.ts` too; the web app is built against `ZBUGS_CACHE_URL`.
68+
69+
and record:
70+
71+
```sh
72+
ZBUGS_TOKEN=<token> node apps/zbugs-video/record.ts
73+
```
74+
75+
That builds the apps, the web app into `apps/zbugs/dist` (`build`), adds 20,000 issues from zbugs' synthetic
76+
data generator to the zero project, once (`seed`, which needs `ZBUGS_DB_URL`
77+
when the database isn't the backend's default `zbugs_swift` on
78+
127.0.0.1:5432), launches the apps and tiles them (`launch`), and records
79+
(`record`); each runs alone too. Chrome runs with a profile of its own in
80+
`~/.zbugs-video/chrome`, logged in through the DevTools protocol. The video, its raw take, and the apps' local
81+
stores are in `~/.zbugs-video` (`ZBUGS_VIDEO_DIR`). Leave the mouse and
82+
keyboard alone while it records. `record.ts quit` quits the apps it
83+
launched.
84+
85+
A take that fails stops at the step that did and says which element it
86+
couldn't find; `ax dump <pid>` (in `~/.zbugs-video`) prints what an app
87+
exposes, and `ax text <pid>` the text on its screen.
88+
89+
## A real iPhone
90+
91+
With `ZBUGS_VIDEO_DEVICE` set to an iPhone's identifier (`xcrun devicectl
92+
list devices`), the iPhone in the video is that one, shown by iPhone
93+
Mirroring, in place of the simulator. It needs:
94+
95+
- zbugs-swift installed on the iPhone, built from Xcode with your team.
96+
- The backend reachable from it: start the backend with `ZBUGS_HOST=0.0.0.0`.
97+
The app reaches it at the Mac's address on en0, or at
98+
`ZBUGS_VIDEO_DEVICE_HOST`.
99+
- iPhone Mirroring unlocked on the Mac, with the iPhone locked.
100+
101+
`launch` starts the app logged in through `devicectl`, and places the
102+
Mirroring window, which keeps its own size, in the phone's cell. Mirroring
103+
exposes none of the app's controls, so the phone's steps find what they tap
104+
in the text on its screen, read by macOS's text recognition. Back and
105+
reopen are icons, tapped at their place on the screen. Each phone step is
106+
checked where its change lands, in Swift on the Mac or on the phone's
107+
screen, before the take goes on.

0 commit comments

Comments
 (0)