Skip to content

Commit f875f4b

Browse files
tmmampagent
tmm
and
ampagent
authored
test(zero-swift): nightly fuzzing: ZeroVirtual, and Swift against JS on new seeds (#110)
* test(zero-swift): fuzz ZeroVirtual on every PR and nightly The fuzz harness that found the #94 bug was a scratch file, rebuilt from thread history three times. It's now FuzzTests: 200 seeds on every PR, 10,000 new ones each night in swift-nightly.yml, and seed 760 kept as a regression. A failure names its seed and how to replay it with a trace. Inserted ids are now unique: the scratch harness could insert the same id twice, which failed seeds 1800 and 2403 without a ZeroVirtual bug. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev> * test(zero-swift): compare Swift with JS on new seeds each night The nightly workflow now regenerates the seeded IVM, sweep, hardening and aggregate client corpora on seeds from its run number, at 5 to 10 times the PR checks' cases, and runs the Swift conformance suites on them. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev> * test(zero-swift): yield the main actor between ZeroVirtual fuzz seeds The 200 seeds ran in one synchronous loop on the main actor, 37 s on a CI runner, and the VirtualList tests that started beside them timed out waiting for it. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev> * test(zero-swift): run 50 ZeroVirtual fuzz seeds on PRs 200 seeds took 80 s of main-actor time on a CI Mac, beside tests with 15 s windows. The nightly workflow still runs 10,000. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev> * test(zero-swift): run the ZeroVirtual fuzz in a CI step of its own Beside the parallel suites, its main-actor work starved them: a NIO connect to a loopback server took 10 s to send its first bytes, and clientReconnectsAfterTheServerGoesAwayMidFrame timed out on Linux and macOS. The random seeds now run only when ZERO_FUZZ_SEEDS or ZERO_FUZZ_SEED is set, and CI runs 200 of them after the other tests. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev> --------- Co-authored-by: Amp <amp@ampcode.com>
1 parent 958734a commit f875f4b

5 files changed

Lines changed: 395 additions & 1 deletion

File tree

‎.github/workflows/swift-macos.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ jobs:
4242
- run: swift --version
4343
- name: Build and test (macOS)
4444
run: swift test
45+
# On its own: see swift.yml.
46+
- name: ZeroVirtual fuzz
47+
run: ZERO_FUZZ_SEEDS=200 swift test --skip-build --filter FuzzTests
4548
- name: Docs
4649
run: scripts/docs.sh
4750

‎.github/workflows/swift-nightly.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
name: Swift Nightly
2+
3+
# Long fuzz runs of the Swift client, on seeds the PR checks don't reach. Each
4+
# run takes new seeds from its run number, and a failure names the seed to
5+
# replay: JS's answers to new random queries, pushes and client scenarios,
6+
# compared with Swift's; and the ZeroVirtual fuzz.
7+
8+
on:
9+
schedule:
10+
# 3 AM US Eastern.
11+
- cron: '0 7 * * *'
12+
workflow_dispatch:
13+
inputs:
14+
seeds:
15+
description: How many seeds to run
16+
default: '10000'
17+
first-seed:
18+
description: The seed (the run number when empty); ZeroVirtual's first seed
19+
default: ''
20+
21+
permissions:
22+
contents: read
23+
24+
jobs:
25+
# JS's answers on new seeds, 5 to 10 times the PR checks' cases. The seeds
26+
# are printed, so a mismatch replays with the same environment variables.
27+
differential-vectors:
28+
name: Differential vectors (JS)
29+
runs-on: ubuntu-latest
30+
timeout-minutes: 30
31+
steps:
32+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
33+
with:
34+
persist-credentials: false
35+
- uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9
36+
with:
37+
version: 11.11.0
38+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
39+
with:
40+
node-version-file: .nvmrc
41+
package-manager-cache: false
42+
- run: pnpm install --frozen-lockfile
43+
- name: Generate
44+
working-directory: packages/zero-conformance
45+
env:
46+
SEED: ${{ inputs.first-seed || format('{0}', github.run_number) }}
47+
run: |
48+
base=$((1000000 + SEED))
49+
export IVM_FUZZ_SEED=$base IVM_FUZZ_CASES=6000
50+
export SWEEP_IVM_SEED=$base SWEEP_IVM_CASES=15000
51+
export HARDEN_IVM_SEED=$((base * 10)) HARDEN_IVM_SEEDS=5
52+
export AGG_FUZZ_SEED=$base AGG_FUZZ_CASES=750
53+
env | grep -E '_(SEED|SEEDS|CASES)=' | sort
54+
pnpm exec vitest run src/ivm/ivm.test.ts src/ivm/ivm-features.test.ts \
55+
src/ivm/sweep-ivm.test.ts src/ivm/harden-ivm.test.ts src/client/bughunt.test.ts
56+
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
57+
with:
58+
name: differential-vectors
59+
path: packages/zero-conformance/generated/
60+
if-no-files-found: error
61+
retention-days: 7
62+
63+
differential-swift:
64+
name: Differential vectors (Swift)
65+
needs: differential-vectors
66+
runs-on: ubuntu-latest
67+
container: swift:6.2.4-bookworm
68+
timeout-minutes: 60
69+
defaults:
70+
run:
71+
working-directory: swift
72+
steps:
73+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
74+
with:
75+
persist-credentials: false
76+
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
77+
with:
78+
name: differential-vectors
79+
path: packages/zero-conformance/generated/
80+
- name: Install system libraries
81+
run: apt-get update && apt-get install -y --no-install-recommends libsqlite3-dev
82+
- name: Build
83+
run: swift build --build-tests
84+
- name: Compare with JS
85+
run: swift test --skip-build --filter 'HardenIVMTests|SweepIVMTests|IVMConformanceTests|BugHuntTests'
86+
87+
virtual-fuzz:
88+
name: ZeroVirtual fuzz
89+
runs-on: ubuntu-latest
90+
container: swift:6.2.4-bookworm
91+
timeout-minutes: 60
92+
defaults:
93+
run:
94+
working-directory: swift
95+
steps:
96+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
97+
with:
98+
persist-credentials: false
99+
- name: Install system libraries
100+
run: apt-get update && apt-get install -y --no-install-recommends libsqlite3-dev
101+
- name: Build
102+
run: swift build --build-tests
103+
# Past the PR checks' seeds (0 to 199), a new range for each run.
104+
- name: Fuzz
105+
env:
106+
ZERO_FUZZ_SEEDS: ${{ inputs.seeds || '10000' }}
107+
ZERO_FUZZ_FIRST_SEED: ${{ inputs.first-seed || format('{0}0000', github.run_number) }}
108+
run: |
109+
echo "Seeds $ZERO_FUZZ_FIRST_SEED to $((ZERO_FUZZ_FIRST_SEED + ZERO_FUZZ_SEEDS - 1))"
110+
swift test --skip-build --filter FuzzTests

‎.github/workflows/swift.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,10 @@ jobs:
9797
run: swift build --build-tests
9898
- name: Test
9999
run: swift test --skip-build
100+
# On its own: it holds the main actor long enough to starve the
101+
# timing-sensitive suites if it ran beside them.
102+
- name: ZeroVirtual fuzz
103+
run: ZERO_FUZZ_SEEDS=200 swift test --skip-build --filter FuzzTests
100104
# Broken symbol links and undocumented parameters fail. ZeroUI is
101105
# Apple-only; swift-macos.yml checks it.
102106
- name: Docs

‎swift/DESIGN.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,8 @@ Vector storage and fuzzing:
264264

265265
- Curated vectors are checked in. CI regenerates them from JS and fails if they are stale.
266266
- Seeded random query and push fuzz cases are generated in CI from fixed seeds (not checked in) and run against Swift.
267-
- A live differential harness (JS drives a Swift CLI over stdio with random seeds and shrinks failures) is deferred to v1.x.
267+
- `swift-nightly.yml` generates the seeded corpora again each night on new seeds, from the run number, at 5 to 10 times the PR checks' cases, and runs the Swift suites on them. A mismatch names its case and the seeds the job printed; replaying it is setting the same environment variables. A live differential harness (JS drives a Swift CLI over stdio and shrinks failures) is deferred to v1.x.
268+
- ZeroVirtual has no JS oracle for scrolling, so `FuzzTests` checks invariants instead: after each random step settles, a row is on screen, the loaded rows are a run of the list, and rows on screen hold still through changes above them. PRs run 200 seeds, in a CI step of their own; `swift-nightly.yml` runs 10,000 new ones each night.
268269

269270
### JS bugs found, not yet reported upstream
270271

0 commit comments

Comments
 (0)