Skip to content

Commit f1c824b

Browse files
committed
ci: delta-pipeline dry-run workflow (build v1 + v2 + verify patch)
New GitHub Actions workflow that exercises the full delta-update producer pipeline end-to-end on a clean runner: 1. Free runner disk (jlumbroso/free-disk-space@main — strips android/dotnet/haskell/tool-cache/large-packages to free ~30 GB; two seforim.db builds need ~22 GB) 2. Install JBR 25 3. Build v1 from scratch (allocator starts fresh) + verify schema_meta.db_version is stamped to 1 4. Stash v1 (seforim.db.v1, buildstate.v1, catalog.pb.v1); drop the v1 Lucene index since we don't need it for the patch 5. Seed allocator from v1's buildstate 6. Build v2 + verify db_version=2 7. Run producePatchAndVerify (strict prev + patch == new invariant). Failure aborts the workflow. 8. Print patch summary (manifest, blobs, per-table upserts/deletes) 9. Upload patch + buildstate snapshots as workflow artefacts (retention 7 days) Triggered via workflow_dispatch only. Nothing is pushed to a GitHub Release — this is purely a validation harness. Resource notes: - timeout-minutes: 240 (two ~25 min builds + verify) - GRADLE_OPTS=-Xmx2g (daemon lean; heavy JavaExec sub-procs use their own -Xmx10g already) - --no-daemon so a leaked daemon doesn't outlive the job
1 parent fb605d4 commit f1c824b

1 file changed

Lines changed: 174 additions & 0 deletions

File tree

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
name: Delta pipeline — dry run
2+
3+
# Runs the full delta-update pipeline end-to-end WITHOUT publishing.
4+
# Useful to validate that:
5+
# - generateSeforimDb works on the CI runner
6+
# - stampSchemaVersion is correctly chained (schema_meta.db_version)
7+
# - the v1 buildstate properly seeds v2's IdAllocator (id stability)
8+
# - producePatchAndVerify's strict invariant holds (prev + patch == new)
9+
#
10+
# All artefacts are uploaded as workflow artefacts for inspection.
11+
# Nothing is pushed to a GitHub Release.
12+
13+
on:
14+
workflow_dispatch:
15+
inputs:
16+
skip_lucene:
17+
description: "Skip Lucene index build (faster dry run, but Lucene side not exercised)"
18+
required: false
19+
default: "false"
20+
21+
permissions:
22+
contents: read
23+
24+
jobs:
25+
dryrun:
26+
name: Build v1 + v2 + verify delta
27+
runs-on: ubuntu-latest
28+
timeout-minutes: 240
29+
30+
steps:
31+
- name: Checkout SeforimLibrary
32+
uses: actions/checkout@v4
33+
with:
34+
submodules: recursive
35+
36+
- name: Free disk space on runner
37+
uses: jlumbroso/free-disk-space@main
38+
with:
39+
tool-cache: true
40+
android: true
41+
dotnet: true
42+
haskell: true
43+
large-packages: true
44+
docker-images: false
45+
swap-storage: true
46+
47+
- name: Disk after cleanup
48+
run: df -h
49+
50+
- name: Install JBR 25
51+
run: |
52+
curl -L -o jbr.tar.gz "https://cache-redirector.jetbrains.com/intellij-jbr/jbrsdk-25.0.1-linux-x64-b268.52.tar.gz"
53+
mkdir -p "$RUNNER_TEMP/jbr"
54+
tar -xzf jbr.tar.gz -C "$RUNNER_TEMP/jbr"
55+
JBR_DIR=$(find "$RUNNER_TEMP/jbr" -mindepth 1 -maxdepth 1 -type d -name "jbr*" -o -name "jbrsdk*" | head -n 1)
56+
echo "JAVA_HOME=$JBR_DIR" >> "$GITHUB_ENV"
57+
echo "$JBR_DIR/bin" >> "$GITHUB_PATH"
58+
59+
- name: Setup Gradle
60+
uses: gradle/actions/setup-gradle@v4
61+
62+
- name: Grant execute permission for gradlew
63+
run: chmod +x gradlew
64+
65+
- name: Gradle version
66+
run: ./gradlew --version
67+
68+
# ─── BUILD v1 (no seed buildstate) ──────────────────────────────────────
69+
- name: Build seforim.db v1 (no previous buildstate → IdAllocator starts fresh)
70+
run: |
71+
./gradlew generateSeforimDb -PdbVersion=1 --no-daemon --stacktrace
72+
env:
73+
# The heavy work runs in JavaExec sub-processes with their own
74+
# -Xmx (10g on Otzaria/Lucene tasks). Keep the Gradle daemon
75+
# itself lean so the two don't compete for the runner's 16 GB.
76+
GRADLE_OPTS: -Xmx2g
77+
78+
- name: Verify v1 schema_meta.db_version
79+
run: |
80+
sudo apt-get install -y --no-install-recommends sqlite3
81+
VERSION=$(sqlite3 build/seforim.db "SELECT value FROM schema_meta WHERE key='db_version'")
82+
echo "v1 db_version = $VERSION"
83+
test "$VERSION" = "1" || { echo "❌ stampSchemaVersion didn't write db_version=1"; exit 1; }
84+
85+
- name: Stash v1 artefacts and free disk
86+
run: |
87+
# Use `mv` (rename, no copy) to save disk for the 7+ GB DB.
88+
mv build/seforim.db build/seforim.db.v1
89+
mv build/seforim.db.buildstate build/seforim.db.buildstate.v1
90+
mv build/catalog.pb build/catalog.pb.v1
91+
# Drop the v1 Lucene indexes (~3 GB) — they're not needed for the
92+
# patch dry-run; the client re-derives them on apply. v2 will
93+
# generate its own.
94+
rm -rf build/seforim.db.lucene build/seforim.db.lookup.lucene
95+
# Drop lexical.db (~54 MB, will be re-downloaded if needed).
96+
rm -f build/lexical.db
97+
ls -lah build/seforim.db.v1 build/seforim.db.buildstate.v1 build/catalog.pb.v1
98+
df -h
99+
100+
# ─── BUILD v2 (seeded by v1 buildstate) ─────────────────────────────────
101+
- name: Seed allocator with v1 buildstate for the v2 build
102+
run: |
103+
cp build/seforim.db.buildstate.v1 build/seforim.db.buildstate
104+
105+
- name: Build seforim.db v2 (seeded — IDs must stay stable wrt v1)
106+
run: |
107+
./gradlew generateSeforimDb -PdbVersion=2 -PtoVersion=2 --no-daemon --stacktrace
108+
env:
109+
# The heavy work runs in JavaExec sub-processes with their own
110+
# -Xmx (10g on Otzaria/Lucene tasks). Keep the Gradle daemon
111+
# itself lean so the two don't compete for the runner's 16 GB.
112+
GRADLE_OPTS: -Xmx2g
113+
114+
- name: Verify v2 schema_meta.db_version
115+
run: |
116+
VERSION=$(sqlite3 build/seforim.db "SELECT value FROM schema_meta WHERE key='db_version'")
117+
echo "v2 db_version = $VERSION"
118+
test "$VERSION" = "2" || { echo "❌ stampSchemaVersion didn't write db_version=2"; exit 1; }
119+
120+
# ─── PRODUCE + VERIFY THE PATCH ─────────────────────────────────────────
121+
- name: producePatchAndVerify (strict prev + patch == new invariant)
122+
run: |
123+
./gradlew :generator-common:producePatchAndVerify \
124+
-PprevDb=$PWD/build/seforim.db.v1 \
125+
-PnewDb=$PWD/build/seforim.db \
126+
-Pout=$PWD/build/patch-v1-v2.db \
127+
-PcatalogPb=$PWD/build/catalog.pb \
128+
-PfromVersion=1 -PtoVersion=2 \
129+
--no-daemon --stacktrace
130+
env:
131+
# The heavy work runs in JavaExec sub-processes with their own
132+
# -Xmx (10g on Otzaria/Lucene tasks). Keep the Gradle daemon
133+
# itself lean so the two don't compete for the runner's 16 GB.
134+
GRADLE_OPTS: -Xmx2g
135+
136+
# ─── PATCH SIZE + MANIFEST SUMMARY ──────────────────────────────────────
137+
- name: Summarise patch
138+
run: |
139+
echo "=== Patch artefacts ==="
140+
ls -lah build/patch-v1-v2.db build/patch-v1-v2.db.manifest.json
141+
echo ""
142+
echo "=== Manifest ==="
143+
cat build/patch-v1-v2.db.manifest.json
144+
echo ""
145+
echo "=== Patch.blobs ==="
146+
sqlite3 build/patch-v1-v2.db "SELECT name, length(content) FROM blobs"
147+
echo ""
148+
echo "=== Patch upserts/deletes by table (top 5 each) ==="
149+
for t in author topic tocText category book line line_toc link tocEntry; do
150+
UC=$(sqlite3 build/patch-v1-v2.db "SELECT COUNT(*) FROM upsert_${t}" 2>/dev/null || echo 0)
151+
DC=$(sqlite3 build/patch-v1-v2.db "SELECT COUNT(*) FROM delete_${t}" 2>/dev/null || echo 0)
152+
echo " $t: +$UC / -$DC"
153+
done
154+
155+
# ─── ARTEFACTS ──────────────────────────────────────────────────────────
156+
- name: Upload patch + manifests
157+
if: always()
158+
uses: actions/upload-artifact@v4
159+
with:
160+
name: delta-dryrun-patch
161+
path: |
162+
build/patch-v1-v2.db
163+
build/patch-v1-v2.db.manifest.json
164+
retention-days: 7
165+
166+
- name: Upload buildstate snapshots
167+
if: always()
168+
uses: actions/upload-artifact@v4
169+
with:
170+
name: delta-dryrun-buildstates
171+
path: |
172+
build/seforim.db.buildstate.v1
173+
build/seforim.db.buildstate
174+
retention-days: 7

0 commit comments

Comments
 (0)