Commit 70350ed
tazhate
fix(ci): runner disk pressure killed batch e2e mid-run
Run 25172351815 finished with conclusion=failure after 2h2min, no
"Run batch test" log in the archive and Upload artifacts never ran —
classic runner OOM/disk-full where the agent dies before flushing
artifacts. ubuntu-latest ships with ~14GB free on /, and the test was
provisioning 5x5Gi PVCs per batch via local-path-provisioner; even
with PVC delete between batches, transient writes from chains that
manage to start (bitcoin pulling block headers, anything that lazily
hydrates) compound across batches faster than reclaim can keep up.
Two fixes:
1. Drop default STORAGE_SIZE from 5Gi to 1Gi. We're only checking
that nodes start, not that they reach any meaningful height — the
PVC is sized for the test, not for actual sync.
2. Free ~25GB on the runner before kind comes up by removing the
bundled .NET SDK, Android NDK, GHC, and CodeQL toolcache (standard
actions/runner free-disk recipe), and prune dangling docker
images. Also dump `df -h` between batches so disk usage is visible
in the live log.
Context: confirmed the symptom by pulling logs zip from the run — the
batch step's log file was missing entirely and the job conclusion went
straight from completing "Apply ChainVersionCatalog samples" to a job
failure with no per-step error, which only happens when the runner
itself is killed. ~10min.1 parent 52d6145 commit 70350ed
2 files changed
Lines changed: 9 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| |||
281 | 281 | | |
282 | 282 | | |
283 | 283 | | |
| 284 | + | |
284 | 285 | | |
285 | 286 | | |
286 | 287 | | |
| |||
0 commit comments