Skip to content

Commit 70350ed

Browse files
author
tazhate
committed
fix(ci): runner disk pressure killed batch e2e mid-run
Run 25172351815 finished with conclusion=failure after 2h2min, no "Run batch test" log in the archive and Upload artifacts never ran — classic runner OOM/disk-full where the agent dies before flushing artifacts. ubuntu-latest ships with ~14GB free on /, and the test was provisioning 5x5Gi PVCs per batch via local-path-provisioner; even with PVC delete between batches, transient writes from chains that manage to start (bitcoin pulling block headers, anything that lazily hydrates) compound across batches faster than reclaim can keep up. Two fixes: 1. Drop default STORAGE_SIZE from 5Gi to 1Gi. We're only checking that nodes start, not that they reach any meaningful height — the PVC is sized for the test, not for actual sync. 2. Free ~25GB on the runner before kind comes up by removing the bundled .NET SDK, Android NDK, GHC, and CodeQL toolcache (standard actions/runner free-disk recipe), and prune dangling docker images. Also dump `df -h` between batches so disk usage is visible in the live log. Context: confirmed the symptom by pulling logs zip from the run — the batch step's log file was missing entirely and the job conclusion went straight from completing "Apply ChainVersionCatalog samples" to a job failure with no per-step error, which only happens when the runner itself is killed. ~10min.
1 parent 52d6145 commit 70350ed

2 files changed

Lines changed: 9 additions & 1 deletion

File tree

.github/workflows/manual-batch-e2e.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,13 @@ jobs:
2020
steps:
2121
- uses: actions/checkout@v4
2222

23+
- name: Free disk space
24+
run: |
25+
df -h /
26+
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL || true
27+
sudo docker image prune -af || true
28+
df -h /
29+
2330
- uses: actions/setup-go@v5
2431
with:
2532
go-version-file: go.mod

scripts/manual-batch-test.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ set -uo pipefail
2020
BATCH_SIZE="${BATCH_SIZE:-5}"
2121
READY_TIMEOUT="${READY_TIMEOUT:-600}"
2222
STORAGE_CLASS="${STORAGE_CLASS:-standard}"
23-
STORAGE_SIZE="${STORAGE_SIZE:-5Gi}"
23+
STORAGE_SIZE="${STORAGE_SIZE:-1Gi}"
2424
NAMESPACE="${NAMESPACE:-chainplane-batch}"
2525
CHAINS_FILTER="${CHAINS_FILTER:-}"
2626
ARTIFACTS_DIR="${ARTIFACTS_DIR:-artifacts}"
@@ -281,6 +281,7 @@ while [[ ${i} -lt ${TOTAL} ]]; do
281281
[[ ${end} -gt ${TOTAL} ]] && end=${TOTAL}
282282
echo
283283
echo "=== batch $((i / BATCH_SIZE + 1)) [$((i+1))-${end}/${TOTAL}] ==="
284+
df -h / 2>&1 | tail -1 | awk '{printf " disk: %s free, %s used (%s)\n", $4, $3, $5}'
284285
run_batch "${ALL_SAMPLES[@]:i:$((end - i))}"
285286
i=${end}
286287
done

0 commit comments

Comments
 (0)