Skip to content

Commit 7b8a55b

Browse files
author
tazhate
committed
feat(ci): manual batch e2e workflow for all 103 chain samples
Adds a workflow_dispatch flow that spins up a fresh kind cluster, installs chainplane via Helm with cert-manager-issued webhook certs, then walks config/samples/chains_v1alpha2_chaininstance_*.yaml in batches of 5, applying each batch and waiting until every ChainInstance reports .status.height > 0 (so we exercise pod start AND that sync actually begins). Each batch is deleted before the next so kind doesn't run out of resources, failed chains get describe + pod logs + events captured into artifacts/, and a passed/failed table is printed to the job summary. Storage is force-overridden to 5Gi against the kind default StorageClass — full sync isn't the goal, only that nodes start. Inputs: batch_size, ready_timeout, chains_filter (regex on filename) so smoke runs (e.g. CHAINS_FILTER=bitcoin|litecoin) finish in ~10min before committing to the full ~3h sweep. Context: spent ~45min sketching this — first verified kind was fine for parallel pod load (config bumps max-pods to 200 per node), confirmed cert-manager v1.16.3 install matches what test/e2e/utils already uses, sanity-checked the sample patcher with yq against bitcoin.yaml, and shellcheck'd manual-batch-test.sh. Triggered manually via gh workflow run when you want a sweep.
1 parent 1214dab commit 7b8a55b

3 files changed

Lines changed: 304 additions & 0 deletions

File tree

‎.github/workflows/manual-batch-e2e.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: Manual batch e2e
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
batch_size:
7+
description: "Number of ChainInstances applied per batch"
8+
default: "5"
9+
ready_timeout:
10+
description: "Per-batch timeout (seconds) waiting for height>0"
11+
default: "600"
12+
chains_filter:
13+
description: "Regex applied to sample filenames (empty = all)"
14+
default: ""
15+
16+
jobs:
17+
batch:
18+
runs-on: ubuntu-latest
19+
timeout-minutes: 350
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- uses: actions/setup-go@v5
24+
with:
25+
go-version-file: go.mod
26+
27+
- name: Install kind
28+
run: |
29+
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
30+
chmod +x ./kind
31+
sudo mv ./kind /usr/local/bin/kind
32+
kind version
33+
34+
- name: Install yq
35+
run: |
36+
sudo curl -Lo /usr/local/bin/yq \
37+
https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
38+
sudo chmod +x /usr/local/bin/yq
39+
yq --version
40+
41+
- name: Install helm
42+
uses: azure/setup-helm@v4
43+
44+
- name: Create kind cluster
45+
run: kind create cluster --config scripts/kind-batch.yaml --wait 120s
46+
47+
- name: Install cert-manager
48+
run: |
49+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml
50+
kubectl wait --for=condition=Available -n cert-manager deploy --all --timeout=180s
51+
52+
- name: Build operator image
53+
run: |
54+
docker build -t chainplane:e2e .
55+
kind load docker-image chainplane:e2e
56+
57+
- name: Install chart
58+
run: |
59+
helm install chainplane ./charts/chainplane \
60+
-n chainplane-system --create-namespace \
61+
--set image.repository=chainplane \
62+
--set image.tag=e2e \
63+
--set image.pullPolicy=Never \
64+
--set webhook.certManager.enabled=true \
65+
--set replicaCount=1 \
66+
--set leaderElection.enabled=false
67+
kubectl -n chainplane-system rollout status deploy/chainplane --timeout=180s
68+
69+
- name: Apply ChainVersionCatalog samples
70+
run: kubectl apply -f config/samples/chains_v1alpha2_chainversioncatalog.yaml
71+
72+
- name: Run batch test
73+
env:
74+
STORAGE_CLASS: standard
75+
BATCH_SIZE: ${{ inputs.batch_size }}
76+
READY_TIMEOUT: ${{ inputs.ready_timeout }}
77+
CHAINS_FILTER: ${{ inputs.chains_filter }}
78+
run: bash scripts/manual-batch-test.sh
79+
80+
- name: Collect controller logs on failure
81+
if: failure()
82+
run: |
83+
mkdir -p artifacts
84+
kubectl -n chainplane-system logs -l control-plane=controller-manager --tail=2000 > artifacts/controller.log 2>&1 || true
85+
kubectl -n chainplane-batch get events --sort-by=.lastTimestamp > artifacts/events.log 2>&1 || true
86+
kubectl get chaininstances -A -o yaml > artifacts/chaininstances.yaml 2>&1 || true
87+
88+
- name: Upload artifacts
89+
if: always()
90+
uses: actions/upload-artifact@v4
91+
with:
92+
name: batch-e2e-artifacts
93+
path: artifacts/

‎scripts/kind-batch.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
nodes:
4+
- role: control-plane
5+
kubeadmConfigPatches:
6+
- |
7+
kind: InitConfiguration
8+
nodeRegistration:
9+
kubeletExtraArgs:
10+
max-pods: "200"
11+
- role: worker
12+
kubeadmConfigPatches:
13+
- |
14+
kind: JoinConfiguration
15+
nodeRegistration:
16+
kubeletExtraArgs:
17+
max-pods: "200"
18+
- role: worker
19+
kubeadmConfigPatches:
20+
- |
21+
kind: JoinConfiguration
22+
nodeRegistration:
23+
kubeletExtraArgs:
24+
max-pods: "200"

‎scripts/manual-batch-test.sh‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
#!/usr/bin/env bash
2+
# Manual batch e2e: applies all ChainInstance samples in batches, verifies
3+
# pod start + .status.height > 0, then deletes the batch before the next.
4+
#
5+
# Env:
6+
# BATCH_SIZE (default 5)
7+
# READY_TIMEOUT (default 600 sec per batch)
8+
# STORAGE_CLASS (default "standard" — kind default)
9+
# STORAGE_SIZE (default 5Gi)
10+
# NAMESPACE (default chainplane-batch)
11+
# CHAINS_FILTER (regex, empty = all)
12+
# ARTIFACTS_DIR (default ./artifacts)
13+
# SAMPLES_DIR (default config/samples)
14+
15+
set -uo pipefail
16+
17+
BATCH_SIZE="${BATCH_SIZE:-5}"
18+
READY_TIMEOUT="${READY_TIMEOUT:-600}"
19+
STORAGE_CLASS="${STORAGE_CLASS:-standard}"
20+
STORAGE_SIZE="${STORAGE_SIZE:-5Gi}"
21+
NAMESPACE="${NAMESPACE:-chainplane-batch}"
22+
CHAINS_FILTER="${CHAINS_FILTER:-}"
23+
ARTIFACTS_DIR="${ARTIFACTS_DIR:-artifacts}"
24+
SAMPLES_DIR="${SAMPLES_DIR:-config/samples}"
25+
26+
mkdir -p "${ARTIFACTS_DIR}"
27+
28+
for bin in kubectl yq jq; do
29+
command -v "${bin}" >/dev/null 2>&1 || { echo "ERROR: ${bin} not in PATH"; exit 2; }
30+
done
31+
32+
kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -
33+
34+
# Build sample list (skip the placeholder aggregate file).
35+
mapfile -t ALL_SAMPLES < <(
36+
find "${SAMPLES_DIR}" -maxdepth 1 -name 'chains_v1alpha2_chaininstance_*.yaml' \
37+
! -name 'chains_v1alpha2_chaininstance.yaml' | sort
38+
)
39+
40+
if [[ -n "${CHAINS_FILTER}" ]]; then
41+
FILTERED=()
42+
for f in "${ALL_SAMPLES[@]}"; do
43+
[[ "$(basename "$f")" =~ ${CHAINS_FILTER} ]] && FILTERED+=("$f")
44+
done
45+
ALL_SAMPLES=("${FILTERED[@]}")
46+
fi
47+
48+
TOTAL=${#ALL_SAMPLES[@]}
49+
if [[ ${TOTAL} -eq 0 ]]; then
50+
echo "No samples matched filter '${CHAINS_FILTER}'"
51+
exit 1
52+
fi
53+
54+
echo "Discovered ${TOTAL} sample(s); batch size=${BATCH_SIZE}; per-batch timeout=${READY_TIMEOUT}s"
55+
56+
declare -a RESULTS_NAME RESULTS_STATUS RESULTS_REASON
57+
58+
patch_sample() {
59+
local in="$1" out="$2"
60+
yq eval "
61+
.metadata.namespace = \"${NAMESPACE}\" |
62+
.spec.storage.size = \"${STORAGE_SIZE}\" |
63+
.spec.storage.storageClass = \"${STORAGE_CLASS}\" |
64+
del(.spec.resources.limits) |
65+
.spec.resources.requests.cpu = \"100m\" |
66+
.spec.resources.requests.memory = \"256Mi\"
67+
" "${in}" > "${out}"
68+
}
69+
70+
dump_artifacts() {
71+
local name="$1"
72+
local logf="${ARTIFACTS_DIR}/${name}.log"
73+
{
74+
echo "=== describe chaininstance/${name} ==="
75+
kubectl -n "${NAMESPACE}" describe chaininstance "${name}" 2>&1 || true
76+
echo
77+
echo "=== describe sts ==="
78+
kubectl -n "${NAMESPACE}" describe sts -l "chains.chainplane.io/instance=${name}" 2>&1 || true
79+
echo
80+
echo "=== pod logs ==="
81+
for p in $(kubectl -n "${NAMESPACE}" get pods -l "chains.chainplane.io/instance=${name}" -o name 2>/dev/null); do
82+
echo "--- ${p} ---"
83+
kubectl -n "${NAMESPACE}" logs "${p}" --all-containers --tail=200 2>&1 || true
84+
done
85+
echo
86+
echo "=== events ==="
87+
kubectl -n "${NAMESPACE}" get events --sort-by=.lastTimestamp 2>&1 | grep -E "${name}|Warning" || true
88+
} > "${logf}"
89+
}
90+
91+
run_batch() {
92+
local -a files=("$@")
93+
local tmpdir
94+
tmpdir="$(mktemp -d)"
95+
local -a names=()
96+
97+
for f in "${files[@]}"; do
98+
local nm
99+
nm="$(yq eval '.metadata.name' "$f")"
100+
names+=("$nm")
101+
patch_sample "$f" "${tmpdir}/${nm}.yaml"
102+
done
103+
104+
echo "--- applying batch: ${names[*]}"
105+
kubectl apply -f "${tmpdir}/" || true
106+
107+
local deadline=$(( $(date +%s) + READY_TIMEOUT ))
108+
local -A done_map=()
109+
110+
while [[ $(date +%s) -lt ${deadline} ]]; do
111+
local all_done=1
112+
for nm in "${names[@]}"; do
113+
[[ -n "${done_map[$nm]:-}" ]] && continue
114+
local height phase
115+
height="$(kubectl -n "${NAMESPACE}" get chaininstance "${nm}" -o jsonpath='{.status.height}' 2>/dev/null || true)"
116+
phase="$(kubectl -n "${NAMESPACE}" get chaininstance "${nm}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
117+
if [[ -n "${height}" && "${height}" =~ ^[1-9][0-9]*$ ]]; then
118+
done_map[$nm]="PASS:height=${height},phase=${phase}"
119+
echo " PASS ${nm} height=${height} phase=${phase}"
120+
else
121+
all_done=0
122+
fi
123+
done
124+
[[ ${all_done} -eq 1 ]] && break
125+
sleep 10
126+
done
127+
128+
for nm in "${names[@]}"; do
129+
if [[ -n "${done_map[$nm]:-}" ]]; then
130+
RESULTS_NAME+=("${nm}")
131+
RESULTS_STATUS+=("PASS")
132+
RESULTS_REASON+=("${done_map[$nm]#PASS:}")
133+
else
134+
local reason
135+
reason="$(kubectl -n "${NAMESPACE}" get chaininstance "${nm}" -o jsonpath='phase={.status.phase} height={.status.height}' 2>/dev/null || echo 'no-status')"
136+
RESULTS_NAME+=("${nm}")
137+
RESULTS_STATUS+=("FAIL")
138+
RESULTS_REASON+=("timeout: ${reason}")
139+
dump_artifacts "${nm}"
140+
echo " FAIL ${nm} (${reason})"
141+
fi
142+
done
143+
144+
kubectl delete -f "${tmpdir}/" --wait=false 2>&1 | tail -5 || true
145+
# Best-effort PVC cleanup (statefulset PVCs aren't auto-deleted)
146+
kubectl -n "${NAMESPACE}" delete pvc -l app.kubernetes.io/managed-by=chainplane --wait=false 2>/dev/null || true
147+
rm -rf "${tmpdir}"
148+
}
149+
150+
i=0
151+
while [[ ${i} -lt ${TOTAL} ]]; do
152+
end=$(( i + BATCH_SIZE ))
153+
[[ ${end} -gt ${TOTAL} ]] && end=${TOTAL}
154+
echo
155+
echo "=== batch $((i / BATCH_SIZE + 1)) [$((i+1))-${end}/${TOTAL}] ==="
156+
run_batch "${ALL_SAMPLES[@]:i:$((end - i))}"
157+
i=${end}
158+
done
159+
160+
echo
161+
echo "=== SUMMARY ==="
162+
fail=0
163+
{
164+
printf "| %-40s | %-6s | %s\n" "chain" "status" "reason"
165+
printf "|%s|%s|%s\n" "$(printf '%.s-' {1..42})" "$(printf '%.s-' {1..8})" "$(printf '%.s-' {1..40})"
166+
for n in "${!RESULTS_NAME[@]}"; do
167+
printf "| %-40s | %-6s | %s\n" "${RESULTS_NAME[$n]}" "${RESULTS_STATUS[$n]}" "${RESULTS_REASON[$n]}"
168+
[[ "${RESULTS_STATUS[$n]}" == "FAIL" ]] && fail=$((fail + 1))
169+
done
170+
} | tee "${ARTIFACTS_DIR}/summary.txt"
171+
172+
echo
173+
echo "passed=$(( ${#RESULTS_NAME[@]} - fail )) failed=${fail} total=${#RESULTS_NAME[@]}"
174+
175+
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
176+
{
177+
echo "## Manual batch e2e summary"
178+
echo
179+
echo "passed=**$(( ${#RESULTS_NAME[@]} - fail ))** failed=**${fail}** total=**${#RESULTS_NAME[@]}**"
180+
echo
181+
echo '```'
182+
cat "${ARTIFACTS_DIR}/summary.txt"
183+
echo '```'
184+
} >> "${GITHUB_STEP_SUMMARY}"
185+
fi
186+
187+
[[ ${fail} -eq 0 ]] || exit 1

0 commit comments

Comments
 (0)