Repository navigation
feat: let matrix Jobs set PER_RUN_TIMEOUT_SEC - #143
Merged
Merged
Conversation
The per-cell timeout was hardcoded at its 900s default for any run launched through run-matrix-k8s.sh: PER_RUN_TIMEOUT_SEC was missing from both the export list and the envsubst allowlist, so a caller setting it got 900 anyway, with no error — the only symptom was 'run-timeout=900s' in the Job log. Thread it through the export list, the envsubst allowlist and the Job template's env block, keeping the 900s default. This is the per-cell cap (a whole agent session, many LLM calls). The per-call ceilings -- UPSTREAM_TIMEOUT_SECONDS, the ingress timeouts, --llm-timeout, all 1200s -- are a separate chain, unaffected by this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PER_RUN_TIMEOUT_SEChas existed inheadless/run_matrix.sh(default 900) for awhile, but
run-matrix-k8s.shnever passed it into the Job: it was missing fromboth the
exportlist and theenvsubstallowlist. A caller doingsilently got 900s. There is no error — the only symptom is
run-timeout=900s/960sin the Job log, which is easy to miss until several hours of cells have been
clipped.
Fix
Thread it through the export list, the
envsubstallowlist, and the Jobtemplate's env block. The 900s default is unchanged, so this is a no-op for every
existing caller.
Scope note — two different timeout chains
This is the per-cell cap. One cell is a whole agent session (many LLM calls),
not a single call. The per-call ceilings —
UPSTREAM_TIMEOUT_SECONDS, theingress.yamltimeouts,--llm-timeout, all 1200s — are a separate chain and areuntouched here, so raising this does not require moving them in step (#136 applies
to that chain, not this knob).
Motivation
A full-tier
qwen(DSE-nimbus) benchmark: 9 of 45 cells hit exactly 900s, whilecompleted cells had a median of 408s, p90 684s, and a max of 891s — three finished
within 9s of the cap. The distribution was being clipped, not converging.
Per-call latency over the same run peaked at 405s against the 1200s budget, so the
per-call chain has ample headroom and only the cell cap needed raising.
Test
bash -n headless/run-matrix-k8s.shpasses.envsubstwithPER_RUN_TIMEOUT_SEC=1800andconfirmed both
value: "1800"in the env block andPER_RUN_TIMEOUT_SEC="1800"on therun_matrix.shinvocation.run_matrix.shalready honors the variable, so no change is neededon that side.