[cboettig]

Two gaps a benchmark run fell into on 2026-10-02/04:

1. Logs lost to the Job TTL. A matrix Job's stdout is the only copy of its transcripts, and ttlSecondsAfterFinished: 86400 deleted a finished 688-cell run (11 Jobs) before anyone was back to collect it.

• New headless/k8s/matrix-archiver.yaml: a CronJob every 15 min copies each finished app=headless-matrix Job's log to s3://logs-open-llm-proxy/benchmark/raw/<job>.log (idempotent; never rewrites).
• The bucket credential (aws secret — read/write/delete on every NRP bucket) lives only in the archiver pod, under a ServiceAccount restricted to get/list jobs, pods, pods/log. The matrix pod — which clones arbitrary app repos and runs LLM-driven code — never gets it.
• matrix-job.yaml TTL 1d → 7d as a backstop.

2. No record of what actually served. Transcripts recorded only the route id. NRP aliases float (deepseek-v4-flash now serves deepseek-ai/DeepSeek-V4-Flash-Vision-Exp, glm-5 serves Inferact/GLM-5.3-NVFP4) and OpenRouter picks an upstream host per call (provider: "Io Net"). run.js now keeps served_model / served_provider on every llm_fetches entry and rolls them up as transcript.served_models: [{model, provider, calls}].

node --check headless/run.js passes; manifest validated with kubectl apply --dry-run=server. Will verify end-to-end with a smoke matrix before the next full run.

[cboettig]

Verified end-to-end on the cluster: smoke matrix hmx-tpl-archiver-smoke-20261004-183859 (PROXY_BRANCH=this branch, models qwen3 + xiaomi/mimo-v2.6-flash) completed; archive-now from the CronJob archived it (matrix-archiver: 1 archived, 0 failed, 39515 bytes at benchmark/raw/). Transcripts carry served_models: qwen3 → Qwen/Qwen3.8-Flash-Next-FP8 (provider null, NRP); xiaomi/mimo-v2.6-flash → xiaomi/mimo-v2.6-flash via InferenceNet. Follow-up commit sets requests == limits (NRP admission warns above a 1.2 ratio). CronJob is applied in biodiversity.