feat(spurd): survive agent restart without interrupting running jobs by yansun1996 · Pull Request #492 · ROCm/spur

yansun1996 · 2026-07-22T23:37:42Z

Summary

spurd's SIGTERM handler unconditionally deregistered the node from the controller, which force-evicts every running job on that node immediately — so any graceful restart (e.g. a binary upgrade via systemctl restart spurd) failed in-flight jobs, even though the underlying process could survive it (it already runs in its own process group and a cgroup outside spurd's own systemd cgroup).

This PR makes a graceful restart transparent to running jobs, as long as the restart completes within heartbeat_timeout_secs (the existing heartbeat-timeout remains the backstop for a restart that takes too long — the same bound Slurm documents for a slurmd upgrade).

Approach

SIGTERM: deregisters only when the agent has no active jobs (none running and none mid-launch). An idle node still deregisters immediately as before; a busy one stops serving and relies on the controller's heartbeat timeout as the safety net if it doesn't come back.
Manifest: a small on-disk record is written per job right after a successful launch (pid, /proc/<pid>/stat start time to detect PID reuse, cgroup path, exact allocated cpu/gpu ids, uid/gid, output paths, and the exit-status sentinel path). Written via a temp file + rename so a crash mid-write can't leave a torn manifest that startup would silently skip.
Completion detection without waitpid: a restarted agent is no longer the parent of a still-running job, so it can't wait() it (a kernel restriction, not a design choice). Completion is detected via cgroup population (cgroup.events), and the exit code is recovered from a sentinel file the job's own wrapper script writes — with a re-exit so the normal (non-restarted) path's exit-status reporting is unchanged.
Reconciliation on startup: before accepting new launches, the agent scans manifests and either re-adopts a still-alive job (restoring its exact resource allocation, so a restarted agent can't double-book GPUs/CPUs a surviving job still holds) or, if the job already finished while the agent was down, reports its real completion. These completion reports are delivered inline before the gRPC server starts serving, rather than fire-and-forget, so a job that finished during the downtime can't be lost and left to a heartbeat-timeout outcome.
Container jobs report real exit codes across a restart too. A container job runs its script inside the pivoted rootfs, so the same exit-status wrapper plain jobs already get is applied to the container script, writing to an in-container path that is host-visible in the rootfs and recorded in the manifest. A re-adopted container job now reports its true exit code (success or failure) instead of an approximate -1.
Cgroups and per-job spool directories are keyed by (job_id, run_attempt), not job_id alone — otherwise a same-node redispatch while an old run is still finishing could collide on the cgroup (a cgroup-wide kill would take the new run down too) or read a stale exit-status sentinel from the prior run.
The per-job spool directory stays root-owned and non-writable by the job (0o711); the job's exit-status sentinel is pre-created and chowned to it, so the job never needs to create files there and a co-located user can't plant a symlink over the root-authored manifest.
spurctld warns at startup if heartbeat_timeout_secs is below 30s, since it's a more load-bearing safety net now than before.

Known limitations (not solved here)

A restart slower than heartbeat_timeout_secs still falls back to today's eviction path.
A job redispatched to a different node during such a slow restart isn't proactively cleaned up on the original node.
PMI rendezvous state for a job that's mid-launch (not yet running) during a restart is still lost.
The completion-detection fallback for a re-adopted job without a cgroup uses /proc/<pid> existence plus start-time. Start-time has clock-tick granularity, so a PID recycled within the same tick could in principle be mistaken for the original — realistically only across a reboot, which this feature doesn't cover (a rebooted node's jobs are already gone). The cgroup path, used whenever a cgroup is present, is not affected.
An interactive srun allocation (no batch process) is not written to a manifest, so it isn't restored on restart; interactive sessions don't survive a restart in any case (the client connection drops), and the allocation is simply absent rather than double-counted.
A container job's rootfs directory is keyed by job id alone (not run attempt), unlike its cgroup and spool dir, so a same-node redispatch reuses one rootfs. Pre-existing; a follow-up could align the keying.

Structurally, the mid-launch PMI loss and the PID-reuse edge above would be avoided by a persistent per-step supervisor process (as Slurm has with slurmstepd); reconstructing state from a manifest after the fact is the trade-off here. A supervisor is out of scope for this PR.

Testing

Unit tests in executor.rs, agent_server.rs, and cons_tres.rs, several exercising real spawned processes rather than simulating behavior — including: a resumed container job reading its real exit code from the sentinel; a mixed reconcile pass that re-adopts a surviving job and reports only the finished one; run-attempt-scoped spool isolation and cleanup; and atomic manifest write (no leftover temp file).
E2E (test_deregistration.py): restart mid-job and assert the job survives, completes, and its allocation is neither stranded nor double-booked; exit code preserved across a restart for the full matrix of plain and container jobs × success and failure; and a job that finishes entirely while the agent is down still reports its true exit code on restart.
cargo clippy --workspace --exclude spur-ffi --all-targets and cargo fmt clean; targeted cargo test passes.
Verified end-to-end on an isolated, throwaway deployment (non-production ports): submitted a job, restarted the agent mid-run, confirmed it was not evicted, watched it get re-adopted with its allocation correctly restored, and complete and clean up normally afterward.

codecov-commenter · 2026-07-22T23:41:09Z

Codecov Report

❌ Patch coverage is 90.68783% with 88 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #492      +/-   ##
==========================================
+ Coverage   74.25%   74.54%   +0.29%     
==========================================
  Files         165      165              
  Lines       59071    59983     +912     
==========================================
+ Hits        43862    44713     +851     
- Misses      15209    15270      +61

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR updates spurd to support transparent graceful restarts (e.g., systemctl restart spurd) without force-evicting currently running jobs, by persisting per-job launch state to disk and reconciling/resuming those jobs on startup while relying on the existing heartbeat timeout as the safety net for overly slow restarts.

Changes:

Update spurd shutdown/startup flow: avoid SIGTERM deregistration when jobs are running, and reconcile manifests on startup to re-adopt surviving jobs and restore their allocations.
Add manifest + resume plumbing in the executor (exit-status sentinel, cgroup-based completion detection, run-attempt-scoped cgroups) and allocation restoration support in the scheduler.
Add/extend unit and e2e coverage for restart survival, manifest scanning/reconciliation, and restore-allocation behavior; add a controller startup warning for low heartbeat timeouts.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/native_host/e2e/test_deregistration.py	Adds an e2e acceptance test asserting agent restart mid-job does not interrupt the job and does not strand/double-book CPU allocation.
crates/spurd/src/main.rs	Reconciles running jobs before serving; changes SIGTERM handling to deregister only when no jobs are running.
crates/spurd/src/executor.rs	Implements job manifests, exit-status sentinel wrapping, resumed-job completion detection via cgroup/proc liveness, and run-attempt-scoped cgroups; adjusts spool-dir behavior.
crates/spurd/src/container.rs	Makes `RootfsMode` serializable for inclusion in job manifests.
crates/spurd/src/agent_server.rs	Adds startup reconciliation to restore allocations/track resumed jobs; writes manifests immediately after launch.
crates/spurctld/src/main.rs	Warns at startup when `heartbeat_timeout_secs` is configured below 30s.
crates/spur-sched/src/cons_tres.rs	Adds `restore_committed()` to pin/restore exact resource IDs after agent restart.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

crates/spurd/src/executor.rs:483

write_job_manifest() uses std::fs::write(), which creates manifest.json with default permissions (typically 0644). Because per-job spool dirs are 0o711, any local user who can guess the path (job IDs are enumerable) can read another job’s manifest and learn paths/UID/GID/cgroup/resource details. The manifest should be root-only readable (0600) and ideally written via an OpenOptions path that refuses symlinks.

pub fn write_job_manifest(spool_dir: &Path, manifest: &JobManifest) {
    let path = manifest_path(spool_dir);
    let tmp = path.with_extension("json.tmp");
    let bytes = match serde_json::to_vec(manifest) {
        Ok(b) => b,

yansun1996 · 2026-07-30T00:14:37Z

+/// A job_id should only ever have one manifest (whichever `SPOOL_ROOT`/tmp
+/// base `create_job_spool_dir` used), but a stale leftover in the other base
+/// (e.g. from a run under a different privilege level) would otherwise make
+/// `reconcile_running_jobs` double-restore its allocation. Keep only the
+/// highest run_attempt per job_id.
+fn dedup_manifests_by_job_id(found: Vec<(PathBuf, JobManifest)>) -> Vec<(PathBuf, JobManifest)> {
+    let mut by_job: HashMap<JobId, (PathBuf, JobManifest)> = HashMap::new();
+    for (dir, manifest) in found {
+        match by_job.get(&manifest.job_id) {
+            Some((_, existing)) if existing.run_attempt >= manifest.run_attempt => {
+                warn!(
+                    job_id = manifest.job_id,
+                    dir = %dir.display(),
+                    "ignoring duplicate/stale job manifest"
+                );
+            }
+            _ => {
+                by_job.insert(manifest.job_id, (dir, manifest));
+            }
+        }
+    }
+    by_job.into_values().collect()
+}


Thanks — I looked at this closely. Two run attempts of the same job can't be simultaneously live on one node at reconcile time: a same-node redispatch reuses the job_id key and the displaced older attempt is SIGKILL'd and reaped before the new one is tracked, and the controller only re-dispatches after the prior attempt has been reported complete (a job re-enters Pending only after its previous run finalized). So a lingering lower-attempt manifest is always a stale leftover, never a live attempt — keeping the highest is correct. The one real wart is that the displaced-kill path doesn't immediately clean up the old attempt's spool dir, so a stale manifest can linger until the new run completes; harmless for reconcile, but I can tidy it in a follow-up. Happy to reconsider if you have a redispatch sequence in mind where both attempts are genuinely live at once.

spurd's SIGTERM handler force-deregistered the node on every graceful restart, which the controller treats as an immediate forced eviction of all running jobs regardless of how briefly the agent was down. A restart for a routine binary upgrade thus always failed in-flight jobs. Now: SIGTERM only deregisters when the agent has no running jobs. When jobs are running, spurd writes a small per-job manifest (pid, start time, cgroup, resource footprint) right after launch, and on the next startup re-adopts any job whose process survived, restoring its exact resource allocation before accepting new launches. Completion for a re-adopted job (which is no longer spurd's child, so waitpid doesn't work) is detected via cgroup population plus an exit-code sentinel written by the job's own wrapper script. Known limitations, documented rather than solved here: a restart slower than heartbeat_timeout_secs still falls back to today's eviction path (now the sole safety net, not a secondary one); a job whose run was redispatched elsewhere during such a slow restart isn't proactively cleaned up on this node; and PMI rendezvous state for a job mid-launch during a restart is still lost.

A job runs as its own uid in a spool directory chowned to it, so it could delete/replace the root-authored job manifest a restarted spurd later trusts (cgroup path, resource ids) — make the directory sticky and world-writable instead of job-owned, so it can still create its exit-status file but can't touch files it doesn't own, and have the manifest write unlink any pre-existing path first to close a pre-creation ownership race. Cgroup paths were keyed by job_id alone, so a same-node redispatch landed the new run in the same cgroup as a still-finishing old (possibly resumed) one; displacing the old run could cgroup-wide-kill the new run too. Key by job_id + run_attempt instead. Also: dedup job manifests by job_id (a stale one under the temp-dir fallback base could otherwise get reconciled alongside a current one under the real spool root, double-reserving resources), make NodeAllocation::restore_committed idempotent as a second guard against the same, and isolate the new manifest-scanning tests against each other so they can't pick up a concurrently-running test's manifest.

Review pass surfaced several security/correctness issues in the persist-and- resume path: - The per-job spool dir was world-writable (0o1777), letting a co-located user symlink-race the root-authored manifest (root arbitrary-file-overwrite) or tamper with another job's files. The dir is now 0o711 root-owned; the job's exit-status sentinel is pre-created and chowned to the job, so the job no longer needs to create files in the dir. The manifest unlink-before-write dance is removed with the writable dir. - scan_job_manifests / spool creation no longer use the user-controlled temp fallback base when spurd is root, so root never trusts a planted manifest. - The SIGTERM shutdown check now treats mid-launch (reserved-but-not-committed) jobs as active, closing a window where a restart could force-evict an in-flight launch. - restore_committed / allocate use saturating_add so a corrupt manifest can't wrap the memory accounting counter.

A container job re-adopted after a spurd restart always reported exit -1, even on success: its script runs inside the pivoted rootfs, where the exit-status sentinel written by the non-container path isn't reachable. Apply the same exit-sentinel wrapper to the container script, writing to an in-container path that is host-visible in the rootfs, and record that path in the job manifest. Reconcile and the completion monitor now read the real code for both plain and container jobs; the -1 shortcut is gone. Adds a unit test for the resumed-container exit code and an e2e that restarts spurd mid-job and asserts both a plain and a container job report their true exit code afterward.

…oped Two hardening fixes to the restart-adopt path plus wider e2e coverage: - write_job_manifest now writes to a temp file and renames into place, so a crash mid-write can't leave a torn manifest that the scanner would skip (silently dropping a survivor). - Job spool dirs are keyed by (job_id, run_attempt), matching the cgroup keying, so a same-node re-dispatch can't read a stale exit-status sentinel from a prior run. cleanup_job_spool removes every attempt for a job id. Expands the exit-code-across-restart e2e to the full matrix: plain and container jobs, each for a success (0) and a failure (7) exit code.

…e serving A job that finished while spurd was down had its completion reported via a detached, fire-and-forget task during reconcile. If that report was dropped (controller briefly unreachable), the job's real exit code was lost and it only cleared via the heartbeat timeout as a synthetic outcome. Report these completions inline (awaited) before the agent starts serving. Registration already requires a reachable controller moments earlier, so awaiting here adds no new startup dependency. Split the manifest-adoption logic from the network reporting so it stays unit-testable without a live controller.

… path - Unit: one reconcile pass with both a surviving job and a finished job adopts the live one into `running` and reports only the dead one's real exit code. - E2E: a job that finishes entirely while spurd is dead (SIGKILL, no deregister) must report its true exit code via the manifest sentinel on restart, not a heartbeat-timeout outcome.

…mpletion A review pass surfaced several correctness and safety issues in the restart-adopt path, fixed here: - cleanup_job_spool is now scoped to a single (job_id, run_attempt) instead of prefix-deleting every attempt, so cleaning a finished run can't wipe a concurrent same-node redispatch's live spool and exit-status sentinel. - A job OOM-killed while spurd was down now recovers the OOM flag from the cgroup on the reconcile Dead path, matching the live monitor path, instead of reporting a bare -1. - read_exit_status opens the sentinel with O_NOFOLLOW so root won't follow a symlink a job may have planted at the in-container sentinel path. - Completion reporting during startup reconcile is bounded by a timeout so an unreachable controller can't keep spurd from serving; the heartbeat timeout remains the backstop. Adds unit tests for per-attempt cleanup isolation, OOM recovery on the Dead path, and symlink refusal.

Cap read_exit_status at 64 bytes so a container job cannot make the restart reconcile path read an arbitrarily large file from its writable rootfs, and write job manifests 0600 with O_NOFOLLOW so a co-located user cannot read another job's uid/gid/cgroup/resource details from the traversable spool dir.

yansun1996 requested a review from Copilot July 23, 2026 00:17

Copilot started reviewing on behalf of yansun1996 July 23, 2026 00:18 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread crates/spurd/src/executor.rs

Comment thread crates/spurd/src/executor.rs Outdated

Comment thread crates/spurd/src/executor.rs Outdated

Comment thread crates/spurd/src/main.rs Outdated

Comment thread crates/spur-sched/src/cons_tres.rs Outdated

yansun1996 marked this pull request as ready for review July 23, 2026 08:55

yansun1996 requested review from powderluv, sajmera-pensando, sgopinath1 and shiv-tyagi as code owners July 23, 2026 08:55

yansun1996 force-pushed the feat/spurd-seamless-upgrade branch 2 times, most recently from fa44b48 to 968f6a7 Compare July 25, 2026 05:03

yansun1996 requested a review from Copilot July 29, 2026 22:12

Copilot started reviewing on behalf of yansun1996 July 29, 2026 22:13 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

yansun1996 added 9 commits July 29, 2026 22:48

yansun1996 force-pushed the feat/spurd-seamless-upgrade branch from 2293453 to 1cbb947 Compare July 30, 2026 00:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(spurd): survive agent restart without interrupting running jobs - #492

feat(spurd): survive agent restart without interrupting running jobs#492
yansun1996 wants to merge 9 commits into
ROCm:mainfrom
yansun1996:feat/spurd-seamless-upgrade

yansun1996 commented Jul 22, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

yansun1996 Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yansun1996 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Approach

Known limitations (not solved here)

Testing

Uh oh!

codecov-commenter commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yansun1996 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yansun1996 commented Jul 22, 2026 •

edited

Loading

codecov-commenter commented Jul 22, 2026 •

edited

Loading