coordinator: stale PendingRestart leaks reply_sender and hangs caller if a daemon dies mid-restart

[phil-opp]

Background

PR #2085 (merged) reworked dataflow restart into a two-phase, event-driven flow to fix the cross-daemon Zenoh race in #2082:

• Phase 1 (initiate_restart) sends StopDataflow to all daemons, persists Stopping, and parks a PendingRestart (descriptor + name + uv + the caller's reply_sender) in pending_restarts.
• Phase 2 completes inside the DataflowFinishedOnDaemon handler when the last daemon reports finish, calling start_dataflow and firing the parked reply.

The core fix is correct. This issue tracks the follow-up gaps that PR #2085 left behind (one of them already flagged as an inline TODO).

Problem 1 — caller hangs forever if a daemon dies mid-restart (primary)

If a daemon crashes/disconnects after receiving StopDataflow but before sending DataflowFinishedOnDaemon, the PendingRestart entry is never removed and its reply_sender never fires. The restart caller (CLI) blocks indefinitely with no error.

This is a behavioral regression relative to the old synchronous restart_dataflow, which surfaced stop/start failures immediately. The two-phase flow trades a visible error for a silent hang.

The PR documents this as an inline TODO in initiate_restart:

// TODO: If a daemon crashes after receiving StopDataflow but before
// sending DataflowFinishedOnDaemon, this entry (and its reply_sender)
// will never be cleaned up, causing the restart caller to hang
// indefinitely. A timeout or a daemon-disconnect cleanup path should
// be added to evict stale PendingRestart entries.

Proposed fix: evict stale PendingRestart entries and reply Err when their restart can no longer complete — either via a timeout, or (preferred) by draining matching pending_restarts entries in the daemon-disconnect cleanup path so the caller gets a clear error.

Problem 2 — double-restart guard runs after StopDataflow (minor)

In initiate_restart, the pending_restarts.contains_key(...) guard is checked after stop_dataflow has already broadcast StopDataflow to every daemon. A duplicate Restart for an already-restarting dataflow therefore incurs a redundant stop round-trip before being rejected. The guard should be moved ahead of the stop_dataflow call (right after extracting the descriptor) so duplicates are rejected before any daemon I/O.

Problem 3 — no coordinator-side test for the new restart path (minor)

The new two-phase restart shipped without a coordinator test. The race itself is hard to reproduce deterministically, but the double-restart guard and the deferred-completion → reply wiring are both testable in binaries/coordinator/tests/ without real daemons. Adding coverage there would lock in the guard behavior and the cleanup path from Problem 1.

References

• PR fix(coordinator): two-phase restart to prevent cross-daemon Zenoh data-plane race after dataflow restart #2085 (the two-phase restart fix)
• Nightly regression since 2026-06-07 #2082 (the original flaky cluster-e2e race)

[phil-opp]

Investigating this as part of an automated issue review pass — the three problems described here all appear to already be resolved in the current main:

Problem 1 (caller hangs on daemon crash mid-restart): cleanup_disconnected_daemons_from_running_dataflows already drains pending_restarts for all affected dataflows and fires the reply_sender with an error (coordinator/src/lib.rs:3046–3058). The inline TODO is gone.

Problem 2 (double-restart guard runs after StopDataflow): initiate_restart already checks pending_restarts.contains_key(&dataflow_uuid) as the first thing it does, before any stop_dataflow call (coordinator/src/lib.rs:4624–4631).

Problem 3 (no coordinator test): Two tests exist:

• cleanup_disconnected_daemons_drains_pending_restarts — verifies the stale entry is removed and the caller is released with an error (lib.rs:7833)
• initiate_restart_rejects_duplicate_request — verifies the double-restart guard (lib.rs:7881)

Both pass under cargo test -p dora-coordinator. This issue can likely be closed unless there are additional cases not captured here.

---

Generated by Claude Code

[phil-opp]

After reviewing the current codebase, both problems described here are already addressed:

Problem 1 — daemon crash during restart hangs caller

Fixed in binaries/coordinator/src/lib.rs around line 3046 (cleanup_disconnected_daemons_from_running_dataflows). When a daemon disconnects, the function drains pending_restarts for every affected dataflow UUID and fires reply_sender with an error:

for uuid in &affected_uuids {
    if let Some(restart) = pending_restarts.remove(uuid) {
        tracing::warn!(dataflow = %uuid, "daemon disconnected while restart was pending; cancelling restart");
        let _ = restart.reply_sender.send(Err(eyre!(
            "daemon disconnected while restart was pending for dataflow `{uuid}`"
        )));
    }
}

This is invoked from the Event::DaemonExit handler (around line 2209), so a daemon crash/disconnect during Phase 1 now returns an error to the CLI caller instead of hanging.

Problem 2 — double-restart guard

Present at binaries/coordinator/src/lib.rs around line 4624, placed before any side effects in initiate_restart:

if pending_restarts.contains_key(&dataflow_uuid) {
    let _ = reply_sender.send(Err(eyre!(
        "dataflow `{dataflow_uuid}` is already being restarted – wait for it to finish"
    )));
    return;
}

Tests

Both behaviors are covered by unit tests in the same file:

• cleanup_disconnected_daemons_drains_pending_restarts (Problem 1)
• initiate_restart_rejects_duplicate_request (Problem 2)

It looks like these fixes landed after this issue was filed. This issue can be closed.

---

Generated by Claude Code

[phil-opp]

All three problems described in this issue have been addressed in the codebase:

Problem 1 (caller hangs on daemon disconnect): cleanup_disconnected_daemons_from_running_dataflows now drains matching pending_restarts entries and fires the reply with an error when a daemon disconnects mid-restart (lib.rs:3046–3059). The inline TODO referenced in the issue has been removed.

Problem 2 (double-restart guard fires after StopDataflow): The pending_restarts.contains_key(...) guard was moved to the top of initiate_restart, before stop_dataflow is called (lib.rs:4626–4631).

Problem 3 (no test): Two unit tests cover the new behavior:

• cleanup_disconnected_daemons_drains_pending_restarts (lib.rs:7833) — verifies the caller receives an Err on daemon disconnect.
• initiate_restart_rejects_duplicate_request (lib.rs:7881) — verifies duplicate restarts are rejected before any daemon I/O.

This issue can be closed.

---

Generated by Claude Code