fix(consensusmanager): don't panic on spawn_blocking cancellation during shutdown (#949) - #1030
Open
atharaldsen wants to merge 1 commit into
Open
Conversation
…ing shutdown (kaspanet#949) `ConsensusSessionOwned::spawn_blocking` did `spawn_blocking(...).await.unwrap()`. When the Tokio runtime shuts down it cancels not-yet-started blocking tasks, so the join returns `Err(JoinError::Cancelled)` and `.unwrap()` panics a runtime-worker thread mid-teardown (observed during graceful shutdown under slow I/O, e.g. flushing a large block backlog to an HDD). Handle the join result explicitly: - Cancellation only happens during runtime shutdown and has no result to return, so park until the task is dropped as part of teardown instead of panicking. - A genuine panic inside the closure is still propagated via `resume_unwind`, preserving the previous behavior (we don't mask real bugs). Adds unit tests covering all three paths (result returned, cancellation parks, closure panic propagates) using a local multi-thread runtime — no extra deps.
atharaldsen
added a commit
to atharaldsen/rusty-kaspa
that referenced
this pull request
May 31, 2026
…shutdown (kaspanet#1031) Several `spawn_blocking(...).await.unwrap()` (and one `try_join_all(...).await.unwrap()`) sites panic a runtime worker thread when the Tokio runtime cancels a not-yet-started blocking task during shutdown (`Err(JoinError::Cancelled)`). This is the same class of bug fixed for `consensusmanager::session` in kaspanet#1030 (issue kaspanet#949). Add `kaspa_core::task::blocking::{join_blocking_or_park, join_result_or_park}`: - on cancellation (runtime shutdown, no result to return) park until teardown drops the task; - on a genuine closure panic, propagate via `resume_unwind` (preserving prior behavior). Apply it to the remaining sites from kaspanet#1031: - `mining::manager::MiningManagerProxy` mempool proxy methods (9 sites) - `indexes::utxoindex::UtxoIndexProxy` methods (4 sites) - `protocol/flows` IBD staging commit and consensus-reset - `connectionmanager` dns_seed blocking call and `try_join_all` The `ConsensusProxy::spawn_blocking` call sites (handled by kaspanet#1030) are intentionally left untouched. `consensusmanager::session` retains its local helper from kaspanet#1030; consolidating it onto this shared helper is left as a follow-up to avoid conflicting with kaspanet#1030 while in review. Adds unit tests for the shared helper (result returned, cancellation parks, closure panic propagates).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #949.
ConsensusSessionOwned::spawn_blockingdidspawn_blocking(...).await.unwrap(). When the Tokio runtime shuts down it cancels not-yet-started blocking tasks, so the join returnsErr(JoinError::Cancelled)and.unwrap()panics a runtime worker thread mid-teardown. This is observed during graceful shutdown under slow I/O (e.g. flushing a large block backlog to an HDD), where the node crashes instead of exiting cleanly — confirmed by @D-Stacks on the issue.Handle the join result explicitly:
resume_unwind, preserving previous behavior (real bugs are not masked).Testing
Adds unit tests covering all three paths — result returned, cancellation parks, closure panic propagates — using a local multi-thread runtime (no new deps). The original crash is an intermittent shutdown race that isn't reliably reproducible, so the tests drive each
JoinErrorbranch deterministically with tokio's ownabort()/panic!().