Skip to content

Panic during graceful shutdown: JoinError::Cancelled in session.rs #949

Description

@anarchia-99

Description / Situation
During a graceful shutdown (SIGTERM) of the kaspad Docker container, the node correctly stops network connections and begins flushing remaining blocks from RAM to the database. However, during this teardown phase, a Tokio worker thread panics because of an unhandled .unwrap() on a cancelled thread in the consensusmanager (session.rs).

To Reproduce
Steps to reproduce the behavior:

Run rusty-kaspad in a Docker container using a slower storage medium (e.g., an external USB HDD) for the /app/data volume.

Run the node with a higher RAM scale (e.g., --ram-scale=3.0) so it holds a larger backlog of blocks in memory.

Initiate a graceful shutdown via docker stop or docker restart (sending a SIGTERM signal).

See error in the logs: The node stops P2P/GRPC, successfully flushes blocks to the database in batches of 100, but then panics and exits ungracefully.

Expected behavior
The node should finish flushing the remaining blocks to the database and exit cleanly without throwing a Tokio runtime panic due to thread cancellation.

Screenshots
Not applicable, but here is the exact log sequence of the crash:
2026-04-09 07:05:20.133+00:00 [INFO ] Interrupted while advancing the pruning point UTXO set: Process is exiting
2026-04-09 07:05:20.504+00:00 [INFO ] P2P Server stopped: 0.0.0.0:16111
2026-04-09 07:05:26.331+00:00 [INFO ] Accepted 100 blocks ... via relay
[...] (successfully flushing multiple batches of 100 blocks)
2026-04-09 07:05:37.539+00:00 [ERROR] thread 'tokio-runtime-worker' panicked at /rusty-kaspa/components/consensusmanager/src/session.rs:168:66: called Result::unwrap() on an Err value: JoinError::Cancelled(Id(12433022))
Exiting...

Desktop:

  • OS: Linux Ubuntu server 24.04.4 LTS - architecture: amd64 (as Docker host)
    
  • Kaspad version: kaspanet/rusty-kaspad:latest (Docker Image)
    

Additional context
The container is configured with stop_grace_period: 20m. Docker did not forcefully kill the container (SIGKILL); the software crashed itself via the unwrap panic while plenty of grace period time was still available. The database remains intact, so this seems to be a pure thread teardown race condition exacerbated by the slower I/O of the HDD during the flush.

Here is the exact docker-compose.yml used for this node:

kaspad:
image: kaspanet/rusty-kaspad:latest
container_name: kaspad
restart: unless-stopped
cpus: '2.5'
cpu_shares: 512
mem_limit: 28g
memswap_limit: 28g
ports:
- "16111:16111/tcp" # P2P Port
volumes:
- /mnt/kaspa-storage:/app/data
command:
- kaspad
- --yes
- --disable-upnp
- --listen=0.0.0.0:16111
- --maxinpeers=10
- --outpeers=8
- --rocksdb-preset=hdd
- --ram-scale=3.0
- --retention-period-days=2
- --loglevel=info
stop_grace_period: 20m
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
mode: "non-blocking"
labels:
- "com.centurylinklabs.watchtower.enable=true"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions