Skip to content

feat(state): Reth height replay when lagging behind #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
bastienfaivre wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat(state): Reth height replay when lagging behind #154

bastienfaivre wants to merge 6 commits into from

Conversation

@bastienfaivre
Copy link
Collaborator

@bastienfaivre bastienfaivre commented Dec 23, 2025

Closes #126.

This is an attempt to replay heights in reth at initialization when needed.

I have not yet tested the correctness of the implementation. We would need to have an integration test framework, or can we test that more easily?

@bastienfaivre bastienfaivre marked this pull request as draft December 23, 2025 11:31
@jmalicevic
Copy link
Contributor

jmalicevic commented Dec 23, 2025

I have not yet tested the correctness of the implementation. We would need to have an integration test framework, or can we test that more easily?

Right now we don't have a proper automated testing framework so the way to test it is to manually reproduce it. How to do that:

  • Remove - "--engine.persistence-threshold=0" in compose.yaml for the reth node you want to crash .
  • make testnet-start
  • bash scripts/kill-node.sh NODE_ID
  • bash scripts/restart-node.sh NODE_ID.
    It should crash. In Emerald logs you will see that SYNC is timing out. In the Reth log you can observe the node is waiting to move forward.

jmalicevic
jmalicevic reviewed Dec 23, 2025
View reviewed changes
bastienfaivre
bastienfaivre commented Dec 23, 2025
View reviewed changes
app/src/app.rs
debug!("🌈 Got block validator set: {:?}", block_validator_set);
state.set_validator_set(start_height, block_validator_set);

// Consensus will start at the next height, so we set the validator set for that height
Copy link
Collaborator Author

@bastienfaivre bastienfaivre Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmalicevic I also fixed the code here. The validator set from the latest accepted block was stored for the wrong height (current instead of next), causing the code to fail later since the validator set for the next height is requested.

@bastienfaivre
Copy link
Collaborator Author

bastienfaivre commented Dec 23, 2025

Height replay is working. Reth successfully updates, and the node syncs eventually. From logs:

2025-12-23T15:38:15.742591Z  INFO Loaded configuration file=nodes/3/config/config.toml
2025-12-23T15:38:15.749946Z  INFO node{moniker=test-3}: Discovery is disabled
2025-12-23T15:38:15.750231Z  INFO node{moniker=test-3}:network: Starting network service peer_id=16Uiu2HAm5yyr4uDFXW2UCp2TQYajeA9xZc4wLfmK6mowokcYGhYX
2025-12-23T15:38:15.755840Z  INFO node{moniker=test-3}:wal.pre_start: Opened WAL at nodes/3/wal/consensus.wal
2025-12-23T15:38:15.756182Z  INFO node{moniker=test-3}:consensus: Consensus is starting
2025-12-23T15:38:15.757212Z DEBUG node{moniker=test-3}:network: Node is listening address=/ip4/127.0.0.1/tcp/27003
2025-12-23T15:38:15.757550Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Consensus has started
2025-12-23T15:38:15.757611Z  INFO node{moniker=test-3}:node: Actor has started actor=0.0
2025-12-23T15:38:15.757618Z  INFO node{moniker=test-3}:node: Actor has started actor=0.1
2025-12-23T15:38:15.757620Z  INFO node{moniker=test-3}:node: Actor has started actor=0.2
2025-12-23T15:38:15.757622Z  INFO node{moniker=test-3}:node: Actor has started actor=0.3
2025-12-23T15:38:15.757877Z  INFO node{moniker=test-3}:metrics: Serving metrics address=127.0.0.1:29003
2025-12-23T15:38:15.758012Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Listening address=/ip4/127.0.0.1/tcp/27003
2025-12-23T15:38:15.758015Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Phase transition prev=Unstarted new=Ready
2025-12-23T15:38:15.758051Z  INFO node{moniker=test-3}:node: Actor has started actor=0.4
2025-12-23T15:38:15.758144Z DEBUG node{moniker=test-3}:network: Adding bootstrap node to dial queue: peer_id=None, queue_len_before=0, in_progress_len=0
2025-12-23T15:38:15.758179Z DEBUG node{moniker=test-3}:network: Adding bootstrap node to dial queue: peer_id=None, queue_len_before=1, in_progress_len=0
2025-12-23T15:38:15.758182Z DEBUG node{moniker=test-3}:network: Adding bootstrap node to dial queue: peer_id=None, queue_len_before=2, in_progress_len=0
2025-12-23T15:38:15.758203Z DEBUG node{moniker=test-3}:network: Dialing peer None at [/ip4/127.0.0.1/tcp/27000], retry #0 connection_id=1
2025-12-23T15:38:15.758308Z DEBUG node{moniker=test-3}:network: Dialing peer None at [/ip4/127.0.0.1/tcp/27001], retry #0 connection_id=2
2025-12-23T15:38:15.758316Z DEBUG node{moniker=test-3}:network: Dialing peer None at [/ip4/127.0.0.1/tcp/27002], retry #0 connection_id=3
2025-12-23T15:38:15.761506Z DEBUG node{moniker=test-3}:network: Connected to peer peer=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 connection_id=1
2025-12-23T15:38:15.762973Z DEBUG node{moniker=test-3}:network: Connected to peer peer=16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji connection_id=3
2025-12-23T15:38:15.763083Z DEBUG node{moniker=test-3}:network: Connected to peer peer=16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu connection_id=2
2025-12-23T15:38:15.764234Z DEBUG node{moniker=test-3}:network: Checking peer 16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 against 3 bootstrap nodes
2025-12-23T15:38:15.764244Z  INFO node{moniker=test-3}:network: Bootstrap peer 16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 successfully identified
2025-12-23T15:38:15.764274Z  INFO node{moniker=test-3}:network: Discovered peer peer=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 connection_id=1
2025-12-23T15:38:15.764284Z DEBUG node{moniker=test-3}:network: Connection is inbound peer=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 connection_id=1
2025-12-23T15:38:15.764292Z  INFO node{moniker=test-3}:network: Connections: 1
2025-12-23T15:38:15.764361Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Connected to peer peer_id=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36
2025-12-23T15:38:15.764372Z DEBUG node{moniker=test-3}:consensus{height=1 round=-1}: Connected to another peer connected=1 total=3
2025-12-23T15:38:15.765978Z DEBUG node{moniker=test-3}:network: Checking peer 16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji against 3 bootstrap nodes
2025-12-23T15:38:15.765988Z  INFO node{moniker=test-3}:network: Bootstrap peer 16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji successfully identified
2025-12-23T15:38:15.765995Z  INFO node{moniker=test-3}:network: Discovered peer peer=16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji connection_id=3
2025-12-23T15:38:15.766000Z DEBUG node{moniker=test-3}:network: Connection is inbound peer=16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji connection_id=3
2025-12-23T15:38:15.766006Z  INFO node{moniker=test-3}:network: Connections: 2
2025-12-23T15:38:15.766450Z DEBUG node{moniker=test-3}:network: Checking peer 16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu against 3 bootstrap nodes
2025-12-23T15:38:15.766458Z  INFO node{moniker=test-3}:network: Bootstrap peer 16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu successfully identified
2025-12-23T15:38:15.766465Z  INFO node{moniker=test-3}:network: Discovered peer peer=16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu connection_id=2
2025-12-23T15:38:15.766471Z DEBUG node{moniker=test-3}:network: Connection is inbound peer=16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu connection_id=2
2025-12-23T15:38:15.766476Z  INFO node{moniker=test-3}:network: Connections: 3
2025-12-23T15:38:15.766568Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Connected to peer peer_id=16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji
2025-12-23T15:38:15.766576Z DEBUG node{moniker=test-3}:consensus{height=1 round=-1}: Connected to another peer connected=2 total=3
2025-12-23T15:38:15.766587Z  INFO node{moniker=test-3}:consensus{height=1 round=-1}: Connected to peer peer_id=16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu
2025-12-23T15:38:15.766592Z DEBUG node{moniker=test-3}:consensus{height=1 round=-1}: Connected to another peer connected=3 total=3
2025-12-23T15:38:15.802816Z  INFO 🟢🟢 Consensus is ready
2025-12-23T15:38:15.809608Z DEBUG 🎁 block size: 545, height: 122
2025-12-23T15:38:15.809620Z DEBUG 🟠 get_latest_block_number
2025-12-23T15:38:15.812466Z DEBUG response body: JsonResponseBody { jsonrpc: "2.0", error: None, result: Object {"baseFeePerGas": String("0x71"), "blobGasUsed": String("0x0"), "difficulty": String("0x0"), "excessBlobGas": String("0x0"), "extraData": String("0x726574682f76312e392e332f6c696e7578"), "gasLimit": String("0x32e3e2c"), "gasUsed": String("0x0"), "hash": String("0xa172b5d0d8c1cd388b21adc02e3f140b16cfabc5ea2f2170bc77792d73a22f61"), "logsBloom": String("0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"), "miner": String("0x4242424242424242424242424242424242424242"), "mixHash": String("0x0000000000000000000000000000000000000000000000000000000000000000"), "nonce": String("0x0000000000000000"), "number": String("0x78"), "parentBeaconBlockRoot": String("0xe54db03ab12295141802e984cbd0bafb8ff3afeeddfcf2d696eae4458fabb588"), "parentHash": String("0xe54db03ab12295141802e984cbd0bafb8ff3afeeddfcf2d696eae4458fabb588"), "receiptsRoot": String("0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"), "requestsHash": String("0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"), "sha3Uncles": String("0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347"), "size": String("0x275"), "stateRoot": String("0x16c8116db8252718a837c4c327050bd0f47aa4e2b59086ffb8dce4be1d7c8b89"), "timestamp": String("0x694ab759"), "transactions": Array [], "transactionsRoot": String("0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"), "uncles": Array [], "withdrawals": Array [], "withdrawalsRoot": String("0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421")}, id: Number(1) }
2025-12-23T15:38:15.812572Z  WARN ⚠️  Execution client is at height 120 but Emerald has blocks up to height 122. Starting height replay.
2025-12-23T15:38:15.812576Z  INFO 🔄 Replaying heights 121 to 122 to execution client
2025-12-23T15:38:15.812688Z DEBUG 🔄 Replaying block at height 121 with hash 0xd425e554a84a43f7fc8f0851a8d318b693d0d21ff6c4e815d93e8d9efd9ac41b
2025-12-23T15:38:15.828809Z DEBUG ✅ Block at height 121 replayed successfully
2025-12-23T15:38:15.828815Z DEBUG 🟠 set_latest_forkchoice_state: 0xd425e554a84a43f7fc8f0851a8d318b693d0d21ff6c4e815d93e8d9efd9ac41b
2025-12-23T15:38:15.830178Z DEBUG ➡️ payload_status: PayloadStatus { status: Valid, latest_valid_hash: Some(0xd425e554a84a43f7fc8f0851a8d318b693d0d21ff6c4e815d93e8d9efd9ac41b) }
2025-12-23T15:38:15.830186Z DEBUG 🎯 Forkchoice updated to height 121
2025-12-23T15:38:15.830255Z DEBUG 🔄 Replaying block at height 122 with hash 0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397
2025-12-23T15:38:15.839990Z DEBUG ✅ Block at height 122 replayed successfully
2025-12-23T15:38:15.839999Z DEBUG 🟠 set_latest_forkchoice_state: 0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397
2025-12-23T15:38:15.841012Z DEBUG ➡️ payload_status: PayloadStatus { status: Valid, latest_valid_hash: Some(0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397) }
2025-12-23T15:38:15.841018Z DEBUG 🎯 Forkchoice updated to height 122
2025-12-23T15:38:15.841021Z  INFO ✅ Successfully replayed all heights to execution client
2025-12-23T15:38:15.841024Z  INFO ✅ Height replay completed successfully
2025-12-23T15:38:15.841028Z DEBUG 🟠 send_forkchoice_updated: 0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397
2025-12-23T15:38:15.841700Z DEBUG Payload is valid
2025-12-23T15:38:15.841705Z DEBUG latest block Some(ExecutionBlock { block_hash: 0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397, block_number: 122, parent_hash: 0xd425e554a84a43f7fc8f0851a8d318b693d0d21ff6c4e815d93e8d9efd9ac41b, timestamp: 1766504282, prev_randao: 0x0000000000000000000000000000000000000000000000000000000000000000 })
2025-12-23T15:38:15.846020Z DEBUG 🌈 Got validator set: ValidatorSet { validators: [Validator { address: Address(5A9245DCE516AA85C8D82A90608A542A151D9E91), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([827532247923365, 2361805573119693, 3492543941731791, 1191675179907230, 237915650423024]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([4366659347211492, 2160934874953051, 581678635108779, 2650494975570035, 111524612954901]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(7D17AA4FE6C1E7C58D1B26F5A68C35BE0BFF6C29), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([1169858868288094, 147681501803464, 1111504367607205, 2144018211257096, 171110521269953]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([1744468881357823, 686784299215419, 1397329182505972, 713931089011640, 127275670879953]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(311E280D2918E93A90EEA22C0773053F325CE409), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([2820261586041093, 1505913081254658, 341150345958183, 3691244156032531, 54358481847654]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([1047127412350518, 3833800791933623, 3295542033967919, 4089411062058281, 57650081492668]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([27967812440236, 1160044011676679, 3854365601680896, 2804222629534947, 172467234474493]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([131342944875488, 1984349185449754, 1239639448107705, 2603967101376622, 211158001754920]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }] } for height 123
2025-12-23T15:38:15.846173Z DEBUG Start height in state set to: Height(123); height in store is Height(122) 
2025-12-23T15:38:15.846503Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: No WAL entries to replay height=123
2025-12-23T15:38:15.846638Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Starting new height height=123
2025-12-23T15:38:15.846726Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Starting new round height=123 round=0 proposer=311E280D2918E93A90EEA22C0773053F325CE409 role=Validator
2025-12-23T15:38:15.850802Z DEBUG node{moniker=test-3}:wal{height=123}: Flushed WAL to disk wal.entries=0 wal.size=12
2025-12-23T15:38:15.850848Z  INFO 🟢🟢 Started round height=123 round=0 proposer=311E280D2918E93A90EEA22C0773053F325CE409 role=Validator
2025-12-23T15:38:15.850890Z DEBUG Found 0 pending proposal parts, validating... height=123 round=0
2025-12-23T15:38:15.850941Z DEBUG Found 0 undecided proposals height=123 round=0
2025-12-23T15:38:15.850969Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Transitioned to new step step.previous=Unstarted step.new=Propose
2025-12-23T15:38:15.850975Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Scheduling timeout round=0 step=Propose
2025-12-23T15:38:15.850983Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Replaying inputs count=0
2025-12-23T15:38:15.850991Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Phase transition prev=Ready new=Running
2025-12-23T15:38:15.851073Z DEBUG node{moniker=test-3}:sync{height.tip=0}: Consensus started new height height=123 is_restart=false
2025-12-23T15:38:18.287392Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=311E280D2918E93A90EEA22C0773053F325CE409
2025-12-23T15:38:18.295884Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=5A9245DCE516AA85C8D82A90608A542A151D9E91
2025-12-23T15:38:18.299975Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=7D17AA4FE6C1E7C58D1B26F5A68C35BE0BFF6C29
2025-12-23T15:38:18.304659Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=7D17AA4FE6C1E7C58D1B26F5A68C35BE0BFF6C29
2025-12-23T15:38:18.308831Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=5A9245DCE516AA85C8D82A90608A542A151D9E91
2025-12-23T15:38:18.312910Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received vote for higher height, queuing for later consensus.height=123 consensus.round=0 vote.height=132 vote.round=0 validator=311E280D2918E93A90EEA22C0773053F325CE409
2025-12-23T15:38:18.711100Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Received peer status peer.id=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36 peer.height=131
2025-12-23T15:38:18.711169Z  INFO node{moniker=test-3}:sync{height.tip=122}: SYNC REQUIRED: Falling behind height.tip=122 height.peer=131
2025-12-23T15:38:18.711209Z  INFO node{moniker=test-3}:sync{height.tip=122}: Requesting sync from peer range=123..=131 peer=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36
2025-12-23T15:38:18.711440Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Sent sync request to peer request_id=1 range=123..=131 peer=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36
2025-12-23T15:38:18.711465Z  INFO node{moniker=test-3}:sync{height.tip=122}: Could not find a range to request up_to_height=131
2025-12-23T15:38:18.756864Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Received peer status peer.id=16Uiu2HAmP8Ur7mwPGN2NeVLL8Zxb2EmLA2rYBUMJJj9pWgYv7rGu peer.height=131
2025-12-23T15:38:18.756882Z  INFO node{moniker=test-3}:sync{height.tip=122}: SYNC REQUIRED: Falling behind height.tip=122 height.peer=131
2025-12-23T15:38:18.756886Z  INFO node{moniker=test-3}:sync{height.tip=122}: Could not find a range to request up_to_height=131
2025-12-23T15:38:18.766657Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Received peer status peer.id=16Uiu2HAkxkfHzm6yX63ekW6A2EbqWJhzHTQu4qdVbCnccUrUq3ji peer.height=131
2025-12-23T15:38:18.766669Z  INFO node{moniker=test-3}:sync{height.tip=122}: SYNC REQUIRED: Falling behind height.tip=122 height.peer=131
2025-12-23T15:38:18.766672Z  INFO node{moniker=test-3}:sync{height.tip=122}: Could not find a range to request up_to_height=131
2025-12-23T15:38:18.843897Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Received response from peer start=123 num_values=9 peer_id=16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36
2025-12-23T15:38:18.843913Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Updating score for peer 16Uiu2HAm9zLmAGdyMS2bSmGmBZhp1Q9oWQmKmJCjAYGgz3XhuU36
2025-12-23T15:38:18.843917Z DEBUG node{moniker=test-3}:sync{height.tip=122}:   Result = Success(132.477375ms)
2025-12-23T15:38:18.843919Z DEBUG node{moniker=test-3}:sync{height.tip=122}:     Prev = 0.5
2025-12-23T15:38:18.843934Z DEBUG node{moniker=test-3}:sync{height.tip=122}:      New = 0.6000000000000001
2025-12-23T15:38:18.843944Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received sync response for processing with 9 values start_height=123 request_id=1
2025-12-23T15:38:18.843949Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Processing value response certificate.height=123 signatures=3
2025-12-23T15:38:18.843952Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Processing certificate certificate.height=123 signatures=3
2025-12-23T15:38:18.844279Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=124
2025-12-23T15:38:18.844283Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=125
2025-12-23T15:38:18.844285Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=126
2025-12-23T15:38:18.844287Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=127
2025-12-23T15:38:18.844289Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=128
2025-12-23T15:38:18.844291Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=129
2025-12-23T15:38:18.844293Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=130
2025-12-23T15:38:18.844295Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Received value response for higher height, queuing for later consensus.height=123 certificate.height=131
2025-12-23T15:38:18.844305Z  INFO 🟢🟢 Processing synced value height=123 round=0
2025-12-23T15:38:18.851708Z DEBUG 💡 Sync block validated at height 123 with hash: 0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3
2025-12-23T15:38:18.851726Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Timeout elapsed step=Propose timeout.round=0 height=123 round=0
2025-12-23T15:38:18.851801Z DEBUG node{moniker=test-3}:wal{height=123}: Wrote log entry type=Timeout entry.size=10
2025-12-23T15:38:18.851818Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Transitioned to new step step.previous=Propose step.new=Prevote
2025-12-23T15:38:18.851825Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Voting vote_type=Prevote value=Nil round=0
2025-12-23T15:38:18.852020Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Received vote consensus.height=123 consensus.round=0 vote.height=123 vote.round=0 vote.msg=Prevote(height: 123, round: 0, value: Nil, from: E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87) validator=E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87
2025-12-23T15:38:18.852098Z DEBUG node{moniker=test-3}:wal{height=123}: Wrote log entry type=Consensus(Vote) entry.size=105
2025-12-23T15:38:18.859919Z DEBUG node{moniker=test-3}:wal{height=123}: Flushed WAL to disk wal.entries=2 wal.size=153
2025-12-23T15:38:18.860026Z DEBUG node{moniker=test-3}:network: Published message channel=Consensus size=96
2025-12-23T15:38:18.864086Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: We have a full proposal for this round, checking... proposal.height=123 proposal.round=0
2025-12-23T15:38:18.864093Z DEBUG node{moniker=test-3}:consensus{height=123 round=0}: Transitioned to new step step.previous=Prevote step.new=Commit
2025-12-23T15:38:18.864095Z  INFO node{moniker=test-3}:consensus{height=123 round=0}: Decided round=0 height=123 value=418583b14a23afeb
2025-12-23T15:38:18.867888Z DEBUG node{moniker=test-3}:wal{height=123}: Flushed WAL to disk wal.entries=2 wal.size=153
2025-12-23T15:38:18.867915Z DEBUG node{moniker=test-3}:sync{height.tip=122}: Consensus decided on new value height=123
2025-12-23T15:38:18.867921Z  INFO 🟢🟢 Consensus has decided on value height=123 round=0 value=418583b14a23afeb
2025-12-23T15:38:18.867942Z DEBUG 🎁 block size: 545, height: 123
2025-12-23T15:38:18.871975Z  INFO 👉 stats at height 123: #txs=0, txs/s=0.00, chain_bytes=67035, bytes/s=1014.68
2025-12-23T15:38:18.871979Z DEBUG 🦄 Block at height 123 contains 0 transactions
2025-12-23T15:38:18.873292Z DEBUG 💡 Block validated at height 123 with hash: 0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3
2025-12-23T15:38:18.873295Z DEBUG 🟠 set_latest_forkchoice_state: 0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3
2025-12-23T15:38:18.876267Z DEBUG ➡️ payload_status: PayloadStatus { status: Valid, latest_valid_hash: Some(0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3) }
2025-12-23T15:38:18.876270Z DEBUG 🚀 Forkchoice updated to height 123 for block hash=0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3 and latest_valid_hash=0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3
2025-12-23T15:38:18.876272Z  INFO Looking for certificate height=123 round=0
2025-12-23T15:38:18.879996Z  INFO Committed block_data[0..32]: a49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397
2025-12-23T15:38:18.887937Z DEBUG timeout_commit=500ms
2025-12-23T15:38:18.887940Z  INFO 👉 stats at Height(123): block_time 3.037067833s
2025-12-23T15:38:18.896050Z DEBUG 🌈 Got validator set: ValidatorSet { validators: [Validator { address: Address(5A9245DCE516AA85C8D82A90608A542A151D9E91), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([827532247923365, 2361805573119693, 3492543941731791, 1191675179907230, 237915650423024]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([4366659347211492, 2160934874953051, 581678635108779, 2650494975570035, 111524612954901]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(7D17AA4FE6C1E7C58D1B26F5A68C35BE0BFF6C29), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([1169858868288094, 147681501803464, 1111504367607205, 2144018211257096, 171110521269953]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([1744468881357823, 686784299215419, 1397329182505972, 713931089011640, 127275670879953]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(311E280D2918E93A90EEA22C0773053F325CE409), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([2820261586041093, 1505913081254658, 341150345958183, 3691244156032531, 54358481847654]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([1047127412350518, 3833800791933623, 3295542033967919, 4089411062058281, 57650081492668]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }, Validator { address: Address(E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87), public_key: PublicKey(VerifyingKey { inner: PublicKey { point: AffinePoint { x: FieldElement(FieldElementImpl { value: FieldElement5x52([27967812440236, 1160044011676679, 3854365601680896, 2804222629534947, 172467234474493]), magnitude: 1, normalized: true }), y: FieldElement(FieldElementImpl { value: FieldElement5x52([131342944875488, 1984349185449754, 1239639448107705, 2603967101376622, 211158001754920]), magnitude: 1, normalized: true }), infinity: 0 } } }), voting_power: 100 }] }
2025-12-23T15:38:18.900015Z DEBUG node{moniker=test-3}:wal{height=124}: Reset WAL height=124
2025-12-23T15:38:18.900040Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: No WAL entries to replay height=124
2025-12-23T15:38:18.900056Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Starting new height height=124
2025-12-23T15:38:18.900059Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Starting new round height=124 round=0 proposer=E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87 role=Proposer
2025-12-23T15:38:18.903952Z DEBUG node{moniker=test-3}:wal{height=124}: Flushed WAL to disk wal.entries=0 wal.size=12
2025-12-23T15:38:18.903974Z  INFO 🟢🟢 Started round height=124 round=0 proposer=E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87 role=Proposer
2025-12-23T15:38:18.903999Z DEBUG Found 0 pending proposal parts, validating... height=124 round=0
2025-12-23T15:38:18.904022Z DEBUG Found 0 undecided proposals height=124 round=0
2025-12-23T15:38:18.904032Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: Transitioned to new step step.previous=Unstarted step.new=Propose
2025-12-23T15:38:18.904036Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Scheduling timeout round=0 step=Propose
2025-12-23T15:38:18.904039Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Requesting value from application height=124 round=0
2025-12-23T15:38:18.904046Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: Replaying inputs count=1
2025-12-23T15:38:18.904048Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Processing value response certificate.height=124 signatures=3
2025-12-23T15:38:18.904050Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: Processing certificate certificate.height=124 signatures=3
2025-12-23T15:38:18.904459Z DEBUG node{moniker=test-3}:sync{height.tip=123}: Consensus started new height height=124 is_restart=false
2025-12-23T15:38:18.904466Z  INFO 🟢🟢 Consensus is requesting a value to propose height=124 round=0
2025-12-23T15:38:18.905417Z  INFO Building a new value to propose
2025-12-23T15:38:18.905420Z  INFO 🟠 current fork is Osaka
2025-12-23T15:38:18.905422Z DEBUG 🟠 generate_block on top of Some(ExecutionBlock { block_hash: 0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3, block_number: 123, parent_hash: 0xa49ba6b1d73203d54442531983c174f84fdc5a5029d897425f017867ef9af397, timestamp: 1766504283, prev_randao: 0x0000000000000000000000000000000000000000000000000000000000000000 })
2025-12-23T15:38:18.910149Z DEBUG 🌈 Got execution payload: ExecutionPayloadV3 { payload_inner: ExecutionPayloadV2 { payload_inner: ExecutionPayloadV1 { parent_hash: 0x05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3, fee_recipient: 0x4242424242424242424242424242424242424242, state_root: 0x1e7b3315163545f76083f2a6e1f16936dfb02e31c03806784914db75ad8c0e5c, receipts_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, prev_randao: 0x0000000000000000000000000000000000000000000000000000000000000000, block_number: 124, gas_limit: 53154085, gas_used: 0, timestamp: 1766504298, extra_data: 0x726574682f76312e392e332f6c696e7578, base_fee_per_gas: 68, block_hash: 0x3e681eb7ce9c5ce7b49c362c7cf4852753c8a8128dc4df6ca7452ea7648900e5, transactions: [] }, withdrawals: [] }, blob_gas_used: 0, excess_blob_gas: 0 }
2025-12-23T15:38:18.910170Z DEBUG 🎁 block size: 545, height: 124
2025-12-23T15:38:18.918080Z DEBUG Streaming proposal part: StreamMessage { stream_id: StreamId(b"\0\0\0\0\0\0\0|\0\0\0\0\0\0\0\0"), sequence: 0, content: Data(Init(ProposalInit { height: Height(124), round: Some(0), pol_round: Nil, proposer: Address(E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87) })) } height=124 round=0
2025-12-23T15:38:18.918093Z DEBUG Streaming proposal part: StreamMessage { stream_id: StreamId(b"\0\0\0\0\0\0\0|\0\0\0\0\0\0\0\0"), sequence: 1, content: Data(Data(ProposalData { bytes: "<...>", len: 545 })) } height=124 round=0
2025-12-23T15:38:18.918106Z DEBUG Streaming proposal part: StreamMessage { stream_id: StreamId(b"\0\0\0\0\0\0\0|\0\0\0\0\0\0\0\0"), sequence: 2, content: Data(Fin(ProposalFin { signature: Signature(ecdsa::Signature<Secp256k1>(0EB8D318D31B8B2129F0C336F52D0883ECCC09E0610D206267898210FD240765001B79684C8B91F6283E78F6D303C960762F287D6A6E3530A39463428F0ECDD8)) })) } height=124 round=0
2025-12-23T15:38:18.918250Z DEBUG node{moniker=test-3}:network: Published message channel=ProposalParts size=48
2025-12-23T15:38:18.918270Z DEBUG Streaming proposal part: StreamMessage { stream_id: StreamId(b"\0\0\0\0\0\0\0|\0\0\0\0\0\0\0\0"), sequence: 3, content: Fin } height=124 round=0
2025-12-23T15:38:18.918414Z DEBUG node{moniker=test-3}:wal{height=124}: Wrote log entry type=LocallyProposedValue entry.size=596
2025-12-23T15:38:18.918487Z DEBUG node{moniker=test-3}:network: Published message channel=ProposalParts size=574
2025-12-23T15:38:18.918579Z DEBUG ✅ Proposal sent height=124 round=0
2025-12-23T15:38:18.918585Z  INFO 🟢🟢 Processing synced value height=124 round=1
2025-12-23T15:38:18.918755Z DEBUG node{moniker=test-3}:network: Published message channel=ProposalParts size=92
2025-12-23T15:38:18.918921Z DEBUG node{moniker=test-3}:network: Published message channel=ProposalParts size=22
2025-12-23T15:38:18.919045Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Proposing value id=89e9fe612efb2d15 height=124 round=0
2025-12-23T15:38:18.919271Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Received proposal consensus.height=124 consensus.round=0 proposal.height=124 proposal.round=0 proposal.msg=Proposal(height: 124, round: 0, pol_round: -1, value: 89e9fe612efb2d15, from: E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87) proposer=E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87
2025-12-23T15:38:18.919282Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: Transitioned to new step step.previous=Propose step.new=Prevote
2025-12-23T15:38:18.919285Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Voting vote_type=Prevote value=89e9fe612efb2d15 round=0
2025-12-23T15:38:18.919498Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Received vote consensus.height=124 consensus.round=0 vote.height=124 vote.round=0 vote.msg=Prevote(height: 124, round: 0, value: 89e9fe612efb2d15, from: E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87) validator=E95EAA9DCD4F9E3B4EEC820355C03B4F4499AB87
2025-12-23T15:38:18.919538Z DEBUG node{moniker=test-3}:wal{height=124}: Wrote log entry type=Consensus(Vote) entry.size=117
2025-12-23T15:38:18.924026Z DEBUG node{moniker=test-3}:wal{height=124}: Flushed WAL to disk wal.entries=2 wal.size=751
2025-12-23T15:38:18.924139Z DEBUG node{moniker=test-3}:network: Published message channel=Consensus size=108
2025-12-23T15:38:18.927181Z DEBUG 💡 Sync block validated at height 124 with hash: 0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0
2025-12-23T15:38:18.936110Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: We have a full proposal for this round, checking... proposal.height=124 proposal.round=1
2025-12-23T15:38:18.936117Z DEBUG node{moniker=test-3}:consensus{height=124 round=0}: Transitioned to new step step.previous=Prevote step.new=Commit
2025-12-23T15:38:18.936120Z  INFO node{moniker=test-3}:consensus{height=124 round=0}: Decided round=0 height=124 value=fcec2b04c0a4dee3
2025-12-23T15:38:18.939939Z DEBUG node{moniker=test-3}:wal{height=124}: Flushed WAL to disk wal.entries=2 wal.size=751
2025-12-23T15:38:18.939969Z DEBUG node{moniker=test-3}:sync{height.tip=123}: Consensus decided on new value height=124
2025-12-23T15:38:18.939976Z  INFO 🟢🟢 Consensus has decided on value height=124 round=1 value=fcec2b04c0a4dee3
2025-12-23T15:38:18.939999Z DEBUG 🎁 block size: 545, height: 124
2025-12-23T15:38:18.943993Z  INFO 👉 stats at height 124: #txs=0, txs/s=0.00, chain_bytes=67580, bytes/s=1021.81
2025-12-23T15:38:18.943999Z DEBUG 🦄 Block at height 124 contains 0 transactions
2025-12-23T15:38:18.945602Z DEBUG 💡 Block validated at height 124 with hash: 0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0
2025-12-23T15:38:18.945606Z DEBUG 🟠 set_latest_forkchoice_state: 0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0
2025-12-23T15:38:18.946416Z DEBUG ➡️ payload_status: PayloadStatus { status: Valid, latest_valid_hash: Some(0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0) }
2025-12-23T15:38:18.946420Z DEBUG 🚀 Forkchoice updated to height 124 for block hash=0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0 and latest_valid_hash=0x1d48e0392248e0bd553b7adbea929a0c0f1f60d863c88d8b331f1c9072d12ab0
2025-12-23T15:38:18.946423Z  INFO Looking for certificate height=124 round=1
2025-12-23T15:38:18.951003Z  INFO Committed block_data[0..32]: 05f77b3fa89ea85af4468854bb4a4782bc7d5938e1b517deeb7e849b3821b4b3
2025-12-23T15:38:18.960995Z DEBUG timeout_commit=500ms
2025-12-23T15:38:18.960999Z  INFO 👉 stats at Height(124): block_time 57.020791ms

@bastienfaivre bastienfaivre marked this pull request as ready for review December 23, 2025 15:48
@bastienfaivre bastienfaivre requested a review from a team as a code owner December 23, 2025 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jmalicevic jmalicevic Awaiting requested review from jmalicevic

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(state): Support height replay on crash

3 participants

@bastienfaivre @jmalicevic