Skip to content

fix(prover): preserve checkpoint work across restarts (STR-4562) - #2390

Merged
storopoli merged 11 commits into
mainfrom
fix/str-4562-preserve-checkpoint-work
Oct 17, 2026
Merged

storopoli merged 11 commits into
mainfrom
fix/str-4562-preserve-checkpoint-work

Conversation

@purusang

@purusang purusang commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Description

Restarting a prover-enabled node deleted checkpoint payloads, proofs, and tasks beyond ASM's verified tip, including saved remote request IDs and signing records. Remove that startup cleanup so completed proof work is reused and unfinished tasks remain available to the existing recovery path.

Document the retention assumptions and add coverage for unaccepted checkpoints surviving restart, failed artifact validation preserving stored work, and task metadata surviving database reopening.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor
  • New or updated tests
  • Dependency Update

Notes to Reviewers

OL only incorporates buried ASM manifests, and protocol upgrades assign rules per epoch without invalidating completed epochs. ASM acceptance lag is therefore not a reason to delete proof work. Existing artifact validation, spec-scoped recovery, and epoch commitment/spec checks remain in place. Deep-reorg recovery and writer-intent cancellation are outside this change.

Validation:

  • The new restart regression fails against the old cleanup and passes with this change.
  • ./run_tests.sh -t test_checkpoint_restart test_checkpoint_artifact_availability passed.
  • just test-unit: 2,468 passed, 2 skipped.
  • just lint-check-ws passed.
  • just pr-lite passed in a clean checkout of this exact patch, including all lint, documentation, doctest, and unit-test stages (2,468 passed, 2 skipped). The clean checkout excludes unrelated ignored local investigation files.

Remote-request preservation is tested through persisted task metadata; these tests do not submit live remote proof jobs.

Is this PR addressing any specification, design doc or external reference document?

  • Yes
  • No

Relevant link: STR-4562. The implementation follows the clarified scope: preserve work across restart without introducing automatic deletion for upgrades or supported shallow reorgs.

AI assistance: Codex authored the implementation, documentation, and tests, performed reviews, ran validation, and drafted this PR body with human direction and review.

Checklist

  • I have performed a self-review of my code.
  • I have commented my code where necessary.
  • I have updated the documentation if needed.
  • My changes do not introduce new warnings.
  • I have added (where necessary) tests that prove my changes are effective or that my feature works.
  • New and existing tests pass with my changes.
  • I have disclosed my use of AI in the body of this PR.

Related Issues

Closes STR-4562

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Commit: 5cb0c1c

SP1 Execution Results

program cycles gas
Checkpoint 2,782,155 3,212,830

@purusang
purusang marked this pull request as ready for review October 9, 2026 06:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T09:54:42.971317Z 2fb8a96 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

@purusang

purusang commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e52eb8a3fb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bin/strata/src/services.rs

@voidash voidash left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but we might want to delete the dead code related to storage manager

Comment thread bin/strata/src/checkpoint_reconcile.rs
Comment thread bin/strata/src/checkpoint_reconcile.rs
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.18241% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.21%. Comparing base (2435e3d) to head (2fb8a96).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/chain-worker/src/handle.rs 50.00% 6 Missing ⚠️
bin/strata/src/sequencer/block_producer.rs 33.33% 4 Missing ⚠️
crates/chain-worker/src/service.rs 50.00% 4 Missing ⚠️
crates/chain-worker/src/state.rs 90.69% 4 Missing ⚠️
crates/chain-worker/src/context.rs 50.00% 3 Missing ⚠️
bin/strata/src/fcm.rs 50.00% 2 Missing ⚠️
crates/consensus-logic/src/fcm/service.rs 99.51% 1 Missing ⚠️
@@            Coverage Diff             @@
##             main    #2390      +/-   ##
==========================================
+ Coverage   89.14%   89.21%   +0.06%     
==========================================
  Files         441      440       -1     
  Lines       75737    75930     +193     
==========================================
+ Hits        67515    67740     +225     
+ Misses       8222     8190      -32     
Flag Coverage Δ
functional 58.71% <56.19%> (-0.22%) ⬇️
unit 81.98% <85.34%> (+0.24%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
bin/strata/src/main.rs 90.54% <ø> (ø)
bin/strata/src/prover/mod.rs 81.96% <ø> (ø)
bin/strata/src/run_context.rs 99.09% <ø> (ø)
bin/strata/src/services.rs 97.33% <ø> (-0.02%) ⬇️
crates/consensus-logic/src/fcm/state.rs 92.56% <100.00%> (+0.17%) ⬆️
crates/db/store-sled/src/ol_checkpoint/db.rs 90.39% <ø> (-0.25%) ⬇️
crates/ol/checkpoint/src/service.rs 73.07% <ø> (-11.54%) ⬇️
crates/storage/src/managers/checkpoint_proof.rs 100.00% <ø> (ø)
crates/storage/src/managers/ol_checkpoint.rs 59.88% <ø> (+0.53%) ⬆️
crates/storage/src/managers/prover_task.rs 90.90% <ø> (-0.30%) ⬇️
... and 8 more

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@storopoli storopoli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The P1 is real and we should address them before anything else. This is a good start but it's not ready to merge yet.

Comment thread crates/ol/checkpoint-artifacts/README.md
Comment thread crates/ol/checkpoint-artifacts/README.md Outdated
Comment thread crates/ol/checkpoint-artifacts/README.md
Comment thread bin/strata/src/prover/mod.rs
Comment thread bin/strata/src/services.rs
Comment thread bin/strata/src/checkpoint_reconcile.rs
Comment thread bin/strata/src/checkpoint_reconcile.rs
Remove startup cleanup of local checkpoint payloads, proofs, and tasks beyond ASM's verified tip. Preserve saved remote requests and existing artifact and epoch-spec validation.

Cover unaccepted checkpoint restart retention, non-destructive artifact-check failures, and persisted task recovery metadata.

Refs: STR-4562
Signal BlockAccepted after FCM finishes block processing and persists Valid. Let chain worker inspect the terminal flag, complete epoch processing, and store and notify the summary.

Keep summary creation failures outside the block rejection path. Cover acceptance ordering, rejected blocks, and ordinary blocks in regression tests.

Refs: STR-4562
@purusang
purusang force-pushed the fix/str-4562-preserve-checkpoint-work branch from e52eb8a to f9ffffe Compare October 15, 2026 09:34
@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 11caae3
Baseline: #2396 (cc5f12f)
SP1 Execution Results

program cycles gas Δ cycles Δ gas
Checkpoint 2,782,155 3,212,830 0 (0.0%) 0 (0.0%)

@purusang
purusang requested a review from storopoli October 15, 2026 09:48
@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

@krsnapaudel krsnapaudel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Good cleanup for epoch summary creation path. We could revisit FCM setting status multiple times later if needed.

@storopoli storopoli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 2fb8a96

@storopoli
storopoli added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit c40542e Oct 17, 2026
31 checks passed
@storopoli
storopoli deleted the fix/str-4562-preserve-checkpoint-work branch October 17, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants