Skip to content

feat(ol): [1/2] detect checkpoints mined on L1 but rejected by the ASM (STR-4084) - #2369

Open
storopoli wants to merge 5 commits into
mainfrom
t3code/detect-rejected-checkpoints
Open

storopoli wants to merge 5 commits into
mainfrom
t3code/detect-rejected-checkpoints

Conversation

@storopoli

@storopoli storopoli commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Description

The ASM only logs a tracing warning when it rejects a checkpoint. It emits no ASM log, so the node keeps no record of the rejection. Around a predicate rotation, a checkpoint proven with the wrong artifact can be mined and rejected while the broadcaster marks it Finalized.

This PR adds a sequencer-only tracker in strata-ol-checkpoint (submission_tracker/). There is no automatic recovery. On each CheckpointState the CSM publishes, and on a 30 s tick, the tracker scans the L1 writer's checkpoint bundles from a persisted cursor and resolves each reveal through the broadcaster, following RBF replacements. A checkpoint counts as confirmed-but-rejected once its canonical inclusion block is btcio.l1_reorg_safe_depth deep under the CSM tip and the ASM's verified epoch is still below the checkpoint's epoch. The CSM already uses that depth for checkpoint finality, so there is no new config.

Each rejection goes into a new ol_checkpoint table (CBOR, keyed by txid), committed together with the cursor, then logged once at warn (checkpoint confirmed on L1 but rejected by ASM) and counted in strata_checkpoint_confirmed_rejected_total. The strata_checkpoint_unresolved_rejections gauge counts rejections whose epoch is still unaccepted. A failed check retries on the next tick and warns once, with repeats at debug and the recovery at info; an undecodable bundle also warns once, with the reason.

#2370 adds the RPC and the functional test.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor
  • New or updated tests
  • Dependency Update

Notes to Reviewers

  • The ASM only accepts verified.epoch + 1, and depth is measured from the CSM tip, so the ASM has processed the inclusion block before the tracker decides. A rejected posting isn't reported if another posting gets its epoch accepted first.
  • The tracker reads writer bundles rather than checkpoint signing entries, because startup reconciliation deletes signing entries for unaccepted epochs. Bundles are append-only. The cursor passes a bundle only once it is settled: not a checkpoint, accepted and final, or recorded as rejected.
  • A restart neither loses nor repeats an alert. A crash before sled flushes can repeat one warning. The first start scans existing bundles once.
  • Out of scope: reorgs deeper than the reorg-safe depth, writer, broadcaster or prover changes, and reconcile_unaccepted_checkpoint_artifacts.
  • No V1 guest crate changes, and provers/sp1/guest-checkpoint/Cargo.lock is unchanged. strata-ol-checkpoint, used only by bin/strata, gains strata-btcio, strata-csm-types, strata-db-types, strata-codec, strata-codec-utils and strata-asm-proto-checkpoint-txs.

Is this PR addressing any specification, design doc or external reference document?

  • Yes
  • No

Checklist

  • I have performed a self-review of my code.
  • I have commented my code where necessary.
  • I have updated the documentation if needed.
  • My changes do not introduce new warnings.
  • I have added (where necessary) tests that prove my changes are effective or that my feature works.
  • New and existing tests pass with my changes.
  • I have disclosed my use of AI in the body of this PR.

Testing

Ran just fmt-ws, just lint-check-ws, the TOML, codespell, style and TODO lints, and cargo nextest on the touched crates. Unit tests cover the classifier and the worker against an in-memory context: repeated observations, a reorg before depth, an acceptance undone by a shallow reorg, restarts, an out-of-order rebuild, and an unmined bundle holding the cursor. Others cover warn-once logging, the sled table, and building the timer input outside the tokio runtime. CI passes, including functional tests. The end-to-end test is in #2370.

AI use

Claude Code wrote the implementation, the tests and this description. OpenAI Codex reviewed the plan and the diff (read-only).

Related Issues

STR-4084

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T14:48:45.584463Z a4bf8e1 New commits
🔒 Security Review ✅ Completed 2026-10-19T14:49:14.938955Z a4bf8e1 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 1f48122

SP1 Execution Results

program cycles gas
Checkpoint 2,801,343 3,230,567

@storopoli
storopoli added this pull request to stack #2371 October 5, 2026 15:03
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.95082% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.32%. Comparing base (508f3c3) to head (a4bf8e1).

Files with missing lines Patch % Lines
...es/ol/checkpoint/src/submission_tracker/context.rs 89.77% 9 Missing ⚠️
...ates/ol/checkpoint/src/submission_tracker/state.rs 98.88% 5 Missing ⚠️
bin/strata/src/services.rs 92.85% 1 Missing ⚠️
@@            Coverage Diff             @@
##             main    #2369      +/-   ##
==========================================
+ Coverage   89.21%   89.32%   +0.10%     
==========================================
  Files         440      445       +5     
  Lines       75934    76666     +732     
==========================================
+ Hits        67747    68481     +734     
+ Misses       8187     8185       -2     
Flag Coverage Δ
functional 59.16% <93.26%> (+0.44%) ⬆️
unit 81.93% <76.09%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
crates/db/store-sled/src/ol_checkpoint/db.rs 90.87% <100.00%> (+0.48%) ⬆️
crates/db/store-sled/src/ol_checkpoint/schemas.rs 100.00% <ø> (ø)
crates/db/types/src/ol_checkpoint.rs 100.00% <100.00%> (ø)
...s/ol/checkpoint/src/submission_tracker/classify.rs 100.00% <100.00%> (ø)
crates/ol/checkpoint/src/submission_tracker/mod.rs 100.00% <100.00%> (ø)
crates/storage/src/managers/ol_checkpoint.rs 61.41% <100.00%> (+1.52%) ⬆️
crates/storage/src/managers/writer.rs 100.00% <100.00%> (ø)
bin/strata/src/services.rs 97.11% <92.85%> (-0.23%) ⬇️
...ates/ol/checkpoint/src/submission_tracker/state.rs 98.88% <98.88%> (ø)
...es/ol/checkpoint/src/submission_tracker/context.rs 89.77% <89.77%> (ø)

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@krsnapaudel

krsnapaudel commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

@storopoli I see that you are adding checkpoint submission tracker here. I like the idea, but in the PR you do this for detecting checkpoints rejected by ASM. I feel the submission tracker could track checkpoint in the submission/publication path in a sequencer. Another observation tracker could take over after ASM emits a log: basically what CSM does now. Both could be part of one OL checkpoint tracker. I have written a rough draft. We can discuss the ideas in tomorrow's sync. We should also discuss whether ASM should emit a log for rejection. Otherwise detection of rejection is indirect.

@storopoli

Copy link
Copy Markdown
Member Author

Thanks, I read the draft. I agree with splitting this into submission tracking on the sequencer and observation on every node, with checkpoint status kept in the OL checkpoint DB.

STR-4084 is detection only, so this PR is narrower. It reads the writer's bundles and the broadcaster's tx status and records rejections of this sequencer's own checkpoints, without changing how the writer, broadcaster or CSM behave. It doesn't catch everything: if another transaction gets the epoch accepted before the rejected one is buried, the rejection isn't recorded.

The ASM does log a tracing warning on rejection, but there's no rejection AsmLogEntry. I'll fix the PR text that says it emits nothing. A rejection log would be the direct signal, but it changes what manifests commit to, so it needs an ASM upgrade, and it should only cover transactions that pass the sequencer-key check so outsiders can't add entries to manifests. The ASM worker could also report rejections to the node outside the STF, which needs no consensus change.

@storopoli
storopoli requested a review from purusang as a code owner October 14, 2026 12:18
@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a6d7fc0299

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ol/checkpoint/src/submission_tracker/context.rs
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 75c4c85
Baseline: #2408 (508f3c3)
SP1 Execution Results

program cycles gas Δ cycles Δ gas
Checkpoint 2,782,155 3,212,830 0 (0.0%) 0 (0.0%)

@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

The ASM emits nothing when it rejects a checkpoint. Around a predicate
rotation, a checkpoint proven with the wrong artifact can be mined and
rejected while the broadcaster marks it finalized, and nothing reports it.

Add a sequencer-side submission tracker to strata-ol-checkpoint. It follows
the checkpoint state the CSM publishes, scans the L1 writer's checkpoint
bundles from a persisted cursor, and resolves each reveal through the
broadcaster's replacement chain. A checkpoint is recorded as
confirmed-but-rejected once its inclusion block is buried at
`l1_reorg_safe_depth` under the CSM tip while the ASM's verified epoch is
still below the checkpoint's epoch. Rejections are committed together with
the cursor, then logged at warn once and counted in
`strata_checkpoint_confirmed_rejected_total`. The
`strata_checkpoint_confirmed_rejected` gauge counts recorded rejections whose
epoch the ASM has still not accepted.

Writer bundles survive the startup artifact reconciliation, so a restart
resumes from the cursor without losing or re-alerting a submission. Detection
only: writer, broadcaster and prover state are untouched.

Refs: STR-4084
A failed check used to warn on every 30 s retry for as long as the failure
lasted. The tracker state now remembers the last failure: the first one is
logged at warn, repeats of the same failure at debug, and the recovery once
at info. Retries still run on the next update or tick.

An undecodable checkpoint bundle had the same problem while an earlier
unsettled bundle held the cursor. The context now reports it as
`BundleContents::Undecodable` instead of logging, and the state warns once
per bundle.

Refs: STR-4084
The submission tracker's module doc said the ASM emits nothing on rejection. At v0.4.0-rc.6 the checkpoint handler logs a tracing warning but emits no ASM log, so the node still keeps no record of the rejection.
The unresolved-rejections gauge was only set at the end of a completed scan, so it kept a stale
value while a check deferred or failed. It is now set after every check and renamed to
strata_checkpoint_unresolved_rejections, so it no longer shares the counter's name for a different
quantity. A deferred check (no genesis, or a CSM tip the node no longer considers canonical) no
longer counts as recovering from a failure, which broke warn-once logging.

Also:
- ConfirmedRejected carries the mined reveal, so the verdict match is exhaustive.
- SubmissionTrackerState::new, which reads storage, is now load.
- Context methods that read storage have get_ names.
- An undecodable checkpoint bundle's warning says why it does not decode.
Reverting the L1 canonical chain purges the height cache before deleting the rows, so the submission tracker's canonical lookup can briefly see a reverted block during a reorg. It cannot produce a false rejection record short of a reorg as deep as the reorg-safe depth; the fix belongs in L1BlockManager.
@storopoli
storopoli force-pushed the t3code/detect-rejected-checkpoints branch from 2480ffd to a4bf8e1 Compare October 19, 2026 14:45
@github-actions

Copy link
Copy Markdown
Contributor

🔒 AI Security Review (claude-opus-5-5)

✅ No security issues found.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants