feat: add migrate batched upload with resume - #656
Draft
SgtPooki wants to merge 8 commits into
Draft
Conversation
Fourth slice of the migrate command split (#652). Drives packed pieces through store, SP-to-SP pull to secondaries, and batched addPieces commits (up to the SDK batch cap), persisting every transition to migrate.db. Commits whose outcome is unknown after a crash are reconciled on resume from the PiecesAdded event, or via findPieceIdsByCid when no tx hash was captured, so a blind on-chain re-add is never issued. pdp-verifier wraps the read-only chain lookups the reconciliation needs (piece id by CID, tx receipt status, PiecesAdded event extraction).
FilOzzy
added
team/filecoin-pin
An add_unconfirmed row whose tx had no receipt fell straight through to the presence check and re-parked, so the next flush could issue a second addPieces while the original tx was still in the mempool. The same age gate the hashless path uses now holds the row until the breadcrumb outlives any realistic confirmation window (the constant is renamed UNCONFIRMED_REQUEUE_AFTER_MS to cover both paths). txLanded returned false on any RPC error, so one 5xx or rate limit looked like a dead transaction and fed that re-queue. Only viem's TransactionReceiptNotFoundError maps to false now; other errors abort the pass. fetchAddPiecesEvent no longer requires a known data set id: a crash on the first commit of a new data set records none, and the event's own setId is the recovery path. A successful commit writes the resolved id back onto the context so later parks carry it. hasPiece treats only the SP's not-found answer as absence, and data set ids stay strings end to end instead of passing through Number.
Follows the repo convention: human-facing progress goes through src/utils/cli-logger.ts, not a migrate-private stderr logger.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Part 4 of 5 splitting #652. Drives packed pieces through store on the primary, SP-to-SP pull to secondaries, and batched
addPiecescommits, persisting every transition tomigrate.db. Resume is receipt-first: a commit whose outcome is unknown after a crash resolves from thePiecesAddedevent, or viafindPieceIdsByCidwhen no tx hash was captured.. a blind on-chain re-add is never issued.pdp-verifier.tswraps the read-only chain lookups reconciliation needs.This intentionally does not reuse
executeUpload: the SDK commits one piece per call, and migrate needs store-many-commit-once batching plus durable per-(piece,provider) state across restarts. Known gaps vsexecuteUploadto resolve before undraft of the stack: norecordUploadResulttelemetry and no IPNI validation.How to verify
migrate-direct-upload-flowcovers park/flush/reconcile against a fake context.Notes / risks
Base:
feat/migrate-3-pack. Merge order 1 through 5.