feat: add migrate command - #652
Draft
SgtPooki wants to merge 8 commits into
Draft
Conversation
Port ipfs2foc's direct-upload flow into filecoin-pin as `filecoin-pin migrate <cid-list-file>`. The command fetches each CID as a canonical block-verified CAR from a trustless gateway, computes its PieceCID, packs CIDs into multi-root CAR pieces under --pack-target-size, and uploads them with GC-aware batched addPieces commits. State lives in a per-network migrate.db (node:sqlite) under the platform data directory, so interrupted runs resume without re-adding pieces on chain. Streaming mode (default) uploads packed pieces while later CIDs are still downloading; --mode staged packs everything first. Egress defaults to none for bulk archival. A migration manifest (source CID to piece CID, data set ids, tx hashes) is imported through the UnixFS path and committed as a final piece tagged migrationManifest. Port source: FilOzone/foc-migrate@e52218a. refs #651
FilOzzy
added
team/filecoin-pin
Explain how migrate works for people running bulk migrations: the four-stage pipeline (download and verify, pack, upload, commit and evict), the disk-budget flow control, the GC-window commit batching, resume semantics, data-set separation, and the exit-code contract. Links key terms to the glossary. refs #651
Replace the two-pass architecture (commP pass, then a pack stage that re-downloaded every member) with one pipeline matching documentation/migrate.md: each CID downloads once into a verified member file on disk (every block hash-checked, DAG walked for completeness, commP computed in the same pass), pieces assemble from local files, and uploads, batched commits, and eviction run concurrently. A staging byte budget (statfs-derived, capped by --max-staged-bytes) gates downloads, so disk usage is hard-bounded and a disk smaller than the migration cycles instead of failing. Also closes the review findings: streaming multi-root assembly (constant memory), store() result verified against the expected PieceCID, missing secondary copies repaired on resume, reconcile uses the row's own data set id, revert clears stale tx hashes, eviction requires only the committed primary (secondary retries pull provider-to-provider), add_unconfirmed counts as incomplete, state scoped per network and owner address, and CIDv0/v1 aliases dedupe at ingest. The canonical re-serialization stack, streaming/staged modes, and the manifest are removed; the LIVE_TESTS gate is replaced by hermetic truncation and corruption fixtures. refs #651
Resume correctness: packed members clear their member-file columns in the same transaction that records the piece, so a resumed run no longer re-downloads already-packed CIDs; staged pieces re-hash on startup and a corrupt one is deleted with its source CIDs re-queued; an add_unconfirmed row that never captured its transaction hash now resolves against the data set on chain (findPieceIdsByCid) instead of being blindly re-parked or stuck forever. Verification: the DAG completeness walk starts from the requested root, not the first declared root, so a response with a complete decoy root cannot smuggle an incomplete requested DAG through. Pipeline liveness and accounting: a commP-mismatched piece drops out of the upload queue via its failed row instead of re-storing in a loop; over-cap CIDs are marked oversized, leave the free pool, and return their staged bytes; a pack-stage failure fails the budget waiters and drains the uploader instead of hanging both sides; the uploader re-checks for work in the same synchronous block that parks it, closing the lost-wakeup window; failed assemblies wake budget waiters when refunding bytes; budget is returned only for files actually removed from disk. refs #651
The corrupt-CAR rebuild path in the startup sweep could delete an add_unconfirmed breadcrumb before reconciliation resolved it against the chain, re-queueing source CIDs whose commit may have landed (a duplicate-add hazard). A rebuild now refuses while any upload row is parked or add_unconfirmed, or the primary is committed; the piece keeps its rows and budget bytes until a later run finds it rebuild-safe. refs #651
An add_unconfirmed row with no transaction hash that is absent from its data set could still have a transaction in flight; re-queueing it immediately risks adding the piece twice. Such a row now re-enters the flow only once the breadcrumb is older than an hour (any real transaction has landed or died by then); younger rows stay unresolved and the run exits incomplete, resolving automatically on a later re-run. refs #651
The migrate modules carried their own byte formatter with the same KiB/MiB/GiB semantics as utils/cli-helpers formatFileSize; the shared helper formats all migrate byte counts now, and metrics.ts shrinks to the duration formatter and Timer. refs #651
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Superseded by a stacked split for review. This PR stays open as the umbrella: full pipeline description below, plus the pending calibnet validation. Review and merge the stack in order instead of this branch:
@ipld/dag-cborshipped as a devDependency)Each PR's base is the previous branch; the command activates in #657. Close this PR once #657 merges.
What changed
Adds
filecoin-pin migrate <cid-list-file>: bulk-migrate IPFS CIDs onto FOC in batched pieces. Refs #651. Ported from FilOzone/foc-migrate@e52218a, then reworked into the pipeline documented in documentation/migrate.md after three rounds of multi-agent review.One pipeline, clocked by a disk budget: each CID downloads once into a verified member CAR on disk (every block hash-checked, DAG walked for completeness from the requested root, CommP computed in the same pass), members bin-pack into ~1000MiB multi-root pieces assembled disk-to-disk in constant memory, and uploads run concurrently with GC-aware batched
addPiecescommits (cap 40). Committed pieces evict and free budget, so a disk smaller than the migration cycles instead of failing;--max-staged-bytescaps it explicitly.Resume is receipt-first from a per-network, per-owner
migrate.db: unknown-outcome commits resolve via thePiecesAddedevent (orfindPieceIdsByCidwhen no tx hash was captured), staged files re-verify against recorded hashes on startup, and a blind on-chain re-add is never issued. Egress defaults tonone.How to verify
A calibnet validation run (kill mid-run, resume, confirm no duplicate adds) is still pending before undraft.
Notes / risks
node:sqlite(DatabaseSync, WAL) is unflagged since Node 23.4; thelts/*CI lane covers it. Noengineschange.migrate.dbare unsupported.@ipld/dag-pb(already in the tree transitively).0only when every CID is committed at the requested copies; anything short (failed, over-cap, unresolved commits, missing copies) exits1and re-running is safe.