feat: add migrate command and pipeline - #657
Draft
SgtPooki wants to merge 8 commits into
Draft
Conversation
Final slice of the migrate command split (#652). run-migrate orchestrates download, pack, upload, and eviction as one pipeline clocked by a staging-disk byte budget: committed pieces evict and free budget so a disk smaller than the migration cycles instead of failing. On startup the staging dirs are swept against migrate.db, re-verifying staged files by recorded hash. Wires up 'filecoin-pin migrate <cid-list-file>' in the CLI, exports getDataDirectory for the per-network state path, and adds the pipeline docs. Closes #651.
FilOzzy
added
team/filecoin-pin
The run could exit 0 with CIDs that never reached chain: per-bin pack failures kept members 'done' outside any recorded piece and nothing counted them. The summary now reports such CIDs as 'unpacked' and migrateIncomplete treats them as incomplete, with a pipeline test covering the pack-failure path. EGRESS_PROVIDER=beam exported for add/import silently defeated the migrate 'none' default through Commander's env binding, opting a bulk run into CDN lockup. Migrate registers its own option without env binding and its help states the 'none' default. A consumer failure now fails the staging budget and both pipeline sides settle before runMigrate returns, so download workers stop instead of racing the caller's db.close(). A member whose own bytes exceed the budget's download ceiling fails terminally instead of retrying forever. --copies validates through parsePositiveInt. Docs: glossary entries for member CAR, parked piece, GC window, and staging budget; migrate.md states the sha2-only hasher scope.
SgtPooki
added a commit
that referenced
this pull request
Aug 26, 2026
documentation/migrate.md and the migrate glossary entries move from the final slice (#657) to the first so the pipeline design, resume guarantees, and exit contract get agreement before the code that implements them is reviewed. The doc describes the completed stack; flags it mentions land in later slices.
Pipeline progress goes through src/utils/cli-logger.ts like every other command, and the raw JSON dump on stdout is replaced by a summary block in the data-set display style: counts per stage and provider, with problem CIDs (over-cap, downloaded-but-unpacked) listed by name. filecoin-pin has no piped-output convention to honor; migrate.db holds the full record and a --json output mode is a possible follow-up for scripted consumption.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Part 5 of 5 splitting #652: this is the slice that turns the command on.
run-migrate.tsorchestrates download, pack, upload, and eviction as one pipeline clocked by a staging-disk byte budget: committed pieces evict and free budget, so a disk smaller than the migration cycles instead of failing (--max-staged-bytescaps it explicitly). Startup sweeps the staging dirs againstmigrate.db, re-verifying staged files by recorded hash. Wiresfilecoin-pin migrate <cid-list-file>into the CLI and exportsgetDataDirectory(the design doc itself rides in #653).Egress defaults to
none. Exit 0 only when every CID is committed at the requested copies; anything short exits 1 and re-running is safe.Refs #651.
How to verify
Notes / risks
Base:
feat/migrate-4-upload. Merge order 1 through 5. The calibnet kill-and-resume validation noted on #652 is still pending before undraft. Concurrent runs against onemigrate.dbare unsupported.