feat: add migrate gateway fetch and CAR verify - #654
Draft
SgtPooki wants to merge 8 commits into
Draft
Conversation
Second slice of the migrate command split (#652). Fetches each CID as a CAR in one trustless-gateway request (format=car, dag-scope=all) and verifies the untrusted response in a single pass: every block is hash-checked, the DAG is walked from the requested root for completeness, and the piece commitment plus a sha256 are computed over the exact bytes landed on disk. Failures map to the resume store's failure categories. Also moves @ipld/dag-cbor from devDependencies to dependencies: verify-car imports it at runtime, so published installs would have failed on any dag-cbor block.
FilOzzy
added
team/filecoin-pin
The completeness walk keyed block presence by multihash alone while links were decoded with the codec the CAR entry declared. A gateway could serve the root bytes under a raw CID with the same multihash, omit every child, and pass verification. The walk now records the arrival codec and rejects any block reached under a different one. Root matching compares multihashes instead of full CIDs: the trustless-gateway spec permits answering a CIDv0 request with the equivalent CIDv1 root, which strict CID equality rejected. Failure paths now destroy the file sink and cancel the response body before unlinking (an open handle blocks deletion on Windows and the fixed temp path then poisons every fallback gateway), buffered sink writes resolve immediately instead of waiting out each flush, and a failed block hash reports car_block_mismatch instead of car_root_mismatch.
Follows the repo convention: human-facing progress goes through src/utils/cli-logger.ts, not a migrate-private stderr logger.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Part 2 of 5 splitting #652. Fetches each CID as a CAR in one trustless-gateway request (
?format=car&dag-scope=all) and verifies the untrusted response in a single pass: every block hash-checked, DAG walked from the requested root for completeness (a truncated response parses cleanly but fails the walk), piece commitment and sha256 computed over the exact bytes landed on disk via temp file + atomic rename. Transport failures map to the resume store's failure categories.Also moves
@ipld/dag-cborfrom devDependencies to dependencies:verify-car.tsimports it at runtime, so published installs would fail on any dag-cbor block.Single-request CAR fetch is deliberate, per-block Helia transport was considered and rejected for now (one request per CID vs thousands for a large DAG). If helia block-brokers grows a streaming CAR transport, this module is the swap point.
How to verify
Notes / risks
Base:
feat/migrate-1-state-store. Merge order 1 through 5.