feat!(cli): default egress to none and reuse existing data sets by SgtPooki · Pull Request #658 · filecoin-project/filecoin-pin

SgtPooki · 2026-08-18T20:23:18Z

What changed

Two isolated commits:

feat!(cli): default --egress-provider to none. FilBeam CDN egress is now opt-in for add and import (--egress-provider beam or EGRESS_PROVIDER=beam). The default no longer creates CDN-enabled data sets or locks the extra 1 USDFC, and the CLI now matches the upload-action, which already defaulted to none. Reverts the default introduced in Add --egress-provider flag with beam as default #464.
feat(cli): reuse existing filecoin-pin data sets by default. Without --data-set-id, --provider-id, or --data-set-metadata, add/import now match any live, active data set with source=filecoin-pin and upload into it. The SDK's smart-select requires exact metadata equality, so after the default flip every upload from an existing user would have skipped their withCDN-tagged data sets and created new ones per copy. The subset match reuses them instead. When more data sets match than copies requested, it picks the ones storing the most pieces across distinct providers (as documented in the glossary's Data Set entry). With --egress-provider beam, only CDN-enabled data sets qualify.

Calibration test runs

Wallet 0x44f0...3759 (972 data sets: 970 non-CDN, 2 CDN from the old beam default).

Old flow: v1.3.0, no flags (beam default)

Filecoin Pin Add

✓ File validated (69.0 B)
✓ Connected to Filecoin - Calibration testnet

Egress: FilBeam
  • Egress consumes the data set owner's locked-up funds.
  • FilBeam routes piece/CAR retrieval only, not IPFS blocks.
  • Each new data set locks an extra 1 USDFC.
  • Disable: --egress-provider none
✓ File packed with root CID: bafkreigajxvcttwifh5vw2vggn3bne6vterq5khcimz6u6uql525j3x7iy
✓ Cost estimate ready

Copies
  Requested: 2 (2 existing data sets)

Every upload defaults into CDN-tagged data sets (egress billed to the owner). For a wallet without CDN data sets, this creates 2 new ones and locks 2 USDFC, even when hundreds of non-CDN filecoin-pin data sets exist: exact metadata matching cannot see them.

New flow: this branch, no flags (none default)

Filecoin Pin Add

✓ File validated (69.0 B)
✓ Connected to Filecoin - Calibration testnet

✓ Reusing existing data sets 231, 54 (966 matched, picked the 2 storing the most data)

✓ File packed with root CID: bafkreigajxvcttwifh5vw2vggn3bne6vterq5khcimz6u6uql525j3x7iy
✓ Cost estimate ready

Copies
  Requested: 2 (2 existing data sets)

No FilBeam notice, no CDN lockup, and the upload lands in the wallet's own long-lived data sets (231: 595 pieces on provider 7, 54: 215 pieces on provider 4). Data set 84 (541 pieces) did not qualify: its PDP payments have ended (pdpEndEpoch set).

New flow with --egress-provider beam: reuses only CDN data sets, confirmed on-chain

✓ Reusing existing data sets 14030, 14031

✓ [Primary] Stored on provider 4
✓ [Secondary] Stored on provider 2
✓ IPNI provider records found. IPFS retrieval possible.
✓ [Primary] Piece added to Data Set (confirmed on-chain)
✓ [Secondary] Piece added to Data Set (confirmed on-chain)

Copies
  [Primary] Provider 4
    Data Set ID: 14030
    Piece ID: 6
  [Secondary] Provider 2
    Data Set ID: 14031
    Piece ID: 5

With beam requested, only the 2 CDN-enabled data sets qualify (not the 966 non-CDN ones), so the egress request is never silently dropped.

How to verify

pnpm run build && npx vitest run. New coverage: resolveDefaultDataSetReuse/pickDataSetsForReuse in upload-flow.test.ts, glue tests in add.test.ts/import.test.ts, and the flipped egress-default tests.

Notes / risks

With fewer matching data sets than requested copies (including none), behavior is unchanged: the SDK resolves or creates data sets. A partial match (1 existing set, 2 copies requested) still creates 2 new sets rather than mixing reuse with creation.
Reuse pins the upload to the picked data sets' providers with no health check. Observed on calibration: provider 7 (backing data set 231) was returning 502s, which fails the default upload until the SP recovers; --provider-id overrides. Candidate follow-up: skip unreachable providers during the pick.
With --egress-provider none, CDN-tagged data sets still qualify for reuse: none means "do not request or create CDN", not "never add to a CDN data set". Excluding them would recreate the new-set churn for wallets whose only data sets came from the old beam default. Documented in the resolveDefaultDataSetReuse docblock.
Piece-count ordering and provider spreading apply only when more data sets match than requested copies; an exact-count match is used as-is.
The pinning server's upload path still uses SDK smart-select; it never set withCDN, so its data sets already match exactly.
The subset matcher now also skips data sets scheduled for termination (pdpEndEpoch set); this applies to --data-set-metadata resolution too.

generated with claude, peer-reviewed with three other models (codex, cursor-agent, gemini)

Copilot

Pull request overview

Updates the filecoin-pin CLI upload flows to make FilBeam egress opt-in by default and to reuse existing filecoin-pin data sets when the user provides no explicit targeting, reducing unnecessary new data set creation and associated lockups.

Changes:

Default --egress-provider to none for add/import, keeping FilBeam routing explicitly opt-in.
Add default data set reuse logic to target existing live source=filecoin-pin data sets (and CDN-enabled ones when egress is requested), with deterministic selection when more match than requested copies.
Update documentation and unit tests to reflect the new defaults and reuse behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
upload-action/README.md	Updates Action docs to reflect egress-off-by-default alignment with CLI.
upload-action/examples/cli-recipe/README.md	Updates CLI recipe docs for new egress default and opt-in behavior.
src/utils/cli-options-egress.ts	Updates CLI egress option documentation/help text for default `none`.
src/test/unit/upload-flow.test.ts	Adds unit coverage for data set reuse resolution and reuse selection logic.
src/test/unit/import.test.ts	Updates import tests for egress default and adds reuse “glue” coverage.
src/test/unit/add.test.ts	Updates add tests for egress default and adds reuse “glue” coverage.
src/import/import.ts	Defaults egress to `none` and wires in default data set reuse when untargeted.
src/core/data-set/resolve-by-metadata.ts	Adds `requireKeys` support and excludes terminating data sets from matches.
src/common/upload-flow.ts	Introduces `pickDataSetsForReuse` + `resolveDefaultDataSetReuse`.
src/add/add.ts	Defaults egress to `none` and wires in default data set reuse when untargeted.
documentation/glossary.md	Updates FilBeam egress glossary entry for opt-in default and network notes.
AGENTS.md	Updates contributor-facing docs to reflect new egress default.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

FilBeam CDN egress is now opt-in for add and import. Pass --egress-provider beam (or EGRESS_PROVIDER=beam) to route piece/CAR retrieval through FilBeam; the default no longer creates CDN-enabled data sets or locks the extra 1 USDFC. This also aligns the CLI default with the upload-action, which already defaulted egressProvider to none.

When add/import run without --data-set-id, --provider-id, or --data-set-metadata, match any live, active data set whose metadata carries source=filecoin-pin and upload into it instead of relying on the SDK's smart-select. The SDK requires exact metadata equality, so data sets carrying extra keys (e.g. withCDN from when FilBeam egress was the default) were skipped and a new data set was created per copy. When more data sets match than copies requested, pick the ones storing the most pieces, spread across distinct providers. When FilBeam egress is requested, only CDN-enabled data sets qualify so the egress request is never silently dropped. With fewer matches than requested copies, or none, the SDK resolves or creates data sets as before. The subset matcher now also skips data sets scheduled for termination (pdpEndEpoch set), which applies to --data-set-metadata resolution too.

SgtPooki

self review. Note that this removes FilBeam as default for add/import operations, and fixes data-set matching so filecoin-pin focuses only on finding data-sets where source: filecoin-pin, so we don't force existing users who are adding data to create new data-sets when this goes through.

I think defaulting to just source matching for data-sets is a net improvement for filecoin-pin users.

beck-8

Although I don't know what the background of the change is. But there seems to be no problem with logic.

rvagg · 2026-08-26T00:44:45Z

+    spinner.stop(
+      `${pc.gray('•')} Found ${resolution.matchedIds.length} existing data set(s) but need ${expectedCopies}; the SDK will resolve or create data sets`
+    )
+    return undefined
+  }
+
+  spinner.stop(`${pc.gray('•')} No existing filecoin-pin data sets found; the SDK will create new ones`)


"the SDK" - this is leaky detail the user doesn't need to see

rvagg · 2026-08-26T00:46:03Z

+  if (resolution.kind === 'too-many-matches') {
+    const chosen = pickDataSetsForReuse(resolution.matchedDataSets, expectedCopies)
+    spinner.stop(
+      `${pc.green('✓')} Reusing existing data sets ${chosen.join(', ')} (${resolution.matchedIds.length} matched, picked the ${expectedCopies} storing the most data)`


"most data" maybe not, but "most pieces"
but, I don't know you really need to be too wordy in these messages, terseness is nice so you don't overload the user with detail they may not want to try to understand

rvagg · 2026-08-26T00:50:17Z

+  for (const ds of sorted) {
+    if (picked.length >= count) break
+    if (!picked.includes(ds)) picked.push(ds)
+  }


I think you need to remove "before doubling up on one" variation in this method, as you say: "copies on the same provider add no redundancy" so why do it? if you can't find unique providers, then it's time to set up a new data set, but becasuse this function is assumed to pass you're going to need to handle a failure case in resolveDefaultDataSetReuse to fall-through to the "let the SDK handle it"

Copilot AI lite review requested due to automatic review settings August 18, 2026 20:23

FilOzzy added team/filecoin-pin "Filecoin Pin" project is a stakeholder for this work. team/fs-wg FOC working group is a stakeholder for this work, and thus wants to track it on their project board. labels Aug 18, 2026

FilOzzy added this to FOC Aug 18, 2026

github-project-automation Bot moved this to 📌 Triage in FOC Aug 18, 2026

Copilot started reviewing on behalf of SgtPooki August 18, 2026 20:23 View session

SgtPooki self-assigned this Aug 18, 2026

SgtPooki requested review from hugomrdias, jennijuju and rvagg August 18, 2026 20:26

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Comment thread src/core/data-set/resolve-by-metadata.ts

Comment thread src/common/upload-flow.ts

SgtPooki force-pushed the feat/egress-default-none-dataset-reuse branch 2 times, most recently from 9652a09 to a0dfbfe Compare August 18, 2026 20:42

SgtPooki force-pushed the feat/egress-default-none-dataset-reuse branch 2 times, most recently from 6ddcf35 to 1bf7c7d Compare August 18, 2026 22:12

SgtPooki moved this from 📌 Triage to 🔎 Awaiting review in FOC Aug 18, 2026

SgtPooki force-pushed the feat/egress-default-none-dataset-reuse branch from 1bf7c7d to 698a649 Compare August 18, 2026 22:51

SgtPooki mentioned this pull request Aug 19, 2026

test: use bigint literals for SDK bigint fields #666

Merged

SgtPooki requested a review from beck-8 August 20, 2026 15:35

SgtPooki commented Aug 20, 2026

View reviewed changes

beck-8 approved these changes Aug 21, 2026

View reviewed changes

github-project-automation Bot moved this from 🔎 Awaiting review to ✔️ Approved by reviewer in FOC Aug 21, 2026

rvagg reviewed Aug 26, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat!(cli): default egress to none and reuse existing data sets - #658

feat!(cli): default egress to none and reuse existing data sets#658
SgtPooki wants to merge 2 commits into
masterfrom
feat/egress-default-none-dataset-reuse

SgtPooki commented Aug 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Uh oh!

beck-8 left a comment

Uh oh!

rvagg Aug 26, 2026

Uh oh!

rvagg Aug 26, 2026

Uh oh!

rvagg Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

SgtPooki commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Calibration test runs

How to verify

Notes / risks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

beck-8 left a comment

Choose a reason for hiding this comment

Uh oh!

rvagg Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

rvagg Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

rvagg Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SgtPooki commented Aug 18, 2026 •

edited

Loading