feat!(cli): default egress to none and reuse existing data sets

AGENTS.md

@@ -92,7 +92,7 @@ src/
 
 **Network**: `--network mainnet|calibration|devnet` (default: `mainnet`). Devnet reads config from foc-devnet's `devnet-info.json` and auto-resolves private key and RPC URL. `--network` and `--rpc-url` are mutually exclusive. When `--rpc-url`/`RPC_URL` is set, the chain is derived by probing the endpoint's `eth_chainId`.
 
-**Egress**: `--egress-provider beam|none` (default: `beam`). Controls whether uploaded pieces are routed via the [FilBeam egress CDN](documentation/glossary.md#filbeam-egress). Today FilBeam serves piece/CAR retrieval only, not IPFS-block retrieval. Pass `--egress-provider none` to opt out; the data set owner is on the hook for piece-retrieval egress.
+**Egress**: `--egress-provider beam|none` (default: `none`). Controls whether uploaded pieces are routed via the [FilBeam egress CDN](documentation/glossary.md#filbeam-egress). Today FilBeam serves piece/CAR retrieval only, not IPFS-block retrieval. Pass `--egress-provider beam` to opt in; the data set owner is on the hook for piece-retrieval egress.
 
 **Required env**: `PRIVATE_KEY=0x...` (with USDFC tokens; not needed for devnet)
 

documentation/behind-the-scenes-of-adding-a-file.md

@@ -173,7 +173,7 @@ As a single transaction, this takes ~30 seconds to be confirmed onchain.
 
 In order to upload a [CAR](glossary.md#car), Filecoin Pin needs to identify the SP to upload to.  This strategy is followed (assuming no overrides are provided):
 
-1. If the chain has record of a [Data Set](glossary.md#data-set) created by the wallet with the Data Set [metadata key](glossary.md#metadata) `source` set to 'filecoin-pin', then that DataSet ID and corresponding SP are used.  If there are multiple, then the one storing the most data will be used.
+1. If the chain has record of a [Data Set](glossary.md#data-set) created by the wallet with the Data Set [metadata key](glossary.md#metadata) `source` set to 'filecoin-pin', then that DataSet ID and corresponding SP are used.  If there are multiple, then those already holding pieces are preferred, at most one per SP.
 2. If there is no existing Data Set, then a new Data Set is created using an approved [Service Provider](glossary.md#service-provider) from the [Service Provider Registry](glossary.md#service-provider-registry).
 
 *Outputs:*

documentation/glossary.md

@@ -33,27 +33,27 @@ Curio is the software that [Filecoin Warm Storage Service](#filecoin-warm-storag
 
 Collections of stored data ([Pieces](#piece)) managed by [Filecoin Warm Storage Service](#filecoin-warm-storage-service). Each Data Set is tied to exactly one [Service Provider](#service-provider); all pieces in a Data Set are stored by the same SP. Each Data Set has [metadata](#metadata), Pieces, and an associated payment rail between [Filecoin Pay](#filecoin-pay) and the SP that handles ongoing storage payments.
 
-Filecoin Pin reuses existing Data Sets by default, matching on [metadata](#metadata) (`source='filecoin-pin'`). If multiple exist, it uses the one storing the most data.
+Filecoin Pin reuses existing Data Sets by default, matching on [metadata](#metadata) (`source='filecoin-pin'`). When more match than the number of copies requested, it prefers the ones already holding pieces and takes at most one per Service Provider. When the matches do not cover enough distinct providers, Filecoin Pin creates new Data Sets instead.
 
 ## FIL
 
 FIL is Filecoin's native token.  While [Filecoin Onchain Cloud](#filecoin-onchain-cloud) storage is currently denominated in [USDFC](#usdfc), gas for transactions on the Filecoin blockchain (e.g., adding a [Piece](#piece)) need to be paid for using FIL. Most transactions in the data onboarding flow for Filecoin Onchain Cloud are submitted by storage providers so client typically have minimal need to interact directly with FIL.
 
 ## FilBeam egress
 
-This concerns the [/piece retrieval](#piece-retrieval) CDN provided by [FilBeam](https://github.com/filbeam). When `filecoin-pin add` or `filecoin-pin import` is run with `--egress-provider beam` (the default), uploaded pieces are retrievable via FilBeam at `https://{wallet-address}.{filbeam-domain}/{pieceCid}` (e.g., `https://0xabc....calibration.filbeam.io/bafk...` on the [Calibration Network](#calibration-network)).
+This concerns the [/piece retrieval](#piece-retrieval) CDN provided by [FilBeam](https://github.com/filbeam). When `filecoin-pin add` or `filecoin-pin import` is run with `--egress-provider beam` (the default is `none`), uploaded pieces are retrievable via FilBeam at `https://{wallet-address}.{filbeam-domain}/{pieceCid}` (e.g., `https://0xabc....calibration.filbeam.io/bafk...` on the [Calibration Network](#calibration-network)).
 
 **What it does today:** Serves [`/piece` Retrieval](#piece-retrieval) only — whole-CAR fetches keyed by [Piece CID](#piece-cid). It does **not** route [`/ipfs` Retrieval](#ipfs-retrieval); for those, use the IPFS retrieval URLs printed alongside the upload result.
 
-**Network support:** FilBeam URLs are only printed on networks with a FilBeam endpoint (mainnet and [Calibration](#calibration-network)). On networks without one (e.g. devnet), `--egress-provider beam` stays the default but no FilBeam URL is shown.
+**Network support:** FilBeam URLs are only printed on networks with a FilBeam endpoint (mainnet and [Calibration](#calibration-network)). On networks without one (e.g. devnet), passing `--egress-provider beam` is accepted but no FilBeam URL is shown.
 
 **Cost:** CDN egress is paid from funds the data set owner locks up for it — it is not billed to their wallet at data-set creation. Instead, lockup is consumed as retrievals happen. Anyone who knows the piece CID and wallet address can trigger a retrieval, which draws down that egress lockup.
 
 **Lockup:** Creating a new FilBeam-enabled [Data Set](#data-set) requires an extra fixed lockup of 1 USDFC (on top of the data-set creation fee and ongoing storage cost). This is why a CDN upload that creates a new data set needs more deposited funds than a non-CDN one. `--auto-fund` accounts for it automatically; without it, deposit enough to cover the lockup or the upload fails with an insufficient-funds error.
 
 **Future state:** FilBeam is working on routing IPFS-block retrievals through the same CDN ([filbeam/roadmap#85](https://github.com/filbeam/roadmap/issues/85)). [Data Sets](#data-set) uploaded with FilBeam enabled today will benefit automatically when that ships.
 
-**Opting out:** Pass `--egress-provider none` (or `EGRESS_PROVIDER=none`) to skip FilBeam routing entirely.
+**Opting in:** FilBeam routing is off by default. Pass `--egress-provider beam` (or `EGRESS_PROVIDER=beam`) to enable it.
 
 ## Filecoin Pay
 

src/add/add.ts

@@ -19,11 +19,10 @@ import {
   estimateUploadCost,
   performAutoFunding,
   performUpload,
-  promptDataSetSelection,
+  resolveUploadTargets,
   validatePaymentSetup,
 } from '../common/upload-flow.js'
 import { carInputError, INPUT_IS_CAR, isCar } from '../core/car/index.js'
-import { resolveDataSetIdsByMetadata } from '../core/data-set/index.js'
 import { normalizeMetadataConfig, withDerivedNameMetadata } from '../core/metadata/index.js'
 import { DEFAULT_COPIES } from '../core/synapse/constants.js'
 import { initializeSynapse } from '../core/synapse/index.js'
@@ -95,7 +94,7 @@ export async function runAddFromCli(path: string, options: Record<string, any>):
     } = options
     const { pieceMetadata, dataSetMetadata } = resolveMetadataOptions(options, { includeErc8004: true })
 
-    const egressProvider = rawEgressProvider ?? 'beam'
+    const egressProvider = rawEgressProvider ?? 'none'
 
     addOptions = {
       ...addOptionsFromCli,
@@ -192,34 +191,17 @@ export async function runAdd(options: AddOptions): Promise<AddResult | AddDryRun
       printEgressNotice('beam')
     }
 
-    // Resolve partial --data-set-metadata locally; SDK metadata matching requires exact equality.
-    let effectiveDataSetMetadata = dataSetMetadata
-    if (dataSetMetadata != null && contextSelection.dataSetIds == null && contextSelection.providerIds == null) {
-      const expectedCopies = options.copies ?? DEFAULT_COPIES
-      spinner.start('Resolving data sets from --data-set-metadata...')
-      const resolution = await resolveDataSetIdsByMetadata(synapse, dataSetMetadata, { expectedCopies, logger })
-      if (resolution.kind === 'matched') {
-        contextSelection.dataSetIds = resolution.dataSetIds
-        effectiveDataSetMetadata = undefined
-        spinner.stop(
-          `${pc.green('✓')} Matched existing data sets ${resolution.dataSetIds.join(', ')} via metadata filter`
-        )
-      } else if (resolution.kind === 'too-many-matches') {
-        const chosenIds = await promptDataSetSelection(resolution.matchedDataSets, resolution.expected, spinner)
-        contextSelection.dataSetIds = chosenIds
-        effectiveDataSetMetadata = undefined
-      } else if (resolution.kind === 'too-few-matches') {
-        spinner.stop(`${pc.red('✗')} --data-set-metadata matched too few data sets`)
-        throw new Error(
-          `--data-set-metadata matched only ${resolution.matchedIds.length} data set(s) (${resolution.matchedIds.join(', ')}) ` +
-            `but expected ${resolution.expected} (lower --copies, widen the filter, or pass --data-set-id).`
-        )
-      } else {
-        spinner.stop(
-          `${pc.gray('•')} No existing data sets matched --data-set-metadata; SDK will create a new data set with the requested metadata`
-        )
-      }
+    const targets = await resolveUploadTargets(synapse, contextSelection, {
+      ...(dataSetMetadata != null && { dataSetMetadata }),
+      ...(options.copies != null && { copies: options.copies }),
+      withCDN,
+      spinner,
+      logger,
+    })
+    if (targets.dataSetIds != null) {
+      contextSelection.dataSetIds = targets.dataSetIds
     }
+    const effectiveDataSetMetadata = targets.dataSetMetadata
 
     // Check payment setup (may configure permissions if needed).
     // Skipped for --dry-run: this can submit an allowance-approval transaction,

src/common/upload-flow.ts

@@ -6,14 +6,15 @@
  */
 
 import { isCancel, multiselect } from '@clack/prompts'
-import type { CopyResult, FailedAttempt, Synapse, UploadCosts } from '@filoz/synapse-sdk'
+import { type CopyResult, type FailedAttempt, METADATA_KEYS, type Synapse, type UploadCosts } from '@filoz/synapse-sdk'
 import type { CID } from 'multiformats/cid'
 import pc from 'picocolors'
 import type { Logger } from 'pino'
+import { resolveDataSetIdsByMetadata } from '../core/data-set/index.js'
 import type { DataSetSummary } from '../core/data-set/types.js'
 import { resolveIpfsIndexedMetadata } from '../core/metadata/index.js'
 import { DEFAULT_LOCKUP_DAYS, type PaymentCapacityCheck } from '../core/payments/index.js'
-import { DEFAULT_COPIES } from '../core/synapse/constants.js'
+import { DEFAULT_COPIES, DEFAULT_DATA_SET_METADATA } from '../core/synapse/constants.js'
 import {
   checkUploadReadiness,
   executeUpload,
@@ -24,6 +25,7 @@ import {
 import { formatUSDFC } from '../core/utils/format.js'
 import { autoFund } from '../payments/fund.js'
 import type { AutoFundOptions } from '../payments/types.js'
+import type { ContextSelectionOptions } from '../utils/cli-auth.js'
 import type { Spinner } from '../utils/cli-helpers.js'
 import { cancel, formatFileSize, isInteractive } from '../utils/cli-helpers.js'
 import { log } from '../utils/cli-logger.js'
@@ -119,6 +121,161 @@ export async function promptDataSetSelection(
   }
 }
 
+/**
+ * Choose up to `count` data sets to reuse, at most one per provider.
+ *
+ * Preference order: data sets already holding pieces first, ties broken by
+ * lowest ID for determinism. Copies on the same provider add no redundancy, so
+ * distinct providers are a hard constraint: when the candidates do not cover
+ * `count` providers, fewer than `count` IDs come back and the caller falls
+ * through to creating new data sets.
+ */
+export function pickDataSetsForReuse(dataSets: DataSetSummary[], count: number): bigint[] {
+  const sorted = [...dataSets].sort((a, b) => {
+    if (a.hasActivePieces !== b.hasActivePieces) {
+      return a.hasActivePieces ? -1 : 1
+    }
+    return a.dataSetId < b.dataSetId ? -1 : 1
+  })
+
+  const picked: bigint[] = []
+  const seenProviders = new Set<bigint>()
+  for (const ds of sorted) {
+    if (picked.length >= count) break
+    if (seenProviders.has(ds.providerId)) continue
+    seenProviders.add(ds.providerId)
+    picked.push(ds.dataSetId)
+  }
+
+  return picked
+}
+
+/**
+ * Resolve existing filecoin-pin data sets to reuse when the user gave no
+ * explicit targeting (`--data-set-id`, `--provider-id`, `--data-set-metadata`).
+ *
+ * Matches on a metadata subset (the SDK's smart-select requires exact
+ * equality, so it skips data sets carrying extra keys such as `withCDN` and
+ * creates new ones instead). Any live, active filecoin-pin data set qualifies,
+ * CDN-tagged ones included: `--egress-provider none` means "do not request or
+ * create CDN", not "never add to a CDN data set". When FilBeam egress is
+ * requested, only CDN-enabled data sets qualify.
+ *
+ * Reuse requires one data set per distinct provider, so every copy lands on a
+ * different provider. Candidates already holding pieces are preferred and
+ * picked one per provider; when they do not cover `expectedCopies` providers,
+ * reuse is abandoned rather than stacking copies on one provider.
+ *
+ * Returns the data set IDs to upload to, or `undefined` when new data sets
+ * should be created instead (no matches, too few matches, or too few distinct
+ * providers among the matches).
+ */
+export async function resolveDefaultDataSetReuse(
+  synapse: Synapse,
+  options: { expectedCopies: number; withCDN: boolean; spinner: Spinner; logger: Logger }
+): Promise<bigint[] | undefined> {
+  const { expectedCopies, withCDN, spinner, logger } = options
+
+  spinner.start('Checking for existing data sets...')
+  const resolution = await resolveDataSetIdsByMetadata(synapse, DEFAULT_DATA_SET_METADATA, {
+    expectedCopies,
+    logger,
+    ...(withCDN && { requiredKeys: [METADATA_KEYS.WITH_CDN] }),
+  })
+
+  if (resolution.kind === 'no-match') {
+    spinner.stop(`${pc.gray('•')} No existing data sets to reuse; creating new ones`)
+    return undefined
+  }
+
+  if (resolution.kind === 'too-few-matches') {
+    spinner.stop(
+      `${pc.gray('•')} Only ${resolution.matchedIds.length} of ${expectedCopies} data sets available for reuse; creating new ones`
+    )
+    return undefined
+  }
+
+  const candidates = resolution.matchedDataSets
+  const chosen = pickDataSetsForReuse(candidates, expectedCopies)
+
+  if (chosen.length < expectedCopies) {
+    spinner.stop(`${pc.gray('•')} Matching data sets share a provider; creating new ones`)
+    return undefined
+  }
+
+  const ranked =
+    candidates.length > expectedCopies ? ` (${candidates.length} matched, picked the oldest ones with pieces)` : ''
+  spinner.stop(`${pc.green('✓')} Reusing existing data sets ${chosen.join(', ')}${ranked}`)
+  return chosen
+}
+
+/**
+ * Decide which data sets an upload targets, before the SDK resolves a context.
+ *
+ * Three paths, in order:
+ * - Explicit `--data-set-id`/`--provider-id`: nothing to resolve, the metadata
+ *   passes through untouched.
+ * - `--data-set-metadata`: resolved locally, since the SDK's matching requires
+ *   exact equality and a partial filter cannot reach existing data sets through
+ *   it. Prompts when more data sets match than copies were requested, and
+ *   throws when too few do.
+ * - Neither: reuse existing filecoin-pin data sets via
+ *   {@link resolveDefaultDataSetReuse}.
+ *
+ * Returns the data set IDs to target (absent when the SDK should resolve or
+ * create them) and the metadata to carry forward. Resolved IDs supersede the
+ * metadata filter, so `dataSetMetadata` comes back undefined alongside them.
+ */
+export async function resolveUploadTargets(
+  synapse: Synapse,
+  contextSelection: ContextSelectionOptions,
+  options: {
+    dataSetMetadata?: Record<string, string>
+    copies?: number
+    withCDN: boolean
+    spinner: Spinner
+    logger: Logger
+  }
+): Promise<{ dataSetIds?: bigint[]; dataSetMetadata?: Record<string, string> }> {
+  const { dataSetMetadata, withCDN, spinner, logger } = options
+  const expectedCopies = options.copies ?? DEFAULT_COPIES
+
+  if (contextSelection.dataSetIds != null || contextSelection.providerIds != null) {
+    return { ...(dataSetMetadata != null && { dataSetMetadata }) }
+  }
+
+  if (dataSetMetadata == null) {
+    const reuseIds = await resolveDefaultDataSetReuse(synapse, { expectedCopies, withCDN, spinner, logger })
+    return { ...(reuseIds != null && { dataSetIds: reuseIds }) }
+  }
+
+  spinner.start('Resolving data sets from --data-set-metadata...')
+  const resolution = await resolveDataSetIdsByMetadata(synapse, dataSetMetadata, { expectedCopies, logger })
+
+  if (resolution.kind === 'matched') {
+    spinner.stop(`${pc.green('✓')} Matched existing data sets ${resolution.dataSetIds.join(', ')} via metadata filter`)
+    return { dataSetIds: resolution.dataSetIds }
+  }
+
+  if (resolution.kind === 'too-many-matches') {
+    const chosenIds = await promptDataSetSelection(resolution.matchedDataSets, resolution.expected, spinner)
+    return { dataSetIds: chosenIds }
+  }
+
+  if (resolution.kind === 'too-few-matches') {
+    spinner.stop(`${pc.red('✗')} --data-set-metadata matched too few data sets`)
+    throw new Error(
+      `--data-set-metadata matched only ${resolution.matchedIds.length} data set(s) (${resolution.matchedIds.join(', ')}) ` +
+        `but expected ${resolution.expected} (lower --copies, widen the filter, or pass --data-set-id).`
+    )
+  }
+
+  spinner.stop(
+    `${pc.gray('•')} No existing data sets matched --data-set-metadata; a new data set will be created with the requested metadata`
+  )
+  return { dataSetMetadata }
+}
+
 export interface UploadFlowOptions {
   /**
    * Context identifier for logging (e.g., 'import', 'add')

src/core/data-set/resolve-by-metadata.ts

@@ -5,13 +5,20 @@ import type { DataSetSummary } from './types.js'
 
 export type MetadataResolution =
   | { kind: 'no-match' }
-  | { kind: 'matched'; dataSetIds: bigint[] }
+  | { kind: 'matched'; dataSetIds: bigint[]; matchedDataSets: DataSetSummary[] }
   | { kind: 'too-many-matches'; matchedIds: bigint[]; matchedDataSets: DataSetSummary[]; expected: number }
   | { kind: 'too-few-matches'; matchedIds: bigint[]; expected: number }
 
 export interface ResolveByMetadataOptions {
   expectedCopies: number
   logger?: Logger
+  /**
+   * Metadata keys that must be present on the dataset, with any value.
+   * Complements `requestedMetadata`, which matches exact key/value pairs.
+   * Used for keys whose value varies across SDK versions (e.g. `withCDN`
+   * is `''` when set by the SDK but `'true'` when set by hand).
+   */
+  requiredKeys?: string[]
 }
 
 /**
@@ -32,11 +39,17 @@ export async function resolveDataSetIdsByMetadata(
     return { kind: 'no-match' }
   }
 
+  const requiredKeys = options.requiredKeys ?? []
+
   const matched = await listDataSets(synapse, {
     filter: (dataSet) => {
       if (!dataSet.isLive) {
         return false
       }
+      // Exclude datasets scheduled for termination; uploads must target active ones.
+      if ((dataSet.pdpEndEpoch ?? 0n) !== 0n) {
+        return false
+      }
       const metadata = dataSet.metadata
       if (metadata == null) {
         return false
@@ -47,7 +60,10 @@ export async function resolveDataSetIdsByMetadata(
        * carry `someKey` at all, which violates the "requested keys are a subset
        * of dataset metadata" rule.
        */
-      return entries.every(([key, value]) => key in metadata && metadata[key] === value)
+      if (!entries.every(([key, value]) => key in metadata && metadata[key] === value)) {
+        return false
+      }
+      return requiredKeys.every((key) => key in metadata)
     },
     ...(options.logger != null && { logger: options.logger }),
   })
@@ -66,5 +82,5 @@ export async function resolveDataSetIdsByMetadata(
     return { kind: 'too-few-matches', matchedIds, expected: options.expectedCopies }
   }
 
-  return { kind: 'matched', dataSetIds: matchedIds }
+  return { kind: 'matched', dataSetIds: matchedIds, matchedDataSets: matched }
 }