Skip to content

Commit 7a39e0e

Browse files
snissnnijoe1
andauthored
docs(storage): refresh store and retrieval examples (#76)
* docs(storage): refresh store and retrieval examples * docs(storage): address retrieval review notes * docs(storage): swap Storacha for fil.one and Pinata in storage lists --------- Co-authored-by: nijoe1 <nick@fil.builders>
1 parent 04a2557 commit 7a39e0e

10 files changed

Lines changed: 100 additions & 109 deletions

File tree

build/cookbook/filecoin-pin/erc-8004-agent-registration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -685,7 +685,7 @@ Get more from the [faucet](https://www.alchemy.com/faucets/base-sepolia) if need
685685
```bash
686686
curl -s "https://ipfs.io/ipfs/<CID>/github-agent-card.json" | jq .
687687
curl -s "https://gateway.pinata.cloud/ipfs/<CID>/github-agent-card.json" | jq .
688-
curl -s "https://cloudflare-ipfs.com/ipfs/<CID>/github-agent-card.json" | jq .
688+
curl -s "https://dweb.link/ipfs/<CID>/github-agent-card.json" | jq .
689689
```
690690

691691
### Issue: PDP proofs not showing

build/cookbook/filecoin-pin/faq.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ On **Calibration testnet** (the demo website), data has no persistence guarantee
5555

5656
A Data Set groups your uploads together. Each upload becomes a "piece" within the Data Set. Multiple files you upload share the same payment rail.
5757

58-
Check your Data Set with `filecoin-pin data-set <id>`.
58+
Check your Data Set with `filecoin-pin data-set show <id>`.
5959

6060
***
6161

@@ -64,7 +64,7 @@ Check your Data Set with `filecoin-pin data-set <id>`.
6464
Three methods:
6565

6666
1. **IPFS Gateways:** Use public gateways with your root CID: `https://gateway.example.com/ipfs/<root-cid>`
67-
2. **Direct from Storage Provider:** Get the direct download URL from `filecoin-pin data-set <id>`
67+
2. **Direct from Storage Provider:** Get the direct download URL from `filecoin-pin data-set show <id>`
6868
3. **IPFS Tools:** Use Kubo, Helia, IPFS Desktop with your root CID.
6969

7070
***
@@ -85,7 +85,7 @@ Both are linked cryptographically on-chain.
8585

8686
Two ways to verify:
8787

88-
1. **CLI:** Run `filecoin-pin data-set <id>` to see on-chain verification. Check proof status and piece details.
88+
1. **CLI:** Run `filecoin-pin data-set show <id>` to see on-chain verification. Check proof status and piece details.
8989
2. **PDP Explorer:** Visit `https://pdp.vxb.ai/calibration/dataset/{datasetID}` to view proofs in your browser.
9090

9191
{% hint style="info" %}

build/cookbook/filecoin-pin/getting-started.md

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ Before starting, make sure you have:
3333
Install the CLI globally with npm:
3434

3535
```sh
36-
npm install -g filecoin-pin
36+
npm install -g filecoin-pin@latest
3737
```
3838

3939
Verify the installation:
@@ -71,22 +71,29 @@ In MetaMask:
7171

7272
The private key is a 64-character hex string, with or without an `0x` prefix.
7373

74-
### 2️⃣ Save the private key to a `.env` file
74+
### 2️⃣ Save or export the private key
75+
76+
The Filecoin Pin CLI does not store your key, but a `.env` file is still a local file on disk. Use a private working directory, keep `.env` out of git, and delete the file when you no longer need it.
7577

7678
Create a file named `.env` in your working directory containing:
7779

78-
```
80+
```sh
7981
export PRIVATE_KEY="0xYOUR_PRIVATE_KEY_HERE"
8082
```
8183

8284
Then secure it and load it into your shell:
8385

8486
```sh
87+
printf '.env\n' >> .gitignore
8588
chmod 600 .env
8689
source .env
8790
```
8891

89-
Add `.env` to your `.gitignore` if you're inside a git repository.
92+
For a one-off shell session, you can avoid writing the key to disk and export it directly instead:
93+
94+
```sh
95+
export PRIVATE_KEY="0xYOUR_PRIVATE_KEY_HERE"
96+
```
9097

9198
***
9299

@@ -223,13 +230,13 @@ filecoin-pin add my-data/
223230

224231
Your file is now retrievable via standard IPFS tooling using its Root CID. For example:
225232

226-
```
233+
```text
227234
https://<YOUR_ROOT_CID>.ipfs.inbrowser.link
228235
```
229236

230237
Or via the dweb.link gateway:
231238

232-
```
239+
```text
233240
https://dweb.link/ipfs/<YOUR_ROOT_CID>
234241
```
235242

@@ -246,13 +253,13 @@ Filecoin storage providers must cryptographically prove daily that they continue
246253
List the data sets associated with your wallet:
247254

248255
```sh
249-
filecoin-pin data-set --ls
256+
filecoin-pin data-set list
250257
```
251258

252259
Then get the full on-chain detail for a specific data set:
253260

254261
```sh
255-
filecoin-pin data-set <DATASET_ID>
262+
filecoin-pin data-set show <DATASET_ID>
256263
```
257264

258265
This queries the smart contracts directly, so the values are live blockchain state.

build/cookbook/retrieve-data.md

Lines changed: 25 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -8,83 +8,50 @@ description: >-
88

99
### <mark style="color:blue;">Retrieve data using retrieval clients</mark>
1010

11-
To retrieve data stored on the Filecoin network, the basic process involves making retrieval requests to Service Providers (SPs) who initially stored the data, using either the Content ID (CID) or the storage deal ID.
11+
To retrieve data stored on the Filecoin network, the basic process involves making retrieval requests to storage providers or IPFS peers using the root Content ID (CID) for the data.
1212

13-
A programmatic option is to utilize Filecoin retrieval clients, which handle the intricate retrieval process behind the scenes. By simply providing a Content ID (CID), retrieval clients can efficiently return your data either via the command-line interface (CLI) or through the programmable method.
13+
Filecoin retrieval clients handle provider discovery and transport selection behind the scenes. Provide the root CID, and the client returns the data through a command-line interface or a library integration.
1414

1515
#### **Ingredients**
1616

17-
With a given CID, you can use any of the following retrieval clients to retrieve content.
17+
With a given CID, you can use the following maintained retrieval tooling:
1818

19-
* [Lassie](https://github.com/filecoin-project/lassie): optimizes for most efficient available retrieval protocols.
20-
* [go-car](https://github.com/ipld/go-car): a content addressable archive utility.
19+
* [Lassie](https://github.com/filecoin-project/lassie): retrieves IPFS and Filecoin content over the best available protocols.
20+
* [go-car](https://github.com/ipld/go-car): reads, lists, and extracts content-addressed archive (CAR) files.
2121

2222
#### **Instructions**
2323

2424
**Retrieving content with Lassie**
2525

26-
Lassie is designed to fetch content in the content-addressed archive (CAR) form. In most cases, you will require additional tooling, such as the go-car library, to work with CAR files effectively.
26+
Install the current Lassie and go-car command-line tools. Make sure [Go](https://go.dev/doc/install) is installed and that your Go binary directory is on your `PATH`, or download the latest binaries from the [Lassie releases](https://github.com/filecoin-project/lassie/releases/latest) and [go-car releases](https://github.com/ipld/go-car/releases/latest):
2727

28-
The Lassie command line interface (CLI) provides the simplest method for retrieving content from the Filecoin/IPFS network. By using the `lassie fetch` command and passing the CID as an argument, you can easily retrieve the desired content.
29-
30-
<pre class="language-jsx"><code class="lang-jsx"><strong>lassie fetch -o - &#x3C;CID> | car extract
31-
</strong></code></pre>
28+
```sh
29+
go install github.com/filecoin-project/lassie/cmd/lassie@latest
30+
go install github.com/ipld/go-car/cmd/car@latest
31+
```
3232

33-
For example,
33+
Lassie fetches content in CAR form. Stream the CAR to go-car when you want to extract the UnixFS files immediately:
3434

35+
```sh
36+
lassie fetch -o - <CID> | car extract -
3537
```
36-
lassie fetch -p bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4 | car extract
38+
39+
To save the CAR for later inspection or extraction:
40+
41+
```sh
42+
lassie fetch -p -o <CID>.car <CID>
43+
car ls -f <CID>.car
44+
car extract -f <CID>.car
3745
```
3846

39-
Lassie can also serve as a go library within your Golang application when programmatically retrieving content from the network. To utilize Lassie in your code, you need to install the dependency and import it into your program following the instructions [here](https://github.com/filecoin-project/lassie?tab=readme-ov-file#golang-library).
40-
41-
The following example demonstrates how to use the Lassie library to fetch a CID.
42-
43-
```go
44-
package main
45-
46-
import (
47-
"context"
48-
"fmt"
49-
"os"
50-
51-
"github.com/filecoin-project/lassie/pkg/lassie"
52-
"github.com/filecoin-project/lassie/pkg/storage"
53-
"github.com/filecoin-project/lassie/pkg/types"
54-
"github.com/ipfs/go-cid"
55-
trustlessutils "github.com/ipld/go-trustless-utils"
56-
)
57-
58-
// main creates a default lassie instance and fetches a CID
59-
func main() {
60-
ctx := context.Background()
61-
62-
// Create a default lassie instance
63-
lassie, err := lassie.NewLassie(ctx)
64-
if err != nil {
65-
panic(err)
66-
}
67-
68-
// Prepare the fetch
69-
rootCid := cid.MustParse("bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4") // The CID to fetch
70-
store := storage.NewDeferredStorageCar(os.TempDir(), rootCid) // The place to put the CAR file
71-
request, err := types.NewRequestForPath(store, rootCid, "", trustlessutils.DagScopeAll, nil) // The fetch request
72-
if err != nil {
73-
panic(err)
74-
}
75-
76-
// Fetch the CID
77-
stats, err := lassie.Fetch(ctx, request)
78-
if err != nil {
79-
panic(err)
80-
}
81-
82-
// Print the stats
83-
fmt.Printf("Fetched %d blocks in %d bytes\n", stats.Blocks, stats.Size)
84-
}
47+
For example:
8548

49+
```sh
50+
lassie fetch -o - bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4 | car extract -
8651
```
8752

53+
For library integrations, use the current [Lassie Go library documentation](https://github.com/filecoin-project/lassie?tab=readme-ov-file#golang-library) instead of copying an example from this reference page.
54+
8855
For quick retrieval of existing datasets with the methods above, check out the [Filecoin Dataset Explorer](https://datasets.filecoin.io/).
8956

9057
* [Filecoin Dataset Explorer](https://datasets.filecoin.io/)

build/cookbook/store-data.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ Use the official resources below for implementation details.
1818
Use these resources to prepare CAR artifacts and storage inputs:
1919

2020
* [Lighthouse documentation](https://docs.lighthouse.storage/)
21-
* [IPFS CAR tooling (`ipfs-car`)](https://github.com/web3-storage/ipfs-car)
22-
* [IPLD CAR specification and libraries](https://github.com/ipld/js-car)
21+
* [go-car command-line tooling](https://github.com/ipld/go-car)
22+
* [IPLD CAR specification and JavaScript libraries](https://github.com/ipld/js-car)
2323
* [IPFS Desktop / Kubo docs](https://docs.ipfs.tech/)
2424

2525
Recommended outputs before proposing storage workflows:
@@ -41,6 +41,7 @@ Use the FOC stack for programmatic, verifiable storage at scale:
4141
For smaller datasets and managed ingestion paths:
4242

4343
* [Storage onramps overview](../../getting-started/how-storage-works/storage-onramps.md)
44+
* [Filecoin Pin getting started](filecoin-pin/getting-started.md)
4445
* [Lighthouse documentation](https://docs.lighthouse.storage/)
4546

4647
### <mark style="color:blue;">Monitor storage deal status from a smart contract</mark>

getting-started/how-retrieval-works/basic-retrieval.md

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -30,15 +30,15 @@ Make sure that you have [Go](https://go.dev/) installed and that your `GOPATH` i
3030

3131
Or download and install Lassie using the Go package manager:
3232

33-
```
34-
go install github.com/filecoin-project/lassie/cmd/lassie@latest
35-
```
33+
```sh
34+
go install github.com/filecoin-project/lassie/cmd/lassie@latest
35+
```
3636

3737
2. Download the [go-car binary from the latest release](https://github.com/ipld/go-car/releases/latest) based on your system architecture or install the [go-car](https://github.com/ipld/go-car) package using the Go package manager. The go-car package makes it easier to work with content-addressed archive (CAR) files:
3838

39-
```
40-
go install github.com/ipld/go-car/cmd/car@latest
41-
```
39+
```sh
40+
go install github.com/ipld/go-car/cmd/car@latest
41+
```
4242

4343
You now have everything you need to retrieve a file with Lassie and extract the contents with `go-car`.
4444

@@ -51,7 +51,7 @@ The video below demonstrates how Lassie can be used to render content directly f
5151
Lassie and `go-car` can work together to retrieve and extract data from Filecoin. All you need is the CID of the content to download.
5252

5353
```shell
54-
lassie fetch -o - <CID> | car extract
54+
lassie fetch -o - <CID> | car extract -
5555
```
5656

5757
This command uses a `|` to chain two commands together. This will work on Linux or macOS. Windows users may need to use PowerShell to use this form. Alternatively, you can use the commands separately, as explained later on this page.
@@ -92,27 +92,27 @@ lassie fetch -p -o <OUTFILE_FILE_NAME> <CID>/path/to/content
9292
* `-p` is an optional flag that tells Lassie that you would like to see detailed progress information as it fetches your data.
9393
9494
For example:
95-
* ```plaintext
96-
Fetching bafykbzaceatihez66rzmzuvfx5nqqik73hlphem3dvagmixmay3arvqd66ng6
97-
Querying indexer for bafykbzaceatihez66rzmzuvfx5nqqik73hlphem3dvagmixmay3arvqd66ng6...
98-
Found 4 storage providers candidates from the indexer, querying all of them:
99-
12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys
100-
12D3KooWNHwmwNRkMEP6VqDCpjSZkqripoJgN7eWruvXXqC2kG9f
101-
12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj
102-
12D3KooWLDf6KCzeMv16qPRaJsTLKJ5fR523h65iaYSRNfrQy7eU
103-
Querying [12D3KooWLDf6KCzeMv16qPRaJsTLKJ5fR523h65iaYSRNfrQy7eU] (started)...
104-
Querying [12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj] (started)...
105-
106-
...
107-
```
95+
96+
```plaintext
97+
Fetching bafykbzaceatihez66rzmzuvfx5nqqik73hlphem3dvagmixmay3arvqd66ng6
98+
Querying indexer for bafykbzaceatihez66rzmzuvfx5nqqik73hlphem3dvagmixmay3arvqd66ng6...
99+
Found 4 storage providers candidates from the indexer, querying all of them:
100+
12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys
101+
12D3KooWNHwmwNRkMEP6VqDCpjSZkqripoJgN7eWruvXXqC2kG9f
102+
12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj
103+
12D3KooWLDf6KCzeMv16qPRaJsTLKJ5fR523h65iaYSRNfrQy7eU
104+
Querying [12D3KooWLDf6KCzeMv16qPRaJsTLKJ5fR523h65iaYSRNfrQy7eU] (started)...
105+
Querying [12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj] (started)...
106+
107+
...
108+
```
108109
* `-o` is an optional flag that tells Lassie where to write the output to. If you don’t specify a file, it will append `.car` to your CID and use that as the output file name.
109110
110-
If you specify `-p`, the output will be written to `stdout` so it can be piped to another command, such as `go-car`, or redirected to a file.
111+
Use `-o -` to write the CAR stream to `stdout` so it can be piped to another command, such as `go-car`, or redirected to a file.
111112
112113
* `<CID>/path/to/content` is the CID of the content you want to retrieve and an optional path to a specific file within that content. Example:
113-
* ```shell
114-
lassie fetch -o - bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki/Cryptographic_hash_function | car extract - | less
115-
```
114+
115+
lassie fetch -o - bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki/Cryptographic_hash_function | car extract - | less
116116
117117
A CID is always necessary, and if you don’t specify a path, Lassie will attempt to download the entire content. If you specify a path, Lassie will only download that specific file or, if it is a directory, the entire directory and its contents.
118118
@@ -128,19 +128,19 @@ car extract -f <INPUT_FILE>[/path/to/file/or/directory] [<OUTPUT_DIR>]
128128
* `/path/to/file/or/directory` is an optional path to a specific file or directory within the CAR. If omitted, it will attempt to extract the entire CAR.
129129
* `<OUTPUT_DIR>` is an optional argument that tells `go-car` where to write the output to. If omitted, it will be written to the current directory.
130130
131-
If you supply `-p`, as in the above example, it will attempt to extract the content directly to `stdout`. This will only work if we are extracting a single file.
131+
If you supply `-p`, `car extract` writes extracted file bytes directly to `stdout`. This only works when extracting a single file.
132132
133133
In the example above, where we fetched a file named `lidar-data.tar`, the `>` operator was used to redirect the output of `car extract` to a named file. This is because the content we fetched was raw file data that did not have a name encoded. In this case, if we didn’t use `-` and `> filename`, `go-car` would write to a file named `unknown`. In this instance, `go-car` was used to reconstitute the file from the raw blocks contained within Lassie’s CAR output.
134134
135135
`go-car` has other useful commands. The first is `car ls`, which can be used to list the contents of a CAR. The second is `car inspect`, which can be used to inspect the contents of the CAR and optionally verify the integrity of a CAR.
136136
137-
And there we have it! Downloading and managing data from Filecoin is super simple when you use Lassie and Go-car!
137+
Lassie and go-car are the recommended command-line tools for retrieving and inspecting Filecoin data by CID.
138138
139139
#### Lassie HTTP daemon
140140
141141
The Lassie HTTP daemon is an HTTP interface for retrieving IPLD data from IPFS and Filecoin peers. It fetches content from peers known to have it and provides the resulting data in CAR format.
142142
143-
```shell
143+
```http
144144
GET /ipfs/{cid}[/path][?params]
145145
```
146146

getting-started/how-storage-works/storage-onramps.md

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,16 @@ description: >-
88

99
Developers use web UIs, APIs, or libraries to send data to storage onramps. Behind the scenes, storage onramps receive the data and handle the underlying processes to store it in a reliable way, making deals with Filecoin storage providers.
1010

11-
The available storage onramps are:
11+
Examples of maintained storage onramps include:
1212

13-
* [Lighthouse](https://lighthouse.storage/) "offers permanent, decentralized storage powered by Filecoin. Secure, scalable, and ideal for individuals, developers, and enterprises."
14-
* [Akave](https://www.akave.ai/) is "revolutionizing data management with a decentralized, modular solution that combines the robust storage of Filecoin with cutting-edge encryption and easy-to-use interfaces."
15-
* [Storacha](https://storacha.network/) is an open hot storage network scales IPFS and Filecoin. Upload any data and Storacha will ensure it ends up on a decentralized set of IPFS and Filecoin storage providers. There Storacha [docs](https://docs.storacha.network/) detail the JavaScript and Go API libraries, and there is a no-code web uploader available as well.
16-
* [Singularity](https://data-programs.gitbook.io/singularity) "facilitates onboarding of large quantities of data (PB-scale) to the Filecoin network in an efficient, secure, and flexible way."
17-
* [CID Gravity](https://www.cidgravity.com/) is a "seamless gateway to the decentralized web", allowing you to drag and drop files through an easy-to-use UI that uploads files to Filecoin and IPFS.
13+
* [Filecoin Onchain Cloud](../../build/advanced/filecoin-onchain-cloud.md) is a programmable, on-chain storage platform with verifiable storage proofs (PDP) and automatic payments (Filecoin Pay), accessed through the Synapse SDK.
14+
* [Filecoin Pin](../../build/cookbook/filecoin-pin/getting-started.md) is a CLI and API path for pinning IPFS-compatible content to Filecoin-backed storage with Filecoin Pay.
15+
* [fil.one](https://fil.one/) is S3-compatible object storage backed by Filecoin, with flat per-terabyte pricing and no egress fees. Point any S3 SDK or tool at its endpoint to store data with cryptographic integrity proofs. See the [fil.one docs](https://docs.fil.one/).
16+
* [Lighthouse](https://lighthouse.storage/) offers permanent, decentralized storage powered by Filecoin.
17+
* [Akave](https://www.akave.ai/) provides a decentralized data-lake and object-storage layer backed by Filecoin.
18+
* [Pinata](https://pinata.cloud/) is an IPFS pinning service for storing and serving files, media, and app data over IPFS. See the [Pinata docs](https://docs.pinata.cloud/).
19+
* [Singularity](https://data-programs.gitbook.io/singularity) facilitates onboarding large quantities of data to the Filecoin network.
20+
* [CID Gravity](https://www.cidgravity.com/) provides a web UI for uploading files to Filecoin and IPFS.
1821
* [Ramo](https://use.ramo.computer/) provides Filecoin-based, S3-compatible storage for data on Filecoin.
1922

2023
[Was this page helpful?](https://airtable.com/apppq4inOe4gmSSlk/pagoZHC2i1iqgphgl/form?prefill\_Page+URL=https://docs.filecoin.io/getting-started/how-storage-works/storage-onramps)

0 commit comments

Comments
 (0)