[WIP] feat: PDP #227

magik6k · 2024-09-30T19:09:41Z

Reading:

Roadmap:

hannahhoward

I've mostly put in comments to check for understanding before I start writing code. But generally LGTM -- looks like it's compliant with the API we agreed on :)

hannahhoward · 2024-10-20T22:47:22Z

pdp/auth.go

+)
+
+// verifyJWTToken extracts and verifies the JWT token from the request and returns the serviceID.
+func (p *PDPService) verifyJWTToken(r *http.Request) (string, error) {


@magik6k if I am reading this right, I send a JWT with a claim of "service" = "storacha" and then I need to make sure the JWT is signed by a private key that corresponds to a public key that was previously inserted into the pdp_services table with probably the RPC api below to add PDP Service (i.e the SP called "AddPDPService" locally using their running node -- assuming "AddPDPService" is not available on the public endpoint)

Yeah, see

curio/cmd/pdptool/main.go

Line 201 in 54af620

func createJWTToken(serviceName string, privateKey *ecdsa.PrivateKey) (string, error) {

hannahhoward · 2024-10-20T22:49:23Z

cuhttp/server.go

@@ -204,5 +215,10 @@ func attachRouters(ctx context.Context, r *chi.Mux, d *deps.Deps) (*chi.Mux, err
 	}
 	ipni_provider.Routes(r, ipp)

+	if sd.EthSender != nil {
+		pdsvc := pdp.NewPDPService(d.DB, d.LocalStore, must.One(d.EthClient.Get()), sd.EthSender)


this is essentially what's adding all the PDP routes correct (i.e. /proof-sets, POST to /piece)

hannahhoward · 2024-10-20T23:02:23Z

pdp/handlers.go

+			// Collect pieceInfos for subroots
+			pieceInfos := make([]abi.PieceInfo, len(addRootReq.Subroots))
+
+			var totalOffset uint64 = 0


am I correct that reading this code right here provides indirect instructions on how to aggregate all the subroot pieces into a root CID? IOW, I can assume that I can basically want to write aggregation code that will satisfy this code by generating the same RootCID? (which appears to be all the pieces concatenated together).

Also it would appear non-ffi is a fork of go-commp-utils -- is this public?

Yes, the subroot aggregation is the same as in filecoin pieces (CommP -> CommD)

The non-ffi thing is the same code as the Rust logic, reimplemented by Riba in Go.

There is one added constraint in PDP - pieces must be sorted by size in descending order - this way no inter-piece padding is needed

ooh good call out good to know!

hannahhoward · 2024-10-20T23:16:41Z

pdp/handlers_upload.go

+
+	// Parse request body
+	var req struct {
+		PieceCID string `json:"pieceCid"`


It looks like you're not requiring piececid v2 here, not in the put handler. Long term, I think we should require this. From the standpoint of the SP, I imagine them to want to filter at the piece level, and I imagine one of the main filter critieria to be size. They should know exactly what size piece they're about to accept before they accept it.

Unless I'm wrong, a PieceCID v1 doesn't specify that, and absent a size parameter, they could be accepting anything from 32 bytes to 256 MB.

Yes, PieceCIDv2 is still todo, can quickly update if that's a blocker

pdp/handlers_upload.go

hannahhoward · 2024-10-20T23:30:52Z

lib/pieceprovider/piecepark_reader.go

+	"github.com/filecoin-project/curio/lib/storiface"
+)
+
+type PieceParkReader struct {


This is what finds and reads PDP pieces for get requests correct?

Yeah, PiecePark is the general "raw piece" storage in Curio, we eventually intend to serve all retrievals from it (instead of from the fr32-padded "unsealed" sector files), but PDP is the first place which will actually do reads directly from it.

hannahhoward · 2024-10-21T02:52:49Z

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

hannahhoward · 2024-10-21T03:09:27Z

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

magik6k

Github thinks I have an unsubmitted review here and can't discard it..

magik6k · 2024-10-21T08:05:00Z

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

Yes, correct, retrievals work already. No ACL mechanism yet, but it's not going to be hard to implement once we have a spec (you can see that all retrieval requests quite quickly become aware what piece they are reading from + ACLs should be quite cachable)

magik6k · 2024-10-21T08:10:28Z

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

Other than delete all the things should be there. Operations on PDP proofsets (other than create) you should probably watch through eth events. I can also make the message-sending endpoints output the message hash somehow, but you still want to watch for all the events to not miss any contract changes made by the SP

hannahhoward · 2024-10-27T22:55:46Z

Upon further review, it's possible we have a design issue here that we need a solution for.

Please let me know if I am understanding the sequence of events correctly when a user uploads a pdp piece:

The piece is put in the “Stash”
A parked piece record is created
Asychronously, the piece moves from the stash to permaneant storage (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/piece/task_park_piece.go)
After that, again asynchronously, the notify task is called, which hits the notify URL. (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/pdp/notify_task.go)

Why this matters: a basic gaurantee of all hot storage, which storacha tries to offer, is read-on-write. The basic expectation is from the users point of view is that when an upload finishes, it should be retrievable.

One thing that is not clear I'd like to clarify: are steps 3 & 4 likely to run in a couple seconds or much longer? If it's only a few seconds, we can work around this. Fortunately our upload process includes a confirmation step AFTER the upload, and what we can do is essentially block the confirmation until step 4 completes. But that only works if it's a short period of time before the piece is fully retrievable.

If not, we really need to revisit the architecture and find a way to shorten the process.

hannahhoward · 2024-10-28T05:33:23Z

Follow-up: a pull based approach for checking for piece cid -- #304

* feat(pdp): add handler to check for piece cid status * feat(pdp): add handler to check for upload exists

Allows accepting JWTs that are signed with Ed25519 as well as ECDSA

x509 does not parse to pointer type on ed25519

magik6k force-pushed the feat/pdp branch 2 times, most recently from 07a5b22 to 5c3f546 Compare October 4, 2024 18:35

magik6k force-pushed the feat/pdp branch from 3ec47d6 to 54af620 Compare October 13, 2024 07:34

hannahhoward approved these changes Oct 20, 2024

View reviewed changes

magik6k commented Oct 21, 2024

View reviewed changes

magik6k force-pushed the feat/pdp branch 4 times, most recently from fa8d937 to a3a1c3a Compare October 24, 2024 13:28

magik6k added 15 commits October 30, 2024 11:15

pdp: Inital scaffold

fc7399d

pdp: Drop unneded handlers

40aad8c

pdp: wip uploads

e3ad509

pdp: Proper pieceref tracking

ececf7c

storage: Implement local stash storage

ed8c966

storage: Make Local use a single URL

afa54ec

pdp: Complete upload

794d167

piecepark: Special long-term storage mode

6f36c6b

dealdata: Handle custore://

041675a

pdp: Notify task

4daac0c

retrieval: Foundations for PiecePark readers

cf7ae93

retrieval: Parked piece /piece retrievals

2082fe9

pdptool for interacting with curio pdp endpoints

cea21f7

pdp webui: Basic page, adding services through UI

76634fd

pdp webui: Remove service button

14487f2

magik6k and others added 26 commits October 30, 2024 11:15

pdp: Working root adds

ee7e30f

pdp: Add-root watcher

dd6a47c

pdp: working root adding

b73adc2

pdp: Watch challenge epochs

8c4e566

pdp: Scaffold prove task

e06f759

pdp: Prove scheduling

4f50a46

pdp: Path from challenge to prove

07ad29f

proof: Sha memtree

7e1798a

proof: Binary memtree proover

04b4bc5

pdp: Require sorted subroot pieces

67ba074

pdp: Getting there with proof verification

5538028

pdp: Proof bugfixing progress

b26d885

pdp: PROOFS WORK!

eeca445

pdp: An actually working poller this time

a03ba8d

pdp: better praams

c555027

pdp: Update contarcts

b910e95

make gen, consistent max piece size

ecc52c9

fix local path test

2e4f9b9

pdp: Implement fees

2adff15

pdp chall lookback in prove

16ccaad

update to new proving scheme

4ac1e85

small fixes

2cea05f

pdp: Support arbitrary hash funcs in uploads

7513819

pdp: api readme

ff3d92f

fix lint

3190e7c

feat(pdp): add endpoint to poll for completed upload (#304)

4cb6643

* feat(pdp): add handler to check for piece cid status * feat(pdp): add handler to check for upload exists

magik6k force-pushed the feat/pdp branch from 8fe488c to 4cb6643 Compare October 30, 2024 04:15

magik6k and others added 3 commits November 1, 2024 12:13

enable chain sched with just eth sender

143411a

feat(pdp): add ed25519 support for JWT (#323)

4b70e1d

Allows accepting JWTs that are signed with Ed25519 as well as ECDSA

fix(pdp): fix ed25519 auth (#324)

921be6f

x509 does not parse to pointer type on ed25519

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat: PDP #227

[WIP] feat: PDP #227

magik6k commented Sep 30, 2024 •

edited

Loading

hannahhoward left a comment

hannahhoward Oct 20, 2024

magik6k Oct 21, 2024

hannahhoward Oct 20, 2024

hannahhoward Oct 20, 2024

magik6k Oct 21, 2024

hannahhoward Oct 25, 2024

hannahhoward Oct 20, 2024

magik6k Oct 21, 2024

hannahhoward Oct 20, 2024

magik6k Oct 21, 2024

hannahhoward commented Oct 21, 2024

hannahhoward commented Oct 21, 2024

magik6k left a comment •

edited

Loading

magik6k commented Oct 21, 2024

magik6k commented Oct 21, 2024

hannahhoward commented Oct 27, 2024 •

edited

Loading

hannahhoward commented Oct 28, 2024

[WIP] feat: PDP #227

Are you sure you want to change the base?

[WIP] feat: PDP #227

Conversation

magik6k commented Sep 30, 2024 • edited Loading

hannahhoward left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hannahhoward commented Oct 21, 2024

hannahhoward commented Oct 21, 2024

magik6k left a comment • edited Loading

Choose a reason for hiding this comment

magik6k commented Oct 21, 2024

magik6k commented Oct 21, 2024

hannahhoward commented Oct 27, 2024 • edited Loading

hannahhoward commented Oct 28, 2024

magik6k commented Sep 30, 2024 •

edited

Loading

magik6k left a comment •

edited

Loading

hannahhoward commented Oct 27, 2024 •

edited

Loading