Skip to content

Conversation

LexLuthr
Copy link
Contributor

@LexLuthr LexLuthr commented May 14, 2025

PR is ready to review

  1. Main concern is to get the database changes right to ensure we don't need any changes in near future.
  2. There are a few places where optimisation and refactoring is required. This can be done in a follow up PRs.
  3. All the strikethrough items will be done in follow up PRs.
  4. There are some testing tooling with bad testing code around. I would like to clean it up in follow up PRs as I might need them till MK2.0 is in production.
  • Establish consensus on design
  • Aggregate Indexing
  • Aggregate IPNI
  • Test contract
  • Force PoRep interface
  • Piece_CID V2 migration for IPNI table
  • PP INIT
  • Clean up index and IPNI task
  • Rename everything to new PDP standard
  • Delete data after 1 hour
  • Update retrieval to work with MPD for PDP
  • PDP pipeline UI
  • Docs
  • Delete old PDP tables
  • PieceCID V2
    • Retrievals
    • Indexing
    • IPNI
    • UI - MK12 and MK20
    • MIgration to v2
    • Indexing Repair task
  • Write deal validation logic tests
  • MK2.0 UI
  • Notification pathway
  • Create mk20 test client
  • Write documentation for MK2.0 standard
  • Move PDP endpoints under market
  • Move any other market related endpoint
  • HTTP PUT UI page
  • Testing
    • Devnet
    • Calibnet
    • Mainnet

Tests:

  1. Make mk20 deal with all data source types
  2. Force a mk12 and mk20 deal in a sector
  3. Deal status
  4. Test deal validation logic on each param
  5. Test any other endpoints
  6. Test UI
  7. Deal cancellation
  8. Test indexing and announcements
  9. Bombard system with mk12 and mk20 deals and see if anything breaks
  10. Payload Retrievals
  11. Piece Retrievals
  12. Payload Retrieval from aggregate piece
  13. Sub piece from aggregate Retrievals
  14. Fix index
  15. Fix IPNI
  16. V1 -> V2 migration of indexes
  17. Bulk Restart mk20
  18. Bulk remove mk20
  19. PDP

Copy link
Collaborator

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just reviewing types for now, didn't look at much beyond that, but generally things make sense.

@LexLuthr LexLuthr force-pushed the feat/market2 branch 2 times, most recently from 6d720c7 to 0e6bd97 Compare May 20, 2025 14:36
@LexLuthr LexLuthr force-pushed the feat/market2 branch 2 times, most recently from 35ba248 to a2e3a77 Compare June 11, 2025 12:49
@LexLuthr LexLuthr requested review from magik6k and snadrus June 11, 2025 12:52
@rvagg rvagg mentioned this pull request Jun 18, 2025
@LexLuthr LexLuthr changed the title [WIP] feat: market 2.0 feat: market 2.0 Aug 21, 2025
@LexLuthr LexLuthr marked this pull request as ready for review August 21, 2025 19:26
@snadrus
Copy link
Contributor

snadrus commented Sep 1, 2025

  • remove the ipni DB code "With..." since chain makes it unique.

  • rm ddo_contracts

  • use: COMMENT 'abcdefg'. for tables & columns
    -- market_mk20_deal: describe the client column
    -- piece_aggregation vs deal_aggregation: clear description

  • disaggregation ideas --> can we ship it?

  • clients change (new idea after break)

@LexLuthr LexLuthr requested a review from a team as a code owner September 3, 2025 17:13
@LexLuthr
Copy link
Contributor Author

LexLuthr commented Sep 9, 2025

PDP pipeline

Screenshot 2025-09-09 at 6 32 56 PM

func verifySignature(db *harmonydb.DB, keyType string, pubKey, signature []byte, cfg *config.CurioConfig) (bool, string, error) {
now := time.Now().Truncate(time.Hour)
minus1 := now.Add(-59 * time.Minute)
plus1 := now.Add(59 * time.Minute)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 59 minutes instead of 1 hour?
Go playground
I believe the intent is to allow messages from the previous and next hour, giving a validity duration between 2 and 3 hours. But by using 59 minutes instead, the validity duration is between 0 and 1 hours, which could also be achieved by only checking one msg.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can change it to 1 hour. I should be fine. This was just to limit the time for testing period. Since, we are planning to switch to session keys, this would hold less significance as timestamp signing will move to original client signing the Auth for session key side. It allows us to allow auth with some expiry period.

}

var allowed bool
err := db.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM market_mk20_clients WHERE client = $1 AND allowed = TRUE)`, client).Scan(&allowed)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't enable the whitelist with DenyUnknownClients, can we still have a blacklist? I imagine some sp's may want to blacklist bad actors but might not want a strict whitelist.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List is flexible. You can mark a client allow/deny without separate lists.

return "", nil, nil, errors.New("missing CustomAuth prefix")
}

parts := strings.SplitN(strings.TrimPrefix(header, Authprefix), ":", 3)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the prefix had a colon instead of a space, this step would be slightly simpler.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can switch. Again, this is just basic Auth for testing purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants