GIP-0083: Substreams On The Network. #63
Conversation
|
Ok I've revised a few things, based on a few feedback elements I've received. |
|
Another comment, the GIP makes no references to Horizon so I assume it's not a pre-requisite, is that assumption correct? |
abourget
changed the title
abourget
changed the title
… what is informational. Linked to documents with payments flow details.
There was a problem hiding this comment.
It is now a lot more detailed, and I believe it covers everything reviewers agreed was missing.
I've added a few comments.
One thing I think is missing, though, is the discussed automated payments mechanism for Indexers to claim payments. We had discussed this as a solution before the more automated and fully trust-minimized approach, right?
…st, and can be allocated and disputed on. More clarity on the attestation signature payload, including the module_hash being covered, which would facilitate pinning down culprits.
Added details about network subgraph, fixed a few typos.
|
|
||
| ## Economic Security and Disputes | ||
|
|
||
| Economic security is achieved through slashing and disputes, similar to subgraphs as of Jan 24th 2025. Indexers providing incorrect data CAN have their stake slashed. Therefore, to be an Indexer recognized by the Payment Gateway, one must staked the Network token, just like for subgraphs. |
There was a problem hiding this comment.
Has there been discussions on setting the minimum staking requirement for substreams or the same as for subgraphs? Any clarifications on the slashing penalties?
There was a problem hiding this comment.
This should be clarified to ensure that the slashing threat is credible. This being pre-Horizon, presumably the same 100k stake and 2.5% slash apply?
|
|
||
| #### Methods of analysis | ||
|
|
||
| THIS SECTION IS NON-NORMATIVE |
There was a problem hiding this comment.
I think it would be nice to add some basic description of how signed attestations is submitted and general timeline for arbitration
| This command allows providers to register their service, specifying their operator wallet, private key file, the service type (`substreams-v1`), the endpoint URL, logo, name, and the supported network. The registration data is signed by the operator's key. An update to an existing registration can be achieved by re-running the command with updated parameters, based on the `service`, `providerId` and `network` tripled. A corresponding `unregister` or `revoke` command would be added to allow providers to take down their service from the front-end. | ||
|
|
||
|
|
||
| # Dependencies & Backwards Compatibility |
There was a problem hiding this comment.
This might not be the best place, but generally think it would be really nice to add link/doc for the indexers on how to deploy and run Substreams alongside their existing subgraph services
|
|
||
| This builds on work done in [GIP-0056 Permissionless Payers](https://github.com/graphprotocol/graph-improvement-proposals/blob/main/gips/0056-permissionless-payers.md), which opened up an opportunity for such new data services to use the network for payment. | ||
|
|
||
| > INFORMATIVE: [TAP](https://docs.rs/tap_core/latest/tap_core/index.html) is not required for Substreams to honor the promises of the protocol. However, as TAP brings augmented trust minimization properties, it would be incorporated in future work. |
There was a problem hiding this comment.
We've renamed TAP to GraphTally! Could you update the name here?
There was a problem hiding this comment.
Can we talk a bit about how you're thinking we would use GraphTally here? Currently, the way things work is that gateways send signed receipts to Indexers. Indexers then send a batch of those signed receipts back to the gateway to get a signed RAV (Aggregate receipt) which Indexers then post onchain to release money from the escrow contract. It sounds to me like with substreams, Indexers and Consumers talk directly, so gateways aren't really generating receipts. Is there an inversion of trust that happens here compared to the subgraph flow? Payment gateways trust Indexers are accurately reporting usage?
|
|
||
| Future opportunities would include self-serve APIs for Indexers to claim payment and do the dance automatically. Thus an Indexer would be able to lower the risks of not receiving payments from the Gateway. | ||
|
|
||
| Eventually, TAP could be used to send payments alongside blockchain data payloads (BlockScopedData), or some other trust-minimized payment mechanism. |
There was a problem hiding this comment.
Another instance of TAP to rename to GraphTally!
|
|
||
| ## Economic Security and Disputes | ||
|
|
||
| Economic security is achieved through slashing and disputes, similar to subgraphs as of Jan 24th 2025. Indexers providing incorrect data CAN have their stake slashed. Therefore, to be an Indexer recognized by the Payment Gateway, one must staked the Network token, just like for subgraphs. |
There was a problem hiding this comment.
Just a small typo here. "one must staked" -> "one must stake"
|
|
||
| Risks associated with data integrity and malicious actors will be mitigated through signed attestations, the arbitration process, and economic security enforced via slashing. | ||
|
|
||
| There are some centralization risks in the Payment Gateway until TAP is integrated. For instance, Indexers need to trust the Gateway for payment. In the same vein, there is some counterparty risk for Indexers, as they have to trust that the Payment Gateway _will pay_ - this can be mitigated by requesting payment early and often. |
|
|
||
| ## Payments | ||
|
|
||
| Payments are handled by the Payment Gateway, which is responsible for collecting payments from consumers and distributing them to indexers. |
There was a problem hiding this comment.
As per TG's style guidelines, "indexers" should be capitalized as "Indexers."
There was a problem hiding this comment.
Looking forward to seeing Substreams becoming a first-class product of the network!
Some GIP out-of-scope comments:
- I'd like to see a support strategy to ensure Indexer participation. There's been a lot of communication and support to indexers over the years for substream support, but integrating with the network will require additional steps and the indexers will require support from SF and/or Pinax to do so. I hope that existing providers will be ready to support competing newcomers.
- It is understood that this GIP is an interim solution. In addition to reducing trust assumptions with TAP/GraphTally, I'd like to see future iterations integrating a solution for inter-Indexer redundancy and geodiversity support. This is a strength of the current subgraph support that lacks here. Instead of having the network providing high availability, Consumer must rely on an individual Indexer for SLA and geo-diversity.
|
|
||
| ## Economic Security and Disputes | ||
|
|
||
| Economic security is achieved through slashing and disputes, similar to subgraphs as of Jan 24th 2025. Indexers providing incorrect data CAN have their stake slashed. Therefore, to be an Indexer recognized by the Payment Gateway, one must staked the Network token, just like for subgraphs. |
There was a problem hiding this comment.
This should be clarified to ensure that the slashing threat is credible. This being pre-Horizon, presumably the same 100k stake and 2.5% slash apply?
|
|
||
| Payments are handled by the Payment Gateway, which is responsible for collecting payments from consumers and distributing them to indexers. | ||
|
|
||
| > INFORMATIVE: The Graph Market acts as the first Payments Gateway. Such a gateway DOES NOT route traffic, but does route payments and receive consumption signals from Indexers. |
There was a problem hiding this comment.
Could we add a quick overall description of how consumption signals works, i.e. Indexer<>Gateway exchange. A link to external document would work.
|
|
||
| > INFORMATIVE: [TAP](https://docs.rs/tap_core/latest/tap_core/index.html) is not required for Substreams to honor the promises of the protocol. However, as TAP brings augmented trust minimization properties, it would be incorporated in future work. | ||
|
|
||
| After collecting consumption signals, the Payment Gateway MUST sum up all what is due to the Indexer, and transfer a payment to the Indexer's wallet, using the `collect` function in the staking contract. |
There was a problem hiding this comment.
There is a new trust assumption here. Because the Payment Gateway does not route requests, because there is no round-robin or comparison possible between Indexers, and because Consumption cannot be easily computed independently, we must trust Indexers to report consumption fairly.
What are the Payment Gateway, or Consumer, recourse if Indexers abuse of their position?
Addition of GIP-0083, to bring Substreams on the network officially.