rfc: routing policies - #20
Conversation
| On bucket creation, after provisioning the bucket (space) with Sprue, Hilt | ||
| invokes `/routing/use` — signed with the tenant key, whose authority flows | ||
| from the bucket's root delegation — pointing the bucket at its region's | ||
| policy. |
There was a problem hiding this comment.
What would happen if Hilt would invoke /routing/use before the space creation?
There was a problem hiding this comment.
It's impossible - the space/bucket DID must be the subject of the invocation.
pyropy
left a comment
pyropy
left a comment
There was a problem hiding this comment.
Looks great to me! I have posted a few questions most of them coming from my inexperience working with UCAN's.
|
|
||
| **Routing policy** is an entity identified by a DID that owns a set of | ||
| storage-node DIDs — its _candidates_. It is a routing constraint for the | ||
| spaces that reference it, not a request that any particular node be selected, |
There was a problem hiding this comment.
It is a routing constraint for the spaces that reference it
Does this mean that there will be a routing policy property on the space? Would you mind explaining how can I find routing policy for a space?
There was a problem hiding this comment.
Good question, we could add a separate capability that returns the policy for a space or potentially augment our existing /space/info capability to return it.
|
|
||
| The success value is an empty object. | ||
|
|
||
| ### `/routing/get` |
There was a problem hiding this comment.
This is related to my previous question -- how can one find routing policy DID for a space?
| and MUST resolve the reference on every routing decision for a constrained | ||
| space. | ||
| - Changes require a `/routing/set` or `/routing/use` invocation before taking | ||
| effect, creating a window in which Sprue may route using outdated state. |
There was a problem hiding this comment.
If there's a routing policy property on the space would it be possible to include it in the space creation request?
There was a problem hiding this comment.
Ah, do you you mean provisioning? I don't think it would be set before a space is provisioned.
The flow I have in mind is that when a regional provider is registered with Hilt you provide the service's DID. At this time Hilt creates a policy with a single node (/routing/set on Sprue) and records the policy DID. Note creating a policy DID == creating a public/private key. The policy DID will be stored in the DB with the provider information and used when Hilt creates a bucket for any tenant of that provider (/routing/use on Sprue).
|
I think this probably the most solid proposal on the topic to date, more detailed review early next week |
| Specified in [dynamic routing candidates](2026-07-dynamic-routing-candidates.md), | ||
| to which this RFC is a mutually exclusive alternative. It requires no Sprue | ||
| state and changes take effect immediately, but the constraint binds only | ||
| invocations that carry it: enforcement depends on every writer supplying the | ||
| correct candidates on every write. |
There was a problem hiding this comment.
I am a little concerned that by making this mutually exclusive with dynamic routing candidates we've drop the problem that RFC was written to solve: Ingot is co-located with a specific Piri, and per-invocation candidates existed to keep the write on the local node.
A region-wide policy only guarantees in-region placement — with N candidates, some writes accepted by a given Ingot instance could still hop to a sibling piri, in the hot path, increasing PUT latency, assuming there is ever more than a single node.
Could we compose these two options? e.g Keep candidates on /blob/add as a preference within the policy? Sprue could then select from the intersection of the invocation's candidates and the policy's, failing with CandidateUnavailable when the intersection is empty. The policy could remain the enforced outer bound for all writers, and picky writers could keep their locality hint.
There was a problem hiding this comment.
Having reviewed this PR and #15, I think this is conceptually the right candidate for how we should handle routing.
Let's take S3/Ingot out for just a moment and consider the product at a high level, where we have arbitrary upload clients and arbitrary node destinations.
From a product standpoint I'd assert two things, which line up with this proposal:
- Shape of policy: Customers generally prefer a routing policy that expresses an intent (i.e. geographic region) over a policy tied to an exact set of nodes.
- Scope of application: Customers generally prefer routing police apply to bucket of data (i.e. space) rather than a set of clients or an individual upload.
Based on the above, this is the right first proposal. It enables the customer to express a high level routing intent for a set of data ("I want this bucket of data to only be stored in Germany"), but allows us to manage that what set of nodes actually serve Germany. While it doesn't handle migrating data when nodes change, it's possible to determine from the change in the policy and the spaces tied to the policy what needs to be moved from where to where.
The only thing not covered which is perhaps what @frrist is hinting at is tieing client IDs to spaces or routing policies. So if I create a bucket on us-forge-1.s3.filonecontent.com, I presumably want uploads to that bucket via eu-forge-1.filonecontent.come to fail, instead of get routed to Piri nodes in the US? I'm not sure.
hannahhoward
left a comment
hannahhoward
left a comment
There was a problem hiding this comment.
Approving but I think there is more do on replication routing see comment.
|
|
||
| #### Replication | ||
|
|
||
| The policy constrains replica placement as well: every replica node selected |
There was a problem hiding this comment.
this is sufficient for now, but I actually don't think it will be sufficient long term, and actually I think we'll eventually end up with a fairly expressive policy language that perhaps does some mix and match.
A very typical use case is "I want my main copy here, but I'd like a couple backups distributed globally"
There was a problem hiding this comment.
I will re-word so that this is left open ended.
5 participants
📖 Preview
Summary
Spaces are configured to use a routing policy which specifies the candidate nodes that can receive a blob. The policy can change over time but the reference to it does not.
It is an alternative affinity routing proposal that allows blobs uploaded to spaces to be routed to a particular storage node without having to specify said node in every invocation.