Skip to content

Add implementation recommendation for metadata #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fryorcraken merged 10 commits into from
Jul 31, 2025
Merged

Add implementation recommendation for metadata #168

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4f31630
Add implementation recomment from metadata
fryorcraken Jul 22, 2025
33bc252
libp2p peer id
fryorcraken Jul 24, 2025
cb98db4
typo
fryorcraken Jul 24, 2025
69fe345
typo
fryorcraken Jul 24, 2025
17386a1
typo
fryorcraken Jul 24, 2025
d55137a
clarification
fryorcraken Jul 24, 2025
0e452f3
Incorporate feedback
fryorcraken Jul 30, 2025
d58b1f6
clarify connection logic
fryorcraken Jul 30, 2025
ef61c97
Fix lint
fryorcraken Jul 30, 2025
d393c19
Fix lint
fryorcraken Jul 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 56 additions & 7 deletions waku/standards/core/66/metadata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ slug: 66
title: 66/WAKU2-METADATA
name: Waku Metadata Protocol
status: draft
editor: Alvaro Revuelta <alrevuelta@status.im>
editor: Franck Royer <franck@status.im>
contributors:
- Filip Dimitrijevic <filip@status.im>
- Filip Dimitrijevic <filip@status.im>
- Alvaro Revuelta <alrevuelta@status.im>
---

## Abstract
Expand All @@ -15,16 +16,19 @@ that can be associated with a [10/WAKU2](/waku/standards/core/10/waku2.md) node.

## Metadata Protocol

Waku specifies a req/resp protocol that provides information about the node's medatadata.
Such metadata is meant to be used by the node to decide if a peer is worth connecting
or not.
The keywords “MUST”, // List style “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”,
“NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).

Waku specifies a req/resp protocol that provides information about the node's capabilities.
Such metadata MAY be used by other peers for subsequent actions such as light protocol requests or disconnection.

The node that makes the request,
includes its metadata so that the receiver is aware of it,
without requiring an extra interaction.
without requiring another round trip.
The parameters are the following:

* `clusterId`: Unique identifier of the cluster that the node is running in.
* `shards`: Shard indexes that the node is subscribed to.
* `shards`: Shard indexes that the node is subscribed to via [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md).

***Protocol Identifier***

Expand All @@ -48,6 +52,51 @@ message WakuMetadataResponse {
}
```

## Implementation Suggestions

### Triggering Metadata Request

A node SHOULD proceed with metadata request upon first connection to a remote node.
A node SHOULD use the remote node's libp2p peer id as identifier for this heuristic.

@Ivansete-status Ivansete-status Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A change in the node's state should also trigger a metadata request to other peers.

Suggested change
A node SHOULD proceed with metadata request upon change of any of its capabilities, such as the supported `cluster-id` or `shard` set.

@jm-clius jm-clius Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a heavy requirement as it will likely affect all connections. We don't want a single subscription change to result in 300+ metadata requests, the vast majority of remote peers who may not even be affected by this change. I don't think it's necessary and we can leave it up to the remote parties to eventually get to know this information (with TTL and metadata-on-error implemented).

@Ivansete-status Ivansete-status Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment @jm-clius !
I lean more to an event-driven approach, i.e., when something happens, this state change is spread everywhere has interest about it. That way will will have more real-time consistency.

Also, if someone configures a TTL==5 minutes, as an extreme example, then we might have metadata requests (+300) that are not needed because nothing changed.

IMHO, having the TTL concept will lead to more unneeded (+300) requests than having an event-driven approach because when a node starts with certain configuration, it is more likely to always run with that configuration forever instead of changing it every minute/hour/day. But I may miss something of course :)

@fryorcraken fryorcraken Jul 30, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lean more to an event-driven approach, i.e., when something happens, this state change is spread everywhere has interest about it.

We have to weigh in the implication. Yes, I agree event-driven approach within a given node, but not systematically when it implies network request.
In this case, a local-first, or lazy-push approach is preferred.

Also, if someone configures a TTL==5 minutes, as an extreme example, then we might have metadata requests (+300) that are not needed because nothing changed.

Note that 5 minutes is not the recommended TTL, but it's

It is RECOMMENDED to set the TTL to 6 hours.

IMHO, having the TTL concept will lead to more unneeded (+300) requests than having an event-driven approach

Let's see how the software behave but I disagree with your assumption.

But I may miss something of course :)

A desktop application that make full usage of autosharding may change relay subscription depending on user needs.

While this would not be expected to happen every 10min, it could happen.

@jm-clius jm-clius Jul 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember that with decentralised network (and services) we don't expect everything to be consistent - we expect some peers to be misconfigured, some peers with buggy protocol behaviour, adversarial peers that try to trick us, etc. With metadata we can gain some probable information about peers' capabilities, but we should always assume that some of this information may be dishonest or outdated. A sensible heuristic for triggering metadata requests (e.g. with TTL, on error response, on connection) gives us more information, but never guarantees consistency. All our protocols should be able to deal with this ambiguity.

A node MAY proceed with metadata request upon reconnection to a remote peer.
Comment thread
Ivansete-status marked this conversation as resolved.

A node SHOULD store the remote peer's metadata information for future reference.
A node MAY implement a TTL regarding a remote peer's metadata, and refresh it upon expiry by initiating another metadata request.

@Ivansete-status Ivansete-status Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't quite see the need for that TTL concept if we perform metadata requests on first connection or reconnection.

@jm-clius jm-clius Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember that a node's metadata might change at any point. Metadata will eventually also be used to advertise more capabilities than just cluster + shard. It makes sense to assume that information like this may grow stale and needs to be refreshed from time to time.

@weboko weboko Jul 29, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what other capabilities?
I feel like any additional capability would require nwaku to restart - which means to re-connect to the network

@fryorcraken fryorcraken Jul 30, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what other capabilities? I feel like any additional capability would require nwaku to restart - which means to re-connect to the network

A remote node, like a Status Desktop app, may change the relay shard they subscribe to dynamically.

This sentence is here exactly for that: remember implementers that metadata information may change.

@Ivansete-status Ivansete-status Jul 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A remote node, like a Status Desktop app, may change the relay shard they subscribe to dynamically.

Then, having a TTL of 6h (or 1h) doesn't sound correct because if my Status Desktop app changes the shard dynamically, I want my app to start participating on that shard right away and not wait for TTL to expire. Then, an event-driven approach seems more convenient, instead of the TTL one, in this case.

@fryorcraken fryorcraken Jul 31, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want my app to start participating on that shard right away and not wait for TTL to expire.

What do you mean? What does "my app start participating" mean?

Then, an event-driven approach seems more convenient, instead of the TTL one, in this case.

So you are saying, if let's say a Status Desktop app that was doing relay on shard 1, stops to do so, then it should go through all the peers it has in its peer store (100s), and trigger a metadata request? do see you how this is not reasonable to have such massive network event triggered?

What we are saying is that the TTL + error handling is enough, because it's lazy.

  1. Alice discovers Bob
  2. Alice connects to Bob, do a metadata request and learns Bob is running relay on shard 2
  3. Alice disconnect to Bob, because we don't maintain connections by default
  4. Alice wants to send message on shard 2
  5. Alice re-connect to Bob
  6. Alice sends light push request to Bob for message on shard 2
  7. Bob stops subscribing to shard 2
  8. Alice wants to send new message on shard 2
  9. Alice reconnects to Bob
  10. Alice sends light push request to bob for message on shard 2
  11. Bob rejects, "Shard not supported"
  12. Alice connects to alternative node.

In the current recommendation, we suggest:

A node MAY implement a TTL regarding a remote peer's metadata, and refresh it upon expiry by initiating another metadata request.

A node MAY proceed with metadata request upon reconnection to a remote peer.

A node MAY trigger a metadata request after receiving an error response from a remote note stating they do not support a specific cluster or shard.

Which means:

At (5), and (9), we do not recommend a metadata request, before the light push request, every time.
If we were to recommend a metadata everytime we reconnect, it would mean we'd have 2 RTT every time we try to send a message over light push, adding necessary network calls to send a message

At (11), we recommend proceeding with a metadata request due to the error returned, so that Alice can avoid using Bob again for shard 2, while keeping Bob as a suitable service node for other shards (depending on metadata response).

At (7), we do not recommend Bob to initiate 100s of connection and metadata requests due to a local event.

It is RECOMMENDED to set the TTL to 6 hours.

@Ivansete-status Ivansete-status Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May I ask why 6h and not 2h or 20h? Just for learning

@fryorcraken fryorcraken Jul 30, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am happy to remove this recommendation as I am trying to infer too much on Chat sdk patterns.

@jm-clius jm-clius Jul 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's good to have an example number here. It means that we roughly think that, if implementing a TTL, it should be in the order of a few hours (as opposed to a few minutes or a few days). We can always adapt this number based on observation, but I think "friendly" specs can have our best current guesstimate, as long as it's merely a recommendation.

FWIW 4-6 hours is not an uncommon refresh period for network contexts. For example, this is more or less the interval a cellphone network will wait before refreshing the context of an idle phone.


A node MAY trigger a metadata request after receiving an error response from a remote note
stating they do not support a specific cluster or shard.
For example, when using a request-response service such as [`19/WAKU2-LIGHTPUSH`](/waku/standards/core/19/lightpush.md).

### Providing Cluster Id

A node MUST include their cluster id into their metadata payload.
It is RECOMMENDED for a node to operate on a single cluster id.

### Providing Shard Information

* Nodes that mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) MAY include the shards they are subscribed to in their metadata payload.
* Shard-relevant services are message related services,
such as [`13/WAKU2-STORE`](/waku/standards/core/13/store.md), [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)
and [`19/WAKU2-LIGHTPUSH`](/waku/standards/core/19/lightpush.md)
but not [`34/WAKU2-PEER-EXCHANGE`](/waku/standards/core/34/peer-exchange.md)
* Nodes that mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) and a shard-relevant service SHOULD include the shards they are subscribed to in their metadata payload.
* Nodes that do not mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) SHOULD NOT include any shard information

### Using Cluster Id

When reading the cluster id of a remote peer, the local node MAY disconnect if their cluster id is different from the remote peer.

@weboko weboko Jul 29, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand MAY but shouldn't we RECOMMEND to disconnect here?

@fryorcraken fryorcraken Jul 30, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use RECOMMEND when it can negatively impact the overall system. I don't think it matters that much here.


### Using Shard Information

It is NOT RECOMMENDED to disconnect from a peer based on the fact that their shard information is different from the local node.

Ahead of doing a shard-relevant request,
a node MAY use the previously received metadata shard information to select a peer that support the targeted shard.

For non-shard-relevant requests, a node SHOULD NOT discriminate a peer based on medata shard information.

## Copyright

Copyright and related rights waived via
Expand Down