MQTT URI Design #292
Comments
|
Talked offline about MQTT URIs. Points:
There is still some discussion to do, but I hope I captured everything. Credits: @egekorkan and @lu-zero. |
|
Let me add that:
Some examples from the mqtt spec on topic names fed to a a commonly used rust parser here: This way if we map 1 query or topic to 1 form we can fit everything that is in mqtt as a plain uri w/out the need of escaping elements beside |
|
While looking for how the languages support empty fragments, we found that the URL Api in JS has a problem, I already reported it upstream whatwg/url#779. |
|
Continuing the discussion about whether having an ending I report some statements about the meaning of the fragment identifiers here for reference:
Even if we are under our custom scheme Unfortunately, this approach is very uncommon for MQTT users as they are used to writing topic filters without escaping. Plus, as suggested offline by @egekorkan, usually:
|
|
I would not use the |
|
I think in an MQTT URI scheme, the question mark is not really required as we don't have query parameters. I think, we have to either encode it, but as @relu91 and @egekorkan mentioned it is not common to do so for the MQTT community. Or Instead, we need to introduce an additional segment (e.g., "wildcard-segment") to represent the multi-level wildcard, a placeholder like "*" e.g., |
|
I just wondering why we not simple use the fragment part to share the MQTT topic: I played around with const parse = require('url-parse');
const validator = require('validator');
let uris = ['mqtt://mybroker.com:1883#/path/+/path/*/#', 'mqtt://mybroker.com:1883#$SYS/monitor/+']
for(let uri of uris) {
let parsed = parse(uri);
console.log(parsed.href);
console.log(" is valid URL?: "+ validator.isURL(uri, { protocols: ['mqtt']}));
console.log(" protocol: "+parsed.protocol);
console.log(" hostname: "+parsed.hostname);
console.log(" port: "+parsed.port);
console.log(" pathname: "+parsed.pathname);
console.log(" hash: "+parsed.hash);
}Sample output: The URI deserialiser of the MQTT binding just has to throw away the # at the beginning to get the origin topic. Did I miss something? |
It is not a violation, we can just state "If fragment is present use it as wildcard".
Also the
|
I understand that |
|
The browser semantics aren't binding to our use of url/uri/iri :) |
As I reported above, in my understanding this is not semantics of browsers but it is explained in the RFC itself:
See also in the introduction, that we bound to that interpretation of the fragment:
So I second the points made by @mahdanoura (regardless of the design we are going to choose - using a query or just a path). Just one clarification:
That is another good point. section 3.5 exampling also this concept of URI Fragment being a "client-side indirect referencing" mechanism. |
|
We had an offline discussion with @Jerady and @skobow from HiveMQ. I'm trying to summarize the discussion:
|
|
Here are some concrete Java code samples comparing using url encoded Evaluating fragments: final String mqttUrl = "mqtt://mybroker:1883/my/topic/#";
final URI uri = URI.create(mqttUrl);
final String scheme = uri.getScheme();
final String host = uri.getHost();
final int port = uri.getPort();
final String path = uri.getPath().replaceFirst("/", "");
final String fragment = uri.getFragment();
// evaluate if fragment is present and append multilevel wildcard if required
final String topic = fragment != null ? path + "#" : path;
final Mqtt5AsyncClient mqtt5Client = Mqtt5Client.builder()
.serverHost(host)
.serverPort(port)
.build().toAsync();
mqtt5Client.subscribeWith()
.topicFilter(topic)
.send();Using url encoded character: final String mqttUrl = "mqtt://mybroker:1883/my/topic/%23"; // multilevel wildcard as such is not human readable
final URI uri = URI.create(mqttUrl);
final String scheme = uri.getScheme();
final String host = uri.getHost();
final int port = uri.getPort();
final String path = uri.getPath().replaceFirst("/", ""); // evaluates to 'my/topic/#'
final Mqtt5AsyncClient mqtt5Client = Mqtt5Client.builder()
.serverHost(host)
.serverPort(port)
.build().toAsync();
mqtt5Client.subscribeWith()
.topicFilter(path)
.send();Even though url encoding is harder to read it is much easier to use and does not require any processing of the |
|
To formalize this: |
I propose something more complex but also more flexible and closer to MQTT semantics: Let me know if you don't like it. |
|
That is how I understand the MQTT spec with regards to Topic Names and Topic Filters (https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901241): A Topic Name identifies the information channel to which Payload data is published. The term "topic name" refers to publishing data. Topic filters indicate the Topics to which the Client wants to subscribe and can contain wildcards such as the single level wildcard So a topic can either be a That being said I think your approach @relu91 is mixing up two different concepts in MQTT (publish & subscribe). Further more I think this approach will also increase implementation complexity as topics would need to be re-constructed by iterating over parameters instead of just using the |
|
Fyi: I just found that wiki page from the mqtt.org wiki: https://github.com/mqtt/mqtt.org/wiki/URI-Scheme. It does not tell anything about the topic part though. |
Yes, this is pretty much how I understand the spec too, but perhaps in the design above, I wanted to clearly differentiate the duality of a topic. In my opinion, the path-only approach would confuse because, in practice, you can have a topic filter without any wildcard and then you don't know if you are filtering or using a topic name (although it is pretty clear when you also specify the mqtt packet type).
Yes, this is the point that I don't like but not because of mixing but rather the fact that we are licking "retival" information into the URI. Basically, it would violate RFC 1.2.2 as if with the @sebastiankb's design is ok, but in my understanding, it is possible to subscribe using multiple topic filters at once that this means that we don't support this use case? Or should we allow for multiple paths in the URI scheme?
Interestingly enough this goes along the current URI design where topic names and filters are expressed as additional information on the form. Even though I belive it would be better to update that with the proposals that we are making here. |
Ok, got your point here. Using your approach would rather implicitly encode the operation by the structure of the URI. In general the term "topic" is commonly used for topic names as well as topic filters. So I would not expect that some distinction based on these terms would be any obvious.
From my understanding a resource is a leaf in a topic tree referenced by a topic name. In the case of publishing data only on specific resource can be addressed. In case of subscribing multiple resources can be addressed. All that might just apply when the topic structure is build using resource oriented semantics rather than operational like come RPC based approach.
I think it would be great to have MQTT URIs being self containing. That would enable to trigger a certain operation by just passing a URi to a client w/o external dependencies. In the following I add another possible approach trying to be explicit and respecting MQTT options as well as connection parameters (which we haven't discussed yet at all): mqtt[s]://[username:password@]hostname:[port]/topic
?operation=pub|sub
[?identifer=clientId]
[?version=3|4]
[?qos=0|1|2]
[?payload=<BASE64 encoded payload>]
[?retain=true|false]
[?messageExpiry=3600]
... # other MQTT options Please share your thoughts about that approach. |
|
the |
My understanding is that you are only designing one topic for a specific purpose. For example, if you are addressing a single data source, you would mainly take a path-only approach (e.g., make sense when you have a property in TD context). If you are interested in multiple data sources, you would go with a wildcard (eg., make sense for readAllProperties). For me, both would be defined as "topic". If I understand @skobow correctly, there seems no real distinction between "topic" and "filter". |
Okay, maybe my knowledge here is still to limited and my thoughts have been to generic. But then of course there is no need to have those options as parameters to the URI.
Yes, correctly. Even though you can find this distinction in the specs usually people are are just talking about "topics". @sebastiankb what is meant with "TD context"? From what I understood the biggest challenge seems to be deriving the operation from the URI. Right? And as there is no such thing like the request verb in Http another mechanism needs to be defined. So the URI pattern could be like
This would explicitly name the operation and would ensure easy handling. final String mqttUrl = "mqtt://localhost:1883/my/sensors/sensorXYZ/temperature?operation=pub";
final URI uri = URI.create(mqttUrl);
final String scheme = uri.getScheme();
final String host = uri.getHost();
final int port = uri.getPort();
final Map<String, String> parameters = getParameters(uri.getQuery());
final String topic = uri.getPath().replaceFirst("/", "");
final var mqtt5Client = Mqtt5Client.builder()
.serverHost(host)
.serverPort(port)
.build()
.toBlocking();
mqtt5Client.connect();
if ("sub".equals(parameters.get("operation"))) {
mqtt5Client.subscribeWith()
.topicFilter(topic)
.send();
} else if ("pub".equals(parameters.get("operation"))) {
mqtt5Client.publishWith()
.topic(topic)
.payload("25°C".getBytes(StandardCharsets.UTF_8))
.send();
} |
In WoT we consider 3 abstract interaction models which we call Properties, Actions and Events. Properties are suitable to specify MQTT subscriptions. Actions can be used to describe MQTT publications. That is the reason why |
Yes exactly as I was pointing out in my previous comment is better to not mix the concepts here. (it is also enforced by the RFC)
Yup, I thought that the MQTT was more used on the concepts described in the Spec, but as I user, I also have to admit that usually the term is mixed up. Then the only thing left to decide is probably how (if) to support the use case when multiple topics need to be used in one MQTT subscribe packet. 🤔 Other than this the simple path-only approach should work. |
|
Using MQTT URIs in TDs in practice, I guess you would need to create one form for reading/subscribing and writing/publishing, right? Would this make the additional {
...
"properties": {
"foo": {
"forms": [
{
"op": "writeproperty",
"href": "mqtt://mybroker:1883//my/topic"
},
{
"op": ["readproperty", "observeproperty"],
"href": "mqtt://mybroker:1883/my/+/awesome/topic/%23" // <- Not allowed for writing
},
{
"op": ["writeproperty", "readproperty", "observeproperty"],
"href": "mqtt://mybroker:1883/my/+/awesome/topic/%23", // <- Only used for reading here
"mqv:topic": "/my/topic" // <- Possible override for writing
}
]
}
}
} |
|
For me, the first two In this context, I would simply call |
|
One thing to note is that while paths starting with double slash are valid, they are difficult to use in relative URI references. For example, base URI |
|
I need a specific example here: Lets have two topics: "my/first/topic" and "/my/second/topic" and the broker mqtt://mybroker:1883. In the TD you would use the base for the broker endpoint:
A propertyA would describe the first topic and uses
A propertyB would describe the second topic and uses
Applying URI arithmetic to the href of propertyA would result in "mqtt://mybroker:1883/my/first/topic", right? What happen with propertyB? Would the result look like this "mqtt://mybroker:1883//my/second/topic" or this "mqtt://my/second/topic"? |
So there is no problem or limitation here, just something to be aware of. (You can't just stick |
|
here what the specification says regarding resolving base + reference. Making people aware of this specific pitfall might be good, but I wonder if In any case it seems another item to consider when reworking |
|
If I understand correctly, pretty much any Unicode string that doesn't contain a null character is a valid MQTT topic, with only One difficulty might be that RFC 3986 places special meaning on |
|
If I understand this correctly, we have an issue here. In MQTT the topic "my/topic" and "/my/topic" addressing different resources. When we want to express "/my/topic" as URI reference in |
@JKRhb I think the whole purpose of the discussion is to get rid of those old form terms and properly use the URI semantic. We could still allow to use them for the same reasons that we allow modbus:unitID -> readability. But it should be clear that there is a one-to-one mapping and URI should be preferred. |
I would say that is a problem (even if it is probably a corner case). Good thing we pinpointed here, if we go on in this direction we need to explain the limitations and the topic that cannot be represented.
Yes, yet another limitation, but less problematic than the one above. Also, we are still not covering resources selected by multiple topics (topic filters). |
|
Ok, I'll summarize the results so far: |
We have agreed before on using the href for indicating the resource but it is not done in MQTT yet. We should come up with a URI Scheme design that also allows topics starting with /.
broker: broker.com
port: 1880
topic: /topiclevel1/topiclevel2
Some ideas:
We should read up on the URI/IRI syntax to make sure to not use invalid syntax.
The text was updated successfully, but these errors were encountered: