[REST] New REST APIs to generate DSL syntax for items and things

[lolodomo]

Related to #4509

Added REST APIs:

• GET /file-format/existing/items to create file format for all existing items in registry
• GET /file-format/existing/item/{itemname} to create file format for an existing item in registry
• GET /file-format/existing/things to create file format for all existing things in registry
• GET /file-format/existing/thing/{thingUID} to create file format for an existing thing in registry
• GET /file-format/existing/thing-from-inbox/{thingUID} to create file format for an existing thing in discovey registry

All these APIs have a boolean parameter named "hideDefaultParameters" to hide or show the configuration parameters having the default value. They are hidden by default.

[lolodomo]

Here is an example of result for things:

Thing astro:moon:e2015c6f37 "Données Astro de Lune" @ "Paris" [geolocation="49.023461463214126,-1.1425781250000002"] {
	Channels:
		Type end : rise#end "Heure Fin" [offset=-1, forceEvent=true]
}
Bridge freeboxos:api:api "Freebox API" [appToken="xxxxx", httpsAvailable=true] {
	Thing active-player player-test "Player Revolution" [remoteCode="123456"]
	Thing player player4k "Player Mini 4k" [id=15]
	Thing revolution serveur-test "Serveur Revolution"
}
Bridge netatmo:account:compte "Netatmo" [clientId="yyyyy", clientSecret="zzzzz"] {
	Bridge weather-station indoor "Indoor" [id="11:22:33:44:55:66"] {
		Thing outdoor exterieur "Sonde extérieure" [id="22:33:44:55:66:77"]
		Thing rain pluviometre "Pluviomètre" [id="33:44:55:66:77:88"]
		Channels:
			Type co2-measurement : max-co2-hour "Max CO2 Last Day" [limit="MAX", period="1day"]
			Type co2-measurement : max-co2-week "Max CO2 Last Week" [limit="MAX"]
			Type co2-measurement : min-co2-hour "Min CO2 Last Day" [period="1day"]
			Type co2-measurement : min-co2-week "Min CO2 Last Week"
			Type noise-measurement : max-noise-hour "Max Noise Last Hour" [limit="MAX", period="1hour"]
			Type noise-measurement : min-noise-hour "Min Noise Last Hour" [period="1hour"]
			Type noise-timestamp : time-max-noise-week "Time Max Noise Last Week" [limit="DATE_MAX"]
			State Number : test-number [param1="Value 1", param2=true, param3=15, 20.5, 25.5, 30]
			Trigger String : test-string [titi=100, toto=true, false, true]
	}
}
Thing ntp:ntp:local "Local Time" [hostname="0.fr.pool.ntp.org", timeZone="Europe/Paris"] {
	Channels:
		Type string-channel : string "Date Heure" [DateTimeFormat="dd-MM-yyyy HH:mm:ss"]
}

[lolodomo]

And the result for items:

Group:Switch:OR (ON, OFF) DemoSwitchGroup [home-group]
Group X "Group X"
Group C "Group C" (X, U)
Group U "Group U" (X)
Color DemoColor (DemoSwitchGroup)
Contact DemoContact
DateTime DemoDateTime
Dimmer DemoDimmer (DemoSwitchGroup) { autoupdate="false" }
Number:Dimensionless DemoHumidity "Humidity [%.0f %%]" <humidity> (U, C) [Humidity, Measurement] { listWidget="" [action="", iconUseState=true], unit="%" }
Number:Dimensionless DemoHumidity2 "Humidity 2" <humidity> (U, C) [Humidity, Measurement] { listWidget="" [action="", iconUseState=true], unit="%" }
Image DemoImage
Location DemoLocation "Demo Location"
Number DemoNumber (U)
Number DemoNumber2 (U) { stateDescription="" [min=-10, max=20, step=0.5, pattern="%.1f"] }
Number DemoNumber3 "Demo Number 3 [%.1f]" (U) { stateDescription="" [min=-10, max=20, step=0.5, pattern="%.1f"] }
Number DemoNumber4 "Demo Number 4" (U) { stateDescription="" [min=-5, max=5, pattern="%.0f", readOnly=true] }
Number DemoNumber5 "Demo Number 5" (U)
Rollershutter DemoRollershutter
String DemoString (X)
String DemoString2 "My String 2 [- %s -]" (X)
String DemoString3 "My String 3 [= %s =]" (X)
String DemoString4 "My String 4 [+++ %s +++]" (X)
String DemoString5 "My String 5" (X)
String DemoString6 "My String 6 []" (X)
String DemoString7 "My String 7" (X)
Switch DemoSwitch "Demo Switch [%s]" <switch> (DemoSwitchGroup) [Control, Light] { matter="OnOffLight" [label="Lampe sur l\'étagère"] }
Number:Temperature DemoTemperature "Setpoint [%.1f °C]" <temperature> (C) [Setpoint, Temperature]
String test { commandDescription="" [options="0=value 0,1,2=Value 2"], stateDescription="" [options="0=Valeur 0,1=Valeur 1,2", pattern="%s"] }
Number test2 "Aujourd\'hui" <none> [Energy, Measurement]
Number:Pressure test_0_1 "Label Test" <if:noto-v1:cityscape-at-dusk> (DemoSwitchGroup, C) [Sensor] { commandDescription=" " [options="0=Valeur 0,1=Valeur 1"], ga="Lock" [inverted=true], unit="bar" }
Number:Pressure test_0_2 "Label , Test = 1" <if:noto-v1:cityscape-at-dusk> (DemoSwitchGroup, C) [Sensor] { commandDescription=" " [options="0=Valeur 0,1=Valeur 1"], ga="Lock" [inverted=true], unit="bar" }
String Donnees_Astro_de_Lune_Evenement_Periode "Événement Période" [Point] { channel="astro:moon:e2015c6f37:rise#event" [profile="system:trigger-event-string"] }
DateTime Donnees_Astro_de_Lune_Heure_Fin "Heure Fin [%1$td/%1$tm/%1$tY %1$tR]" [Point] { channel="astro:moon:e2015c6f37:rise#end" [profile="system:timestamp-offset", offset=30.0], channel="astro:moon:e2015c6f37:rise#start", autoupdate="false" }
DateTime Lune_Heure_Debut "Heure Début [%1$td/%1$tm/%1$tY %1$tR]" [Point] { channel="astro:moon:e2015c6f37:rise#start" }
Color MagicColor { channel="magic:color-light:magicColor:color" }
Dimmer MagicDimmer { channel="magic:dimmable-light:magicDimmer:brightness" }
Switch MagicOnOff { channel="magic:onoff-light:magicOnOff:switch" }

[lolodomo]

@kaikreuzer and @openhab/core-maintainers : for your information, the PR is now ready for review.

[lolodomo]

Go back to draft as I discovered a bug.

[lolodomo]

Problem with items sorting is solved. So ready again.

[spacemanspiff2007]

You would like to remove the API that is for me the most important and the most magic, the one that will allow for a user just by calling one API without providing anything to create a file format for all his items or things currently existing in the registry. This is an easy way for the user to move to the current file format or a future file format.

Fair point - things should be easy for the user.
I don't know you intend that things work because I'm not a java programmer and can't properly understand the code but wouldn't be generating stuff for the whole registry just be a matter of passing around responses?
e.g.

GET /items  >> POST /fileFormat/create/items-file
GET /things >> POST /fileFormat/create/things-file

What is your idea about resolving ambiguities?
E.g. if I have a TestItem in my installation and want to generate syntax for an item that is also named TestItem?

just by providing no thing data in body.

That would also work. If the body is empty ([]) the api could return the file format for the whole registry.

I'm still not a hundred percent sold on that because I think making requests and posting data between endpoints is cheap and it's something that's not done very often. The benefit is that there is a clear flow of data and an arbitrary input always produces the same output.

But if it's clear from the use of the endpoint that the item registry is getting accessed or that it's not getting accessed than I have no objections.

---

I prefer as it is now a query parameter to select the file format, rather than creating multiple sub-endpoints, one per format. It makes the API already OK for the future even if a new formats are introduced in the future.

This was my idea at first, too. However I then thought about how things will documented e.g. in api explorer and as far as I know there is currently no way to document the input/output data structure based on the query parameter.
E.g.
/file-format/create?format=thingsfile takes a different input than
/file-format/create?format=itemsfile
and I think it makes sense that the api documentation reflects that accordingly.

---

(You just wrote your response while I was typing the second paragraph out so I just left everything in)

Just a small suggestion: How about a sub-endpoint which returns the registry?
e.g.

GET  /file-format/existing/things
POST /file-format/create/things

[mherwege]

Would this allow us to get rid of the nearly parsers in the UI for items and things? These are hard to keep in sync with the xtext definitions in core.
And probably the hardest one is for sitemaps. I know this is not part of this PR yet, but I see potential applying the same principle to sitemaps. There is a lot of code doing parsing and generating DSL for sitemaps in the UI right now, which is a pain to maintain.

Combined with #4585, this would also open up an easier route to defining sitemaps in yaml, and switch between UI configuration, DSL and yaml in the UI.

One challenge I potentially see is showing errors in the UI. For sitemaps, as the parsing is done in the UI code, it will show (to a degree, and not always perfect) where the error in the sitemap DSL is. If the whole translation is done in core, the API response should be able to indicate where the error is as well, to be able visualize it in the UI. It shouldn't just be a log entry.

[lolodomo]

Fair point - things should be easy for the user. I don't know you intend that things work because I'm not a java programmer and can't properly understand the code but wouldn't be generating stuff for the whole registry just be a matter of passing around responses? e.g.

GET /items  >> POST /fileFormat/create/items-file
GET /things >> POST /fileFormat/create/things-file

Too much complicated to achieve by a lambda user.
For a developer, of course it is possible but it is two requests instead of one, a massive and useless additional network traffic generated between the client and the server.

What is your idea about resolving ambiguities?
E.g. if I have a TestItem in my installation and want to generate syntax for an item that is also named TestItem?

If the POST APIs are fully independent from our server registries, we have no conflicts to resolve. It is the role of the client to finally not request the creation of a new item/thing in the registry that already exists. I believe there is nothing new there.

Just a small suggestion: How about a sub-endpoint which returns the registry? e.g.

GET  /file-format/existing/things

That would be fine for me, Looks better than my proposal: GET /file-format/things-from-registry

[lolodomo]

Would this allow us to get rid of the nearly parsers in the UI for items and things? These are hard to keep in sync with the xtext definitions in core.

Yes, that is what we are discussing in a RFC I recently opened and that I finally started to embed in that PR.

And probably the hardest one is for sitemaps. I know this is not part of this PR yet, but I see potential applying the same principle to sitemaps. There is a lot of code doing parsing and generating DSL for sitemaps in the UI right now, which is a pain to maintain.

Yes, that was my intention to come to you in a second step to also consider sitemap. But I will do that in a next step and for that I would need first some explanations from you how it works today, what are the used APIs and how is triggered the code in UIComponentSitemapProvider that already generates DSL model. The syntax for sitemap is certainly more complex than the syntax for item or thing, but as we have the existing class UIComponentSitemapProvider with code I can simply reuse, it should be finally not too much complex, even maybe very easy ;)

Combined with #4585, this would also open up an easier route to defining sitemaps in yaml, and switch between UI configuration, DSL and yaml in the UI.

Yes, that is exactly my current intention started with items and things but that I can continue later with sitemaps.

One challenge I potentially see is showing errors in the UI. For sitemaps, as the parsing is done in the UI code, it will show (to a degree, and not always perfect) where the error in the sitemap DSL is. If the whole translation is done in core, the API response should be able to indicate where the error is as well, to be able visualize it in the UI. It shouldn't just be a log entry.

That is an interesting point, in my current (basic) implementation, the API that parses the DSL syntax will return a 400 status code and the errors are logged. One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI. But I will not be able to be more clever and more precise than what currently provides the core framework in logs when the DSL file is parsed.

[spacemanspiff2007]

Too much complicated to achieve by a lambda user.
For a developer, of course it is possible but it is two requests instead of one, a massive and useless additional network traffic generated between the client and the server.

I think the rest api users are only developers and even for large installations (>1k items) it's almost instant.
It's additional code that has to be created and maintained.
However if you want to provide the additional convenience that's fine with me. 🚀

It is the role of the client to finally not request the creation of a new item/thing in the registry that already exists. I believe there is nothing new there.

I fully agree 👍

One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI

That would be great. Would it then need to be logged at all?

[mherwege]

One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI. But I will not be able to be more clever and more precise than what currently provides the core framework in logs when the DSL file is parsed.

I think that’s a good start. We may then have to look at the errors generated by core at some point to refine the messages (e.g. the position of the error would always e useful if it can be found), but that’s a next step. Also the logs would benefit from improvements there.

[lolodomo]

I was certainly too much ambitious when I decided to try also covering #4585 in this PR.
I finally decided to come back to the original perimeter of this PR, that is the needs described in #4509 even if discussion in #4585 oriented the final implementation of this PR in particular with the creation of a new endpoint /file-format..

As a consequence, the PR is smaller and ready for review.

@kaikreuzer : that would be great if you could have a look.

For #4585, another PR will be created based on this one.

[lolodomo]

[lolodomo]

I don't know if I should or not keep the "language" parameter as it is not used by these APIs.

[mherwege]

I don't know if I should or not keep the "language" parameter as it is not used by these APIs.

I agree, not used at this time, so not needed. In the future, when we would support e.g. a YAML format format as well, we can still introduce it with a default value equal to text\vnd.openhab-dsl. Alternative would be to already annotate it with a default so there is no requirement to set it at this time. But it would be an unused parameter for now.

I would be keen to try to create a proper documentation of the generated response, rather than just string if achievable. But that can be a follow-up and doesn't need to be part of this PR.

[lolodomo]

I agree, not used at this time, so not needed. In the future, when we would support e.g. a YAML format format as well, we can still introduce it with a default value equal to text\vnd.openhab-dsl. Alternative would be to already annotate it with a default so there is no requirement to set it at this time. But it would be an unused parameter for now.

Language parameter is to provide a locale, not to select the output format.
For the output format selection I moved as you suggested to a media type combined with the Accept header. Another media type will be added later when a YAML format is added.

I will remove the language parameter.

[lolodomo]

I will remove the language parameter.

I will first check if it makes sense to get a localized version of a thing/item before generating DSL syntax.

[mherwege]

Language parameter is to provide a locale, not to select the output format.

Oops, sorry, of course. I was mixing up things.

[mherwege]

If you are interested, the code to convert to YAML is working in the prototype I created, and could easily be integrated here. See https://github.com/mherwege/openhab-core/tree/yaml

[lolodomo]

If you are interested, the code to convert to YAML is working in the prototype I created, and could easily be integrated here.

This PR introduces nothing related to YAML.
But I will consider you code in next PRs when YAML will be introduced.

[lolodomo]