Belgium DSMR does not parse correctly

[rvdbreemen]

Hi Matthijs,
I have seen reports that belgium smartmeters do not parse correctly using the 5.0.2 dsmr parsing. This is due to the extention made by the Belgium netbeheerder.

The document for Belgium can be found here

I will be looking into making changes to the library to enable parsing of the DSMR of Belgium.
Robert

[matthijskooijman]

Interesting! Would be nice to also support the Belgian standard, looking forward to see what you come up with :-)

[rvdbreemen]

Found the differences:

At least the OBIS numbers differ, from Belgium to the Dutch standards.
Minimal difference, but it will confuse the parsing. Version number P1 and gas_delivered.

How to tackle this best I wonder?

[matthijskooijman]

I haven't looked in detail (bit busy atm), but if it is just the OBIS numbers and the associated formats that are different while the telegram format itself is the same, this might be a matter of defining some extra fields for them? Using a namespace belgium, you could end up with e.g. a field belgium::gas_delivered rather than just gas_delivered in your struct definition?

[rvdbreemen]

Yes, so far it really looks like just some extra OBIS codes and fiels. In belgium they have gas meters that are not temperature corrected, so they legally needed to use different OBIS codes it seems.

[rvdbreemen]

My first attempt at fixing this for Belgium users is to define the OBIS codes. And get some help getting it validated. Before I try to understand how to correctly extend using your suggestion of namespaces in the code.

[rvdbreemen]

@matthijskooijman we have a working field.h for belgium (confirmed by belgium user). I can share the fields.h (for belgium situation). I looked at your library, but I am wondering how best to integrate a second fields.h with an alternative for the OBIS parsing.

I realize you are busy, but it would be great to update your library with the belgium dsmr specification fields. I attached the fields.h in the ZIP in this comment.

fields.zip

[rvdbreemen]

Suggested a PR, added namespace for belgium in fields.h (see pull request #29 )

[matthijskooijman]

Let's keep this issue open until it is actually solved.

I read a bit through the Belgian specification, still playing around with it a bit to figure out the best way to support this. It seems that almost all fields are identical, just that the belgian spec adds some fields, so I think we can support this without duplicating all fields.

Anyway, what I really wanted to ask: @rvdbreemen, @pdemeyer, does either of you have some example telegrams for testing? I looked at the e-MUCS-H spec, but there are none. In #29 there is a voorbeelden.txt, but it is empty?

[matthijskooijman]

I've looked closely at the belgian spec, and wrote a bit to put into the README later (added at the bottom of this command). As for implementing this, I'm just thinking out load here.

Overall, it is just a matter of adding extra fields.

Gas value

The two different gas meter fields have different ids, so that would result in two different fields. However, you would probably want to refer to them under a single name. This could be be implemented by defining both fields with the same name in different namespaces (e.g. use gas_delivered for the default DSMR versoin and emucs::gas_delivered to include the e-MUCS version in your parsed telegram definition).

But then you need to choose which one to use at compiletime, you cannot put both of them in the same struct if they have the same name. We could then decide to define them separately with different names (e.g. gas_delivered_corrected and gas_delivered_not_corrected), but then when you need the gas value, you would need to check both fields to see which is present (which would work, but could be a bit cumbersome).

One alternative (not sure how to implement that yet, though) would be to allow specifying multiple Obis IDs to a single gas_delivered field definition. Then if either of the IDS is found, the associated value is stored in this field. If both are present (should never happen), this would be handled as a duplicate field with an error.

You could then still also offer two distinct fields (so you can choose at compile time) in case you want to parse one but not the other, or both and know which one was found.

I just realized (looking at the DSMRloggerAPI code with DSMR-3 support) that the gas_delivered value in DSMR-3 actually uses a different ID than the wone in DSMR-4. Looking at the comments in my code, I previously only noticed that the description is different (5-minute vs hourly value), but the ID is also different. This means that everything that applies to the eMUCS gas value above, also applies to the DSMR-3 gas value equally.

Limiter threshold

This field was originally specified in DSMR3, then removed in DSMR4 and now re-added in e-MUCS. However, DSMR3 specified a unit of A, while eMUCS specifies a unit of kW. DSMR as it is now strictly checks the unit in the telegram and fails to parse when the unit is different from what was expected. Again, this could be solved by splitting it into two fields with the same ID. You then have to choose either one at compiletime, and cannot support DSMR3 and eMUCS with the same struct. Since this is a bit of a specific field, that might not be relevant, but it would be nicer to support this, but I'm not sure how yet.

[matthijskooijman]

Oh, here's the README addition I mentioned above:

Differences between Dutch and Belgian meters

e-MUCS H spec v1.3, based on DSMR 5.0.2 P1, except:

• Telegram version number has a different format, so uses a different
  ID.
• Adds additional M-bus device ID field, apparently because the first
  field is apparently supposed to onlly contain the "fabrication
  number" (e.g. I think that means serial number, but not model number
  or version numbers).
  than 96 characters to identify themselves).
• Adds gas measurement without temperature correction. ID is different,
  values should be encoded the same.
• Re-adds gas valve position and breaker/switch position (removed in
  DSMR4). IDs are identical to DSMR3, but DSMR3 defines values as
  "on/off/released" and e-MUCS defines "Disconnected/Connected/Ready for
  reconnection".
• Re-adds limiter threshold field (removed in DSMR4). Again, IDs are
  default, but DSMR3 defined this in Ampere, while e-MUCS specifies kW.
• Adds "Fuse supervision threshold (L1)" field this field is specified
  in Ampere.
• Re-adds a "consumer message code" field (removed in DSMR5). The ID is
  the same, but the max length is increased from 8 numeric digits to
  128 characters, and where DSMR4 talks about displaying the message on
  the display, e-MUCS only talks about sending messages to home
  automation or energy managment (CEMS) systems attached to P1.

Some of these changes (gas reading and mbus device ids) refer to the
"IDIS pack2" specification, implying that the DSMR specification
actually uses some values wrongly (but the differences seem to be minor,
and should not influence the parsing of fields, though maybe the
interpretation of them).

[matthijskooijman]

I also wonder about the updated "version information" field. AFAICS, the eMUCS H spec has no info at all about what valus that field should have (unlike DSMR, which specifies what version number should appear in that field). Anyone see any documentation or examples of this field?

[rvdbreemen]

@matthijskooijman just added some examples in the voorbeelden.txt of telegrams, two meters.

[rvdbreemen]

I read through your comments. And you are coming to most of the conclusions I came too. I made it work using a doubling of the field definitions and made a switch to implement and get some field testing going. Users of Willem's hardware from Belgium now used it twice.

As you see in the code the difference are very limited. Yes, there is some specification difference as you point out. Like the protocol version ID is suddenly 5 digits instead of 2.

Like Willem, I think it would be useful to enable the library to parse more dynamically. When setting up the parser you should able switching the MBUS devices around from default. And also the protocol version.

To limit space usage you could opt for a compile time switch, but I think since the dfference are so very small a more dynamic approach would be better.

[rvdbreemen]

On the comment to parse the fields to either gas_delivered and gas_delivered_not_corrected is a valid discussion. Uncorrected gas measurements are used in Belgium. Normally in the Netherlands, we do correct for temperature, but it seems to be more explicit in Belgium. That's why I did not bother to make a new field in my version. I later did notice the difference, functionally there is no difference, but it would be good to have field alternatives.

However, for real-world implementations like Willem's DSMRlogger, you could say it complicates the situation where it is really not needed. So yes, it would be good to reflect the spec more correctly, but let's keep in mind too that the fields are used in real-world situations too.

[matthijskooijman]

Thanks for the examples. Looking at the version number:

0-0:96.1.4(50213)

That probably means DSMR 5.0.2, e-MUCS H 1.3

Also, the limiter threshold is actually in kW, as documented:

0-0:17.0.0(999.9*kW)

And in DSMR-3, it is actually in A as well (from #22):

0-0:17.0.0(999*A)

[rvdbreemen]

Yes, true, I think you are right. V5.0.2-1.3. That does make sense.

So it differs from version to version... that's simple to cover, using different field specfications per protocol versions.

[hemeleerse]

Hello, here's a Belgian user trying to contribute :-)
I found the eMUCs P1 version 1.4 that answers some of the questions raised (e.g. version information)

e-MUCS_P1_Ed_1_4.pdf

[hemeleerse]

The parser gives an 'unknown field' error on this part of the data : 1-0:31.4.0(999*A)
I tried to add this to fields.h and fields.cpp but still get the same error.
Would appreciate some help as I'm not an expert in these matters.

[hemeleerse]

Here's an example of P1 output:
BE-Fluvius-slimme-meter.txt

[erhapp]

Here's an example of P1 output:
BE-Fluvius-slimme-meter.txt

Please forgive me my off topic question.

@hemeleerse I started looking into the workings of my own Belgium meter yesterday and I'm wondering about the circuit you are using to read the P1 values. Are you willing to share your circuit?

[hemeleerse]

@erhapp this article will help you : https://opencircuit.be/Blog/Slimme-meter-uitlezer

It’s based one the ESP-01 module with ESP8266 processor

[rvdbreemen]

@erhapp or you could buy the DSMR logger print: https://opencircuit.be/Product/Slimme-meter-uitlezer-V4.5-Geassembleerd

[keukenrol]

I've just tried the example "read.ino" on a teensy 2.0.
As output it only seems to parse missing items, no actual data is shown.
Is this linked to the standard not being supported? The condition if (reader.parse(&data, &err)) never gets met. I tried this by blinking a LED if it matches this condition.
If i directly use serial.print i get version number 50215 (Sagecom T211 Belgium)

[karelv]

Any update for Belgium?

[matthijskooijman]

No progress on my part, I haven't found any time to work on this project yet...

[karelv]

Thanks for your heads up!

FYI:
Mr Wheel (Willem Aandewiel) created: https://github.com/mrWheel/arduino-dsmr-be

However it did not compile on Teensy 4.1: https://github.com/mrWheel/arduino-dsmr-be/issues/5