Improve OBIS ID mappings in "main/src/dsmr_parser/fields.h"

[Boilerplate4u]

I might be wrong or have missed something, but after checking OBIS IDs for DSMR in several countries the mapping in the file [dsmr_parser/src/dsmr_parser/fields.h] seems incomplete and in some cases incorrect.

For example, the OBIS ID 1-0:3.8.0 is currently defined as total_imported_energy, but in some countries it is used as energy_reactive_delivered (reactive energy import) as the comment correctly explains: "Meter Reading Reactive energy delivered". The others are mapped using more correct names like reactive_energy_delivered_tariff1, 3, 4.

Is there a practical way to make this mapping more general and aligned with real-world usage? OBIS IDs are static by nature, so this only needs to be organized once and for all based on the current definitions.

One idea is to allow OBIS IDs as an alternative to the predefined names.

Ps..
Does anyone maintain a reference or documentation of the different OBIS IDs currently in use across countries?

[geduxas]

It should be referred to this

IEC 62056-61-2006.pdf - Read.pdf

[Boilerplate4u]

Yeah, but 62056-61 only defines the basic OBIS object format. It does'n define country-specific mappings or meaning variations so this somehow needs to be reflected in the current mappings in fields.h

These are the official DSMR v5 references:

1. IEC 62056-21 Electricity metering, Data exchange for meter reading, tariff and load control, Part 21 Direct local data exchange
2. IEC 62056-61 Electricity metering, Data exchange for meter reading, tariff and load control, Part 61 OBIS Object Identification System
3. IEC 60747-5-5 Electrical safety standard
4. IEC 61000-4-2 Electromagnetic compatibility, Part 4-2 Testing and measurement techniques, Electrostatic discharge
5. IEC 61010 Safety requirements for electrical equipment for measurement, control, and laboratory useut

We still need to figure out where each country defines its own OBIS interpretations for DSMR or DLMS/COSEM and then somehow fold those variations into the mapping.

[geduxas]

They're is not such thing as country specific, all obi's should be strict to standard, also there is manufacturer specific ID's which also could be tied to country specific, but all basic info should be strict to standard.

[geduxas]

Sorry, yes, wrong document i have attached..

[Boilerplate4u]

Well, I’m pretty sure the implementation differs from country to country as I explained in my first post. That’s why sellers of DSMR P1 gadgets need to specify which country IDs they support. I’ll get back with some examples later. I need to pay attention to my wife now.

But if you have any ideas on how to fix fields.h, please give it a shot.

[Boilerplate4u]

@geduxas: Instead of me dumping a bunch of unclear examples I think it is easier to spot lots of country specific OBIS ID mappings simply by looking directly in fields.h even though some of them might be misleading or wrong.

I will try to gather and summarize which countries use DSMR as a base and what their differences currently look like. It might take some time though...

One way to make things easier for an ESPHome DSMR user is to add OBIS IDs in the docs and include a reference to fields.h for country specific and meter specific OBIS IDs.

[geduxas]

Yes, i am sorry again for this.

I like the idea of country related config. Also it could take off load if it could be selected prior compile.. now each obis code should go through list to match it's filter.. and all unnecessary filter line add's extra to esp cpu load.. with large filter file, and large telemetry, esp8266 can't handle load.

I just wanted to suggest fact that all obis are standartized, and eatch obis group code already have it's meaning.. for example 1 in beginning is always be tied to electric meters. 0 or 1 (second) is memory bank.. later code is related to measurements or billing data.. phase, and it's value. So from OBIS code should be already known for what group it belongs, and it not required filtration at all.

[Boilerplate4u]

Yeah, sounds like a solid idea. Now we just need to figure out a simple way to implement it. Anyway, down below is a Python snippet that extracts all OBIS IDs from fields.h, which I’ll use later to put together as a supplement / apendix for the DSMR docs:

Output like this:

| Field Name                                         | OBIS ID                 |
|----------------------------------------------------|-------------------------|
| identification                                     | 255-255-255-255-255-255 |
| p1_version                                         | 1-3:0.2.8               |
| p1_version_be                                      | 0-0:96.1.4              |
| timestamp                                          | 0-0:1.0.0               |
| equipment_id                                       | 0-0:96.1.1              |
| energy_delivered_lux                               | 1-0:1.8.0               |
| energy_delivered_tariff1                           | 1-0:1.8.1               |
| energy_delivered_tariff2                           | 1-0:1.8.2               |
| energy_delivered_tariff3                           | 1-0:1.8.3               |
| energy_delivered_tariff4                           | 1-0:1.8.4               |
| energy_returned_lux                                | 1-0:2.8.0               |
| energy_returned_tariff1                            | 1-0:2.8.1               |
| energy_returned_tariff2                            | 1-0:2.8.2               |
| energy_returned_tariff3                            | 1-0:2.8.3               |
| energy_returned_tariff4                            | 1-0:2.8.4               |
.   .   .
.   .   .

Python snippet:

import re

def extract_constants(file_path):
    """Extracts constants like DSMR_GAS_MBUS_ID 1 from the header file."""
    constants = {}
    with open(file_path, "r", encoding="utf-8") as f:
        for line in f:
            match = re.match(r'#define\s+DSMR_(\w+)\s+(\d+)', line)
            if match:
                name = match.group(1)
                value = match.group(2)
                constants[name] = value
    return constants

def extract_define_fields(file_path):
    """Extracts DEFINE_FIELD(fieldname, ..., ObisId(...)) entries."""
    with open(file_path, "r", encoding="utf-8") as f:
        content = f.read()

    pattern = re.compile(
        r'DEFINE_FIELD\(\s*(\w+)\s*,\s*[\w:<>]+,\s*ObisId\(([^)]+)\)',
        re.MULTILINE
    )
    return pattern.findall(content)

def resolve_constants(obis_parts, constants):
    """Replace any constants in OBIS parts with their numeric values."""
    resolved = []
    for part in obis_parts:
        part = part.strip()
        if part in constants:
            resolved.append(constants[part])
        else:
            resolved.append(part)
    return resolved

def format_obis(obis_raw, constants):
    """Format OBIS ID in standard A-B:C.D.E notation (or fallback)."""
    parts = obis_raw.split(',')
    resolved = resolve_constants(parts, constants)
    if len(resolved) == 6:
        return "-".join(resolved)  # e.g. 255-255-255-255-255-255
    elif len(resolved) == 5:
        return f"{resolved[0]}-{resolved[1]}:{resolved[2]}.{resolved[3]}.{resolved[4]}"
    else:
        return obis_raw  # fallback if format unexpected

def print_aligned_table(matches, constants):
    """Print a nicely aligned Markdown table."""
    rows = [
        (fieldname, format_obis(obis, constants))
        for fieldname, obis in matches
    ]

    max_field_len = max(len(row[0]) for row in rows)
    max_obis_len = max(len(row[1]) for row in rows)

    header_field = "Field Name".ljust(max_field_len)
    header_obis = "OBIS ID".ljust(max_obis_len)
    print(f"| {header_field} | {header_obis} |")
    print(f"|{'-' * (max_field_len + 2)}|{'-' * (max_obis_len + 2)}|")

    for fieldname, obis in rows:
        print(f"| {fieldname.ljust(max_field_len)} | {obis.ljust(max_obis_len)} |")

if __name__ == "__main__":
    # Path to the downloaded fields.h file
    file_path = "fields.h"

    constants = extract_constants(file_path)
    fields = extract_define_fields(file_path)
    print_aligned_table(fields, constants)

[PolarGoose]

now each obis code should go through list to match it's filter.. and all unnecessary filter line add's extra to esp cpu load.. with large filter file, and large telemetry, esp8266 can't handle load.

It is a separate issue, in my opinion. It doesn't need to be this way. We can use std::unordered_map and look up the obis code instantly.

[geduxas]

It is

Thank you @PolarGoose, i am not developer, just pointing out where i got issues, and how i think it should work :)

[PolarGoose]

Thank you @PolarGoose, i am not developer, just pointing out where i got issues, and how i think it should work :)

Np. I just merged the changes that use std::unordered_map. The performance should be faster now. I don't know how much, but at least the amount of defined fields doesn't affect performance anymore.

[PolarGoose]

For example, the OBIS ID 1-0:3.8.0 is currently defined as total_imported_energy, but in some countries it is used as energy_reactive_delivered

Is there any difference between total_imported_energy and energy_reactive_delivered ? Or is it just two ways of saying the same thing?

Yeah, but 62056-61 only defines the basic OBIS object format. It doesn't define country-specific mappings or meaning variations, so this somehow needs to be reflected in the current mappings in fields.h

I looked at the 62056-61. I'm very skeptical about its usefulness. It is very vague. Probably 50% of the fields that we need are declared as "manufacturer-specific". Also, this document says nothing about the units or precision used.
It is more of a document for the smart meter manufacturers so that they can see which OBIS codes suit them best. But for the user, it doesn't help, because the document gives the manufacturer too much freedom.

Also, I'm not sure how much manufacturers actually follow this standard. For example, the gas_valve_position has code 0-1:24.4.0. But the standard says that anything related to Gas should start with 7, not 0.

[Boilerplate4u]

Is there any difference between total_imported_energy and energy_reactive_delivered ? Or is it just two ways of saying the same thing?

Yeah, they're different forms of energy. "energy" in everyday language means the form of electricity that the consumer benefits from and pays for (kWh), while "reactive energy" (kvarh) is a byproduct that private consumers are normally not charged for. For example, "reactive energy" is often generated if you have an AC or ground-source heat pump with a compressor, and you'll often see this as exported "reactive energy," meaning it flows back into your local grid.

Also, I'm not sure how much manufacturers actually follow this standard. For example, the gas_valve_position has code 0-1:24.4.0. But the standard says that anything related to Gas should start with 7, not 0.

I'm currently compiling OBIS codes for each country that has a customer interface. This is the kind of task where LLMs, especially Perplexity Pro, really excel in terms of performance, analysis, and formatting.

I'll follow up with examples. I'm considering submitting a PR with a sub-library for these codes in this repository and a refined variant as a basis for the Esphome DSMR documentation.

And once again, thank you so much for your excellent work on fixing DSMR for ESP-IDF! I'm using it right now.

Have a great weekend!

[geduxas]

@Boilerplate4u i could provide all obis used in Lithuania if it helps.