Incorrect attributes added to nimrod format radar data on load

[cgsandford]

🐛 Bug Report

Nonsensical attributes are added to radar data when reading from nimrod file format

How To Reproduce

Steps to reproduce the behaviour:

1. Retrieve a nimrod file from the radar MASS or FRASIA archives (or I can provide a test file on request)
2. Read it in using iris.load_cube
3. In addition to useful attributes, attributes "recursive_filter_iterations" and "probability_period_of_event" have been added. These are not relevant except to IMPROVER and I suspect this may come from a fix made by @MoseleyS to the nimrod reader at some point.

Expected behaviour

Not having attributes added by iris that are not relevant to the data.

Environment

• OS & Version: RHEL7 (VDI)
• Iris Version: 3.1.0 (error is new since 3.0.1)

Additional details

The attributes I get from reading in a random hourly accumulation file are as follows:

{'nimrod_version': 2,
 'field_code': 214,
 'recursive_filter_iterations': 94,
 'probability_period_of_event': 0,
 'source': 'Radar composite\x00\x00\x00\x00\x00\x00\x00\x00\x00',
 'title': 'Unknown',
 'institution': 'Met Office'}

So there is also a formatting error with "source" that was also present in iris version 3.0.1, that needs to be fixed as well.

[MoseleyS]

Thanks @cgsandford for reporting this. These attributes come from the data-specific header section of the Nimrod data format, which have different meanings for different data (as described in the data format document). We need some method for differentiating the two meanings so that we don't describe the data incorrectly.
In the mean time, it is possible to rename these attributes on load:

def radar_metadata_callback(cube, field, filename):
    """Update specific attribute names"""
    for old_name, new_name in {
        "recursive_filter_iterations": "additional_radar_sites_bits",
        "probability_period_of_event": "calibration_type_code",
    }.items():
        if old_name in cube.attributes.keys():
            cube.attributes[new_name] = cube.attributes.pop(old_name)

cube = iris.load_cube(nimrod_filename, callback=radar_metadata_callback)

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[trexfeathers]

I'm assuming this is actually still relevant. Just not something that has risen to the top of the work list yet.

It may be a good time to highlight that we are actively encouraging Iris plugins. Loading code is especially well suited to this. We are not expecting anyone to write and manage a Nimrod Iris loading plugin, but I just wanted to advertise that this is now possible. Especially useful when waiting for the Iris core developers is not an option.

[SGallagherMet]

Just commenting to say this is very much still relevant.

I recently identified a problem merging cubes in a deprecated bit of software that was still on iris 2 due to the 'more_radars_bitmask' attribute and a colleague in the Radar team was able to identify what was going on very quickly.

If I'd been asking about the "recursive_filter_iterations" attribute it probably would have taken us a bit longer.

[stephenworsley]

@SciTools/peloton Unfortnately nimrod falls outside our teams usual area of expertise. If there is some specific logic you would want implemented in the nimrod loading do let us know, otherwise we're somewhat at a loss for how to resolve this.

[MoseleyS]

The issue is with the Data Specific section of the Nimrod header. The documentation shows that there are multiple tables that could be followed and I believe only table 2 has been implemented in the Iris loader. I'm not sure where to find a public version of this document, so I will email it to @stephenworsley separately.
Perhaps the best solution is a keyword selector for the loader specifying which table should be used.