Time series files metadata

[brey]

For efficiency reasons, local 1-D time series data are store in a single file per location in modern data formats such as parquet. The location could be an in-situ observation point or a model simulation output on a specific node of the mesh.

It would be useful if these files include appropriate metadata that provide enough info for visualisation (e.g. on a dashboard) or for validation purposes.

What could be the minimum info required?

[JackReevesEyre-NOAA]

• model name and version
• latitude and longitude
• datum information (if applicable)
• [models only] something like cell_methods -- describing whether the time series is a single node, average of some number of nodes, etc.

CF conventions might provide some useful guidance too:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#_attributes
and:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#discrete-sampling-geometries

[brey]

@tomsail, I see some duplication above. There is no need to report the model/solver twice.

More importantly there are 2 different kind of time series. The model data which do/don't refer to observation data (in-situ/sat) and the obs time series.

• case 1: Sim data

{
    "standard_name": "sea_surface_height_above_mean_sea_level",
    "long_name": "Storm surge water level",
    "units": "m",
    "featureType": "timeSeries",
    "cf_role": "timeseries_id",
       
    "vertical_datum": "MSL",
    "horizontal_datum": "WGS84",
    
    "model_name": "TELEMAC-2D",
    "model_version": "v9.0",
    "cell_methods": "point",
    "spatial_representation": "single_node",
    "node_id": 21165,
    "lon": 144.32,
    "lat": 44.05
}

• case 2: obs data

 {
  "lon": 144.29,
  "lat": 44.02,
  "standard_name": "sea_surface_height_above_reference_ellipsoid",
  "long_name": "Sea surface height",
  "water_level_type": "storm_surge",
  "tidal_component": "excluded",
  "description": "Storm surge component only. obtained with UTIDE",        
  "units": "m",
  "source": "ioc",
  "platform_id": "abas",
  "platform_name": "Abashiri",
  "instrument": "rad",  or "radar"??
  "country": "Japan",
  "featureType": "timeSeries",
  "cf_role": "timeseries_id"
}

Note that lat/lon values are not the same even when the simulation data are referred to the observation data (stations).

The real question is how to link the sim data to the obs data when they are indeed related (stations). One can leave it as is and use lat/lon to infer the connection or add the obs data, or an appropriate subset, as another dictionary in the simulation data i.e.

{
    "standard_name": "sea_surface_height_above_mean_sea_level",
    "long_name": "Storm surge water level",
    "units": "m",
    "featureType": "timeSeries",
    "cf_role": "timeseries_id",
    
    "water_level_type": "storm_surge",
    "tidal_component": "excluded",
    "description": "Storm surge component only, excludes astronomical tides",
    
    "vertical_datum": "MSL",
    "horizontal_datum": "WGS84",
    
    "model_name": "TELEMAC-2D",
    "model_version": "v9.0",
    "cell_methods": "point",
    "spatial_representation": "single_node",
    "node_id": 21165,
    "lon": 144.32,
    "lat": 44.05

    "reference" : 
        {
          "lon": 144.29,
          "lat": 44.02,
          "standard_name": "sea_surface_height_above_reference_ellipsoid",
          "long_name": "Sea surface height",
          "water_level_type": "storm_surge",
          "tidal_component": "excluded",
          "description": "Storm surge component only. obtained with UTIDE",        
          "units": "m",
          "source": "ioc",
          "platform_id": "abas",
          "platform_name": "Abashiri",
          "instrument": "rad",  or "radar"??
          "country": "Japan",
          "featureType": "timeSeries",
         "cf_role": "timeseries_id"
       }
}

Another issue is how to deal with multivariable parquet files. Then the description is not unique anymore.

We could add

{
    "variables" :  {
                          "elev" : {
                                        "standard_name": "sea_surface_height_above_mean_sea_level",
                                        "long_name": "Storm surge water level",
                                         "units": "m"
                                         },
                           "T" :     {
                                        "standard_name": "sea_surface_lever_temperature",
                                        "long_name": "Sea Temperature",
                                         "units": "degrees C"
                                         },
                           "S" :     {
                                        "standard_name": "sea_surface_lever_salinity",
                                        "long_name": "Salinity",
                                         "units": "..."
                                         },
         
                   },             
    "featureType": "timeSeries",
    "cf_role": "timeseries_id",
       
    "vertical_datum": "MSL",
    "horizontal_datum": "WGS84",
    
    "model_name": "TELEMAC-2D",
    "model_version": "v9.0",
    "cell_methods": "point",
    "spatial_representation": "single_node",
    "node_id": 21165,
    "lon": 144.32,
    "lat": 44.05
}

[JackReevesEyre-NOAA]

I agree that allowing multivariate parquet files is helpful. I think your metadata schema looks good, @brey , but I suggest moving the datum attributes to be variable-wise, instead of for the whole file.

[JackReevesEyre-NOAA]

The question of how to relate model time series to another location (usually a station) is interesting. I think the "reference" attribute seems like a good idea. But I think it would better to include only the minimal number of attributes in that - lat, lon, and maybe one or two name-based attributes.

[JackReevesEyre-NOAA]

And quoting @pmav99 message about attributes in data frames and parquet files:
"So to handle real roundtrips you need extra, application specific code to cast back to the original types (i.e int, float, Timestamp etc)."

I think this is basically what I was hacking together a couple of months ago! Coming from climate data world (netCDF files and xarray in memory, particularly) it is frustrating that pandas and parquet don't allow arbitrary and persistent metadata. I guess it's a cultural kind of difference with database world, where there are fixed schema etc.

[pmav99]

Well another option is to ditch parquet files (i.e. 1-D file formats) and stick with N-dimensional formats (e.g. zarr) instead. For the record, we wouldn't be the first ones to do that. Quoting from here:

Since Parquet performed the best in nearly all of the categories of Comparison 2, you may be wondering, “why didn’t you use Parquet to store all of the discharge data instead of Zarr?” The answer to that is flexibility. Zarr is a more flexible format than Parquet. Zarr allows for chunking along any dimension. Parquet is a columnar data format and allows for chunking only along one dimension. Additionally, Xarray’s interface to Zarr makes it very easy to append to a Zarr store. This was very handy when we were making all of the NWIS web service calls.

The chunking along any dimensions is probably not a significant factor in this case, since AFAIK we only deal with 1-D data, but appending efficiency should be considered, too. That being said, at least on version 2, zarr didn't allow for chunks of variable length which can be problematic, too.

Disclaimer: I haven't checked zarr v3 yet, so it might have improvements

[JackReevesEyre-NOAA]

FYI @brey @pmav99 @tomsail -- I have implemented some basic metadata in seanode (e.g., https://github.com/oceanmodeling/seanode/blob/d4bff7ecd0b80683315224a6ddec6f7f6ceb63b8/seanode/analysis_task.py#L147) and my STOFS event dashboard.

It uses pandas DataFrame's attrs property. This appears to persist, copy, and be saved out in parquet files, as well as decoded from parquet files written in that way.

My basic schema is:

df.attrs = {
    'DataFrameMetaData':{
        'example_metadata_variable':'some information'
    },
    'ColumnMetaData':{
        'var1':{
            'units':'whatever'
        },
        'var2':{
            'units':'metric_whatever',
            'datum':'MLLW'
        }
    }
}

[JackReevesEyre-NOAA]

And note that this is more about the organization and backend for the metadata, rather than what kinds of things should be included. Your earlier messages above do a good job of that already.