Skip to content

Incorporating non-datalogger data into processing pipeline #213

@bpbond

Description

@bpbond

@Fausto2504 @nickdward request :

Ben and Steph, for the AquaTroll’s specifically we have had issues in the past with data not migrating from the troll to the Campbell logger. This is also the case for some of our YSI Exo sondes in the surface water. In this case, the data is generally stored internally on the sondes and we can recover it.

So in some cases we have a complete dataset downloaded from the sonde, but the L1 automated pipeline would not be retrieving this data. Do we want to address this at some stage in the pipeline, or would that perhaps be considered L2 data? In other words, at what stage of the pipeline would it be desirable to say “looks like loggernet missed all of 2022, let’s replace it with the data downloaded directly from the sonde”?

I want to think through in detail how this would work. As an example, here are the fields in a datalogger "WaterLevel600A" file versus the sample raw file:

Datalogger file AquaTROLL raw file
TIMESTAMP Date Time
Aquatroll_IDA(1) Device Id (in file header)
Barometric_Pressure600A 16: Barometer (16) mmHg (22)
Temperature600A 13: Temperature (1) °C (1)
Actual_Conductivity600A 7: Actual Conductivity (9) µS/cm (65)
Specific_Conductivity600A 8: Specific Conductivity (10) µS/cm (65)
Salinity600A 9: Salinity (12) PSU (97)
TDS600A 12: Total Dissolved Solids (13) ppt (114)
Water_Density600A 11: Water Density (14) g/cm³ (129)
Resistivity600A 10: Resistivity (11) ohm-cm (81)
pH600A 1: pH (17) pH (145)
pH_mV600A 2: pH mV (18) mV (162)
pH_ORP600A 3: ORP mV (19) mV (162)
RDO_concen600A 4: Dissolved Oxygen (20) mg/L (117)
RDO_perc_sat600A 5: Dissolved Oxygen (21) %sat (177)
RDO_part_Pressure600A 6: Partial Pressure Oxygen (30) Torr (26)
Pressure600A 17: Pressure (2) psi (17)
Depth600A 18: Depth (3) cm (34)
Voltage_Ext600A 14: External (32) V (163)
Battery_Int600A 15: Battery (33) % (241)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthardThis is non-trivial

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions