Improve local storage to TSV

[patricktokeeffe]

Can we transition the TSV files to psuedo-TOA5? (relocated from comment on #9)

Campbell Scientific TOA5

• comma-separated values
• multi-line header

---

Excerpted from CR1000 User Manual, rev 2/18, section 7.6.3.11.1:

Line 1: Environment Line

Field	Description	Proposed value for Urbanova AQ node
1	TOA5	TOA5 (part of standard)
2	Station name	name/ID of assembled sensor package?
3	Datalogger model	lookup values corresponding to specific hardware configurations?
4	Datalogger serial number	serial number of just raspberry Pi? of entire assembled AQ node?
5	Datalogger OS version	Git tag/hash of installed software version
6	Datalogger program name	name of DAQ script?
7	Dataloger program signature	blank/not used? other sentinel to emphasize not a real TOA5 file? use for Git tag/hash instead?
8	Table name	?

Line 2: Header

This line consists of comma-delimited strings that identify the name of individual fields as given in the datalogger program.

Line 3: Units

This line identifies engineering units for that field of data, as declared at the beginning of the CRBasic program.

Line 4: Aggregation

The fourth line reports abbreviations of the data process used to produce the field of data. See table 20 within user manual section 7.6.3.17 (rev 2/18).

All additional lines

Subsequent lines are observed data and associated record keeping. The first field being a timestamp, and the second being the record (sequential) number.

Notes

• station name
  • sensor packages record lat/long... but that's not a name. Monitoring locations need names and multiple units can be installed at any one location so that name cannot be the same as the sensor unit.
  • if station represents assembled package, then logger serial can represent the Pi specifically (or whatever hardware is doing DAQ)

[vonw]

In response to the question marks...

Field 2: Station names will be assigned when units are deployed. Does it also make sense to include the coordinates (lat/lon), or are these part of the data stream? If lat/lon are not already being acquired and stored from the GPS breakout, they should be.

Field 3: I think it's OK to list the RPi as the datalogger at this level. We will be specifying the complete description of the units (including serial numbers of all devices) when we log the unit as an IoT device at AWS. So the list of IoT devices at AWS can serve as catalog of what unit contains which sensors. However, I think we should also keep track of this ourselves, perhaps just in an Excel file somewhere...

[vonw]

More...

Field 6: Sure. Name of DAQ script works for me.

Field 7: I don't understand what this is, but I'll take your recommendation for what it should be.

Field 8: I'm not sure what this is. I see the "table" as the "data", and this is adequately described by the header lines 2, 3, and 4. But again, I'll take your recommendation for what this should be, if anything.

[vonw]

Thanks again, Patrick, for defining this. Nice work!