Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Group data by variable not month #272

Draft
wants to merge 31 commits into
base: dev-v2
Choose a base branch
from
Draft

Group data by variable not month #272

wants to merge 31 commits into from

Conversation

bpbond
Copy link
Member

@bpbond bpbond commented Mar 13, 2025 •
edited
Loading

This is an experimental PR -- DO NOT MERGE YET

This PR changes the sensor data system to produce L1 outputs organized by variable (research name) rather than month, as suggested in #171 . For example:

PTR_OW_20230101-20231231_battery-voltage_L1_v1-2.csv
PTR_OW_20230101-20231231_battery-voltage_L1_v1-2.pdf
PTR_OW_20230101-20231231_sonde-cable_L1_v1-2.csv
PTR_OW_20230101-20231231_sonde-cable_L1_v1-2.pdf
PTR_OW_20230101-20231231_sonde-depth_L1_v1-2.csv
PTR_OW_20230101-20231231_sonde-depth_L1_v1-2.pdf
PTR_OW_20230101-20231231_sonde-fdom_L1_v1-2.csv
PTR_OW_20230101-20231231_sonde-fdom_L1_v1-2.pdf

This has advantages and disadvantages -- many more output files, but it's much easier (and probably faster) to read in only the desired data.

This primarily involves changes to write_to_folders() and the design and variable metadata tables.

Closes #171

bpbond added 21 commits February 23, 2025 09:34
@bpbond
Add timing information to driver
a5f8102
@bpbond
Add timing data and change to using readr::write_csv
62a4bfc
@bpbond
Better handle release date and enforce clean repo (#254)
12a4f9e 
* Separate release date from date of data generation
@bpbond
Skip raw file if there's a corresponding L0 file (#256)
421b86d 
L0 file name now includes hash of raw file, so we can distinguish same-named but different-contents files
@bpbond
Don't check dir.create() return value (#255)
c8a909b
@bpbond
Fix reset() behavior so that files and folders are removed cleanly (#257
80bffe3 
)
@bpbond
Embed resources in qmd-generated html files (#258)
f91a6af
@bpbond
Sapflow: fix deep sensor labeling and shallow depth value (#259)
abc57cb 
* Change sapflow 2.5cm to 3.5cm
* Change "C12D" to "C12D", etc.
@bpbond
Check for error returned by parallel jobs (#260)
fc0bf32
@bpbond
Change L0 step to handle removed and edited raw files (#261)
e71b855
@bpbond
helpers.R cleanup (#266)
873c47e 
* write_to_folders cleanup
* Move list_directories from pipeline to compasstools
@bpbond
Move functions from compasstools to helper file (#267)
069ed70
@bpbond
Rework out-of-service to be more general and flexible (#269)
478c917
@bpbond
Update helpers.R
2c6b5f7
@bpbond
Change L1_normalize to write by research name
b23fbce
@bpbond
Change research names to use hyphens, not underlines
563d4a8
@bpbond
Update L1 outputs to be by research name
4cb51ae
@bpbond
Update helpers.R
f33d674
@bpbond
Update helpers.R
69a2818
@bpbond
Update design_table.csv
31e6057
@bpbond
Create README_v2-0.txt
0636cda
@bpbond bpbond requested a review from stephpenn1 March 13, 2025 22:51
@bpbond bpbond modified the milestones: L2, L1 v2-0 Mar 13, 2025
@bpbond
Copy link
Member Author

bpbond commented Mar 13, 2025

All checks have passed

🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stephpenn1 stephpenn1 Awaiting requested review from stephpenn1

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
L1 v2-0
Development

Successfully merging this pull request may close these issues.
1 participant
@bpbond