Landuse + CBP flow widget

[rburghol]

Overall Workflow for Openmi-OM component

• Generate shell script to loop through all landsegment outflow WDMs for all land uses in a segment, see WDM Formatting below
• take input from CBP runoff/iflow/gwflow WDM exports (see below WDM Data Formatting)
• multiply by landuse grid (rows match csv columns)
  • Obtain from VAHydro (TBD - for now, play with
• sum all runoff
• produce text file with date and runoff total and gw iflow totals, and total Qout, and Runit
• make sure file format is compatible with OM runtime timeseries file used in cache mode
• save file name with OM runfile convention with elementid and runid

WDM Data Formatting
This will be a preliminary step in this evaluation, exporting the data to use in your widget. For the short term, we will just export the data for one land segment, for a river segment of interest

• Options:
  • Write in shell script?
  • Write in R, calling "system" command to execute quick_wdm - https://stat.ethz.ch/R-manual/R-devel/library/base/html/system.html
  • How do we know what land uses to parse?
    • Maybe obtain a directory listing? (ex: http://deq2.bse.vt.edu/p6/p6_gb604/tmp/wdm/land/ )
    • Parse a CBP config file?
    • Use a pre-configured list (my least favorite alternative)
• Locate CBP runoff/iflow/gwflow WDMs - these WDM files contain "unit area" runoff values
  • Example:
    • river segment: OR2_8130_7900
    • land segment: A51121
    • wdm: p532c-sova/wdm/land/for/p532cal_062211/forA51121.wdm
    • uci: p532c-sova/uci/land/for/p532cal_062211/forA51121.uci
    • DSN: 111 (SURO, ak surface runoff)
• Export Data from WDM (will produce 3 fils fro each land use IFWO, SURO, AGWO)
  Use quick_wdm_2_txt_hour_2_hour to convert wdms to csvs. Steps are as follows:
  • ssh into deq2.bse.vt.edu
  • cd /opt/model/p53/p532c-sova/tmp/wdm/land/(3 LETTER LAND USE)/p532cal_062211
  • /opt/model/p53/p532c-sova/code/bin/quick_wdm_2_txt_hour_2_hour
  • input wdm name, start year, end year, dsn
    • ex: afoA51121.wdm, 1984, 2005, 111
  • navigate to location on http://deq2.bse.vt.edu/p532c-sova/ in web browser, download -- EDIT: this step is obsolete -- the script can pull the wdms straight from the URL, no need to download files.
• Reformat WDM files into best format (we may need to experiment, the first option is easy and simple):
  • 1 File per Land Segment, 1 column for each land use with summed IFWO, SURO and AGWO
    • timestamp: the timestamp of the model output
    • afo
    • for
    • ...
  • Try 1 file per landuse and land segment, with 4 columns
    • timestamp: the timestamp of the model output
    • suro: suface runoff -- DSN 111
    • ifwo: interflow -- DSN 211
    • agwo: "active" groundwater -- DSN 411
• Model Output file: a csv output
  • Name: runlog[runid].[vahydro om elementid].log -- example: runlog2.256687.log
  • Columns (at minimum, you may include more than this, and we may later REQUIRE more than this)
    • timestamp
    • Qout
    • area_sqmi

Example Land Use Table
This is a Phase 5.3 table example, so some names may have changed in 5.3.2.

luname	1984	1987	1992	1997	2002	2005
afo	0.91	0.86	0.86	0.79	0.66	0.66
alf	9.83	10.27	11.29	9.13	11.91	7.83
bar	0.80	0.80	0.81	0.84	0.87	0.87
css	0.00	0.00	0.00	0.00	0.00	0.00
ext	25.40	25.40	25.40	25.40	25.40	25.40
for	1187.98	1190.02	1184.11	1197.78	1203.24	1216.05
hom	1.26	1.17	1.72	1.56	2.06	1.90
hvf	12.00	12.02	11.96	12.10	12.15	12.28
hwm	8.79	6.85	5.94	2.28	3.84	3.86
hyo	8.61	11.09	7.19	9.54	7.62	6.91
hyw	31.24	33.18	33.08	36.18	38.81	39.90
imh	12.30	12.30	12.30	12.80	13.20	13.30
iml	28.80	28.80	28.80	28.80	28.80	28.80
lwm	18.29	16.39	17.83	14.26	11.32	11.39
nal	0.00	0.00	0.00	0.00	0.00	0.00
nhi	0.00	0.00	0.00	0.00	0.00	0.00
nho	0.00	0.00	0.00	0.00	0.00	0.00
nhy	0.00	0.00	0.00	0.00	0.00	0.00
nlo	0.00	0.00	0.00	0.00	0.00	0.00
npa	0.00	0.00	0.00	0.00	0.00	0.00
pas	130.87	128.08	135.54	125.50	117.21	108.50
puh	11.40	11.40	11.40	11.80	12.10	12.00
pul	144.00	144.00	144.00	144.00	144.00	144.00
trp	6.89	6.74	7.13	6.61	6.17	5.71
urs	0.00	0.00	0.00	0.01	0.00	0.00
wat	4.00	4.00	4.00	4.00	4.00	4.00

Table 2: Sam[ple export from http://deq2.bse.vt.edu/p532c-sova/wdm/land/p532cal_062211_A51121_eos_all.csv

ix	Year	Month	Day	Hour	afo_0111	afo_0211	afo_0411	alf_0111	alf_0211	alf_0411	ccn_0111	ccn_0211	ccn_0411	cex_0111	cex_0211	cex_0411	cfo_0111	cfo_0211
1	1984	1	1	1	0.000171	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0.000171	0
2	1984	1	1	2	0.0001	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0.0001	0
3	1984	1	1	3	6.61E-05	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	6.61E-05	0
4	1984	1	1	4	4.67E-05	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	4.67E-05	0
5	1984	1	1	5	3.47E-05	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	3.47E-05	0
6	1984	1	1	6	2.68E-05	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	2.68E-05	0
7	1984	1	1	7	0.000179	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0.000179	0
8	1984	1	1	8	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0
9	1984	1	1	9	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0
10	1984	1	1	10	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0
11	1984	1	1	11	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0
12	1984	1	1	12	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0
13	1984	1	1	13	0	0	0	0	0	1.98E-05	0	0	1.98E-05	0	0	1.98E-05	0	0

[rburghol]

The code for this seems to work nicely @hdaniel7 . The basic script "CBP_flow_widget.R", does the following (from my testing it, this is what I think):

1. Downloads all the wdm files that have already been parsed as csv.
2. Merge these into a single large data stored somewhere in memory, under a variable name in the form '[landsegment][luname]_ALL'.
3. Multiply those by a landuse array that you imported from a file.
4. Writes a file with the resulting edge of stream input flows.

That seems to work well, and I'm impressed at how rapid it works given how damn much data is in memory! The next steps will be needed to make this more modular, and to save time (no multiple downloads).:

1. Create a single merged CSV, named "[cbp_scenario][landsegment]_eos_all" with [luname_111], [luname_211], [luname_411] for all (90 columns!). We need to test and see if this will be reasonable to work with, if so, it is the easiest for us to keep track of, i.e., 1 single file for each landuse and scenario, example would be p532cal_062211_A51121_eos_all.csv'
2. We will use the landuse * Runit code later in a separate script -- so don't toss it! But we will use it with the new eos_all file.
3. Put all future code in a directory called "R/utils" to keep things clean -- I created R/utils and moved CBP_flow_widget.R there already.
4. Add a script to export the single land use bash that you ran to the new directory "sh" (I just added this to the master and pushed it).

[hdaniel7]

Created and pushed land_use_eol_all.R -- this script generates a single .csv file named "[cbp_scenario][landsegment]_eos_all" with [luname_111], [luname_211], [luname_411] for all. The generated .csv seems to be reasonably easy to work with -- however, it is approximately 150mb and is too large to upload to github.

As follows is visual documentation of the script's functions:

It does not currently export this generated .csv to a new directory called "sh", it simply exports it to "utils" -- I did a fresh pull from the master branch but could not find the "sh" directory. @rburghol should this "sh" directory be located at "utils/sh"? And any ideas on next steps?

[rburghol]

Any ideas on next steps?
Per the meeting today - stash that sh code in the cbp6 project.

[hdaniel7]

On previous read-throughs I thought you wanted me to export the full land use .csv file to cbp6/utils/sh -- upon re-examination, I'm pretty confident that you're asking me to write a shell script to generate the individual land use .csvs on the deq2 machine, and to then stash that script in cbp6/utils/sh -- correct?
I can try to do this but I have never written a shell script before -- I can do some research into it and try to knock it out, if this script is what you want.

[rburghol]

Sorry - what I meant by:

Add a script to export the single land use bash that you ran to the new directory "sh" (I just added this to the master and pushed it).

Was to "export the runoff data for a single land use...", using quick_wdm_2_txt_hour_2_hour . I was assuming that you didn't manually export each Land Segment runoff WDM, rather, you wrote a shell script to do so (albeit a simplistic one). So, that makes me ask:

How did you generate the land segment runoff files that you parsed in "CBP_flow_widget.R"??

[hdaniel7]

I actually did just run quick_wdm_2_txt_hour_2_hour enough times to generate each land use .csv for one land segment, and have been using this land segment in all testing of the code so far -- none of the other land segments have any .csv files generated at all! I spent the last half hour or so looking into shell scripts a bit -- definitely would have been a more effective method, it seems.

[rburghol]

Just catching up on this -- yeah, shell would be great! I AM concerned about the file sizes that you indicate though... gonna do some thinking there. It might be prudent for us to add a zip and unzip to the steps. Can you give me the full file name and directory path the one CSV that you generated as a test?

[hdaniel7]

fn_land_use_eos_all.R has been updated in the openmi branch (not yet in cbp6, as it lacks roxygen headers and other aspects required for incorporation into the cbp6 package, at the moment). The unlabeled "observation number" row at the start of the output file has been removed, the year/month/day/hour columns have been combined to form a "timestamp" column, and the individual land use columns have been changed from the format "afo_0111" to "afo_suro". An example capture of the table is as follows:

timestamp	afo_suro	afo_ifwo	afo_agwo	alf_suro	alf_ifwo	alf_agwo	ccn_suro	ccn_ifwo	ccn_agwo
1/1/1984 1:00	0.000171	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 2:00	0.0001	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 3:00	6.61E-05	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 4:00	4.67E-05	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 5:00	3.47E-05	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 6:00	2.68E-05	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 7:00	0.000179	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 8:00	0	0	0	0	0	1.98E-05	0	0	1.98E-05
1/1/1984 9:00	0	0	0	0	0	1.98E-05	0	0	1.98E-05

The full table can be found at http://deq2.bse.vt.edu/p532c-sova/wdm/land/, until a better location to store it is determined.

That being said, the function still receives the list of land uses hardcoded through the line " land.use.list <- c('afo','alf','ccn','cex','cfo','cid','cpd','for','hom','hvf','hwm','hyo','hyw','lwm','nal','nex','nhi','nho','nhy','nid','nlo','npa','npd','pas','rcn','rex','rid','rpd','trp','urs')" -- these land uses are only accurate for phase 5 of the model. I will begin looking into dynamically reading in the land use input list by looking at which directories are within the "/wdm/land" directory of the inputted scenario.

[hdaniel7]

The land use list is now generated by reading the list of directories within the "/tmp/wdm/land" directory of the inputted model. The list.dirs() function I used only lists directories, and not files, so the output files from this script can still be stored in the "/tmp/wdm/land" directory, if desired.

[rburghol]

Good stuff!!!

[rburghol]

@hdaniel7 -- this file looks good. I had to change the header for the time column to "thisdate" instead of "timestamp". There was an undocumented "feature" that if you use "timestamp" it assumes it is already in proper unix integer format, but if you use "thisdate" it will translate it from a string. I think the string is just fine, and perhaps preferable for our purposes. Sorry I gave the wrong column name.

I still have to work some bugs out with using this in the model, but we are looking good!

[rburghol]

Model object loads the file, adds that file to a database table, and caches that table permanently (or until we update the file or ask it to be removed). The routine that finds the runoff values and multiplies it against the land use area is not yet functioning. Not sure if the cached table part is a problem or something else. Will sort it out tomorrow.