Determine how and where to store outputs from GreenHEART in a straightforward way

[johnjasa]

How and where should we store outputs from a GreenHEART run?

WISDEM outputs a slew of files by default into an outputs folder at the run script level which includes a .csv, .pkl, .mat, and other default output files.

OpenMDAO creates a folder with the name of the run script at the level it's run at, e.g. a problem called 'wind_electrolyzer_plant_MN` would make a folder with the same name and store many reports there by default.

Separately but quite related, which outputs should be included by default from GreenHEART?

[jaredthomas68]

Where

we are currently doing the output/ directory approach, but I'm fine with whatever as long as there is a place so files are not all dumped in the directory where the code is run and the organization is intuitive.

What

Pretty much everything ideally. At a minimum we need system sizes, costs, financial metrics, locations (micro siting), etc. I don't think I could reasonably write a quick list of outputs we need to include.

How

I have been wrestling with this for a while. We have many different data types (scalars, lists (length of years of plant operation), lists (length of simulation, usually 8760, etc). The 8760s can get pretty cumbersome. I think perhaps we should have separate output files for each type. One file with all the information that corresponds to years of operation, one that corresponds to simulated time steps (8760 right now), and one that carries all the scalars/tuples/smallish types. The final file could then be organized as a yaml or sql or json in the same way the input files are organized and may carry the input values as well as outputs. The others files could also include inputs and outputs of the same length as well so there are fewer files to keep track of and it is very clear what inputs were used to get the results. The larger files could probably be saved as a csv or a more efficient file type like Parquet or HDF, though I've never used those.

[bayc]

I like the OpenMDAO format off the top of my head. It seems nice from an HPC perspective as well as separate output folders would be created for different script names.

[johnjasa]

Cool, we'll go forward with an output directory at the driver_config.yaml-defined name where everything will land.

I like your discussion on the how, Jared. Noted and I don't think we need to resolve that immediately, good goal to shoot for having different files based on output type and have them clearly organized and centrally accessible in some way.

[kbrunik]

I'd like to second Jared's point about type. Some of the outputs right now are mixed with 8760's and floats and at a minimum separating the 8760's into a separate output would be nice even if it's not consolidated.