Failing VX tasks

[gspetro-NOAA]

Initial Question from @mkavulich
Can anyone point me to documentation on the diag_table variables for the Weather Model? I see that there's generic guidance on the format of the tables in the Weather Model User's Guide but no specifics on what each variable means/is. I'm specifically looking for information on the variable names and how they correspond to physical quantities coming out of the model. For example, what physically is cnvprcp_ave, or rainwat, etc.? To be even more specific, I'm trying to figure out where the 1-hour accumulated precipitation is in the model output. But there are a lot of variables in the diag_table that say "prcp", "rain", etc and I don't know what each one is specifically.

Problem Background:
When trying to make a new MET verification test case, I randomly chose FV3_GFS_v17_p8 as my physics suite. And when it got to the verification tasks, some of them started failing (specifically, the tasks that combine hourly precip into 3- and 6-hourly precip). I did a bunch of testing and confirmed that it is only the FV3_GFS_v17_p8 suite that is causing this; identical runs using other physics suites all run successfully

The point I'm at now is that the wgrib2 file output from UPP is the problem: instead of having hourly accumulated precipitation and total accumulated precipitation like it should (and like it does for all other physics suites), the grib2 output has two copies of total accumulated precipitation. The most obvious candidate for this problem is the diag_table since each suite has its own, but now I'm a bit stuck because there are tons of differences and not a lot of info. It could also be a UPP thing, but I hadn't started digging there.

More details:

Platform: Derecho
Version: HEAD of develop
Configuration: ATM (through the SRW)
Stage of the model when the issue appeared: The issue only appears in the verification step run_MET_PcpCombine_fcst, but the problem is obviously occurring before that, given the problem with two copies of the total accumulation variable
Log & Config Files: My log and config files are in /glade/derecho/scratch/kavulich/UFS/workdir/issue_966/expt_dirs/FV3_GFS_v17_p8/MET_verification/ on Derecho.
Environment: bash, default modules
Compiler + MPI: Derecho default (Intel)
Rundir: /glade/derecho/scratch/kavulich/UFS/workdir/issue_966/expt_dirs/FV3_GFS_v17_p8/MET_verification/
Codedir: /glade/derecho/scratch/kavulich/UFS/workdir/issue_966/ufs-srweather-app (but I have made debugging changes so the failure may not appear as advertised here)

[gspetro-NOAA]

Update from @mkavulich: I did discover what the issue is, though I still don't fully understand the scope of the problem. The diag_table for suite FV3_GFS_v17_p8 has settings so that history files are only supposed to be output every 3 hours.....even though the history files are still being written hourly, though some things are different, for some reason I don't understand.

[mkavulich]

Just updating with some more details: I thought I had solved the problem but it turns out that I was just running with the "wrong" suite to replicate the failure again. Switching back to suite FV3_GFS_v17_p8 even after a potential fix (see below) shows that potential fix didn't work.

The idea was that lines 7 and 8 of the diag_table currently read:

"fv3_history",             3,  "hours",    1, "hours", "time"
"fv3_history2d",           3,  "hours",    1, "hours", "time"

This is very confusing, as the in-line documentation for the diag table seems to imply that this should only be outputting dynf###.nc and phyf###.nc files every 3 hours, but that is not what is happening: we still get these every hour. Still, since the problem is with hourly precip accumulation, I still thought maybe updating lines 7 and 8 of the diag_table to read this (with "0" supposed to make output every hour) would help:

"fv3_history",             0,  "hours",    1, "hours", "time"
"fv3_history2d",           0,  "hours",    1, "hours", "time"

Sadly, this does not seem to fix the problem: verification tasks still fail, because there is still duplicate "accumulated precipitation" variables in the UPP output.

[mkavulich]

Okay a bit more progress now: I determined another thing that is different by looking at UPP log files: In all other suites (yes, I tried them all), the UPP log indicates that the variable fhzero= 1. But in the failing run, fhzero= 6. And while the documentation doesn't seem to have anything on this variable, a perusal of the source code gives a single, promising comment next to this variable declaration: !bucket

UPP is reading in this "bucket" value fhzero from the netCDF files. So my next step is to figure out why the forecast model is outputting fhzero=6 just for the FV3_GFS_v17_p8 suite

[junwang-noaa]

@mkavulich UPP has two precips, one for bucket (bucket is emptied every $FHZERO), the other is for continuous precip without bucket. GFS has been using 6 hour bucket for operational products for long time. The diag table field description can be found at:

https://github.com/NOAA-EMC/fv3atm/blob/develop/ccpp/driver/GFS_diagnostics.F90#L102

[mkavulich]

Thanks for the insight @junwang-noaa! So does this affect other variables aside from accumulated precipitation? I'm trying to think ahead to if there will be any negative effects if I set fhzero=1 for all suites in SRW.

[junwang-noaa]

Currently the single bucket is applied to the accumulated/averaged fields excepted those marked as continuous accumulation fields. If you change the bucket (FHZERO), the diagnostic fields (e.g. T2m max/min, radiation fluxes etc) will be impacted

[gspetro-NOAA]

@jkbk2004 Do you have some insight on this issue? It would be helpful to understand why the history files are still being written hourly even though they are only supposed to be output every 3 hours. Also, do you know if diag_table variables for the WM are documented somewhere? FMS just seems to have this documentation about the diag_table manager but nothing on the variables.

[gspetro-NOAA]

Update: The UPP provides this documentation related to model output. The “Model Variable NetCDF Name” column in this table corresponds to the third column in diag_table ("output name"), which, according to @mkavulich, is often the same as the second column, "field name". The UPP table can be used as a reference when field name and output name are the same in diag_table.