Add realistic global tests for Omega

[cbegeman]

Checklist

• User's Guide has been updated
• Developer's Guide has been updated
• API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
• Documentation has been built locally and changes look as expected
• Testing comment in the PR documents testing used to verify the changes
• New tests have been added to a test suite

[hyungyukang]

This is the deep-ocean noise I mentioned during today’s meeting:

I have already tried several things, but none of them resolved the issue. Using the linear Eos is one of the remaining options, so I will create the initial condition using the linear Eos by using #562 opened by @katsmith133.

[cbegeman]

@hyungyukang Are you using explicit bottom drag for both models?

[hyungyukang]

@hyungyukang Are you using explicit bottom drag for both models?

No, bottom drag is disabled in both models.
I am trying to simplify the case as much as possible, so I disabled several components - like vertical mixing.

[xylar]

I think there was some confusion on Slack where I wasn't clear on the goal. But I would not try to use these partition files. I would let Polaris generate them on the fly. You just need to supply the mpas-o.graph.info.200904 file as the graph file.

[cbegeman]

Thanks for pointing that out. I'm using the default filename so that these lines can be removed

[cbegeman]

@hyungyukang I'm having some trouble getting this test to work with MPAS-O. The error I get is 21: forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read. I think it would be helpful to have you compare what I have with your configuration. I did a diff on the namelist files and I didn't see the culprit:

diff /lcrc/group/
e3sm/ac.cbegeman/scratch/MPAS-Ocean-output/chrys/polaris-globalocean-mpaso-20260721/
ocean/spherical/realistic_global/EC30to60E2r2/analysis_members_test/forward/namelist
.ocean namelist.ocean.global 
9c9
<     config_run_duration = '0030_00-00-00'
---
>     config_run_duration = '0001_00:00:00'
14,16c14,16
<     config_write_output_on_startup = .true.
<     config_pio_num_iotasks = 6
<     config_pio_stride = 64
---
>     config_write_output_on_startup = .false.
>     config_pio_num_iotasks = 0
>     config_pio_stride = 1
20c20,21
<     config_block_decomp_file_prefix = 'graph.info.part.'
---
>     config_block_decomp_file_prefix = 'mpas-o.graph.info.230313.part.'
>     config_proc_decomp_file_prefix = 'mpas-o.graph.info.230313.part.'
23d23
<     config_proc_decomp_file_prefix = 'graph.info.part.'
26c26
<     config_dt = '00:01:30'
---
>     config_dt = '0000_00:01:30.000'
39c39
<     config_mom_del2 = 1.0e3
---
>     config_mom_del2 = 1000.0
80,81c80,81
<     config_GM_closure = 'EdenGreatbatch'
<     config_GM_constant_kappa = 600.0
---
>     config_GM_closure = 'constant'
>     config_GM_constant_kappa = 800.0
108c108
<     config_eddyMLD_use_old = .true.
---
>     config_eddyMLD_use_old = .false.
115,116d114
<     config_cvmix_background_diffusion_passive = 1.0e-5
<     config_cvmix_background_diffusion_passive_enable = .false.
117a116,117
>     config_cvmix_background_diffusion_passive = 1.0e-4
>     config_cvmix_background_diffusion_passive_enable = .false.
130,131c130,131
<     config_cvmix_shear_mixing_scheme = 'PP'
<     config_cvmix_shear_PP_nu_zero = 0.005
---
>     config_cvmix_shear_mixing_scheme = 'KPP'
>     config_cvmix_shear_PP_nu_zero = 1.0
171c171
<     config_use_bulk_wind_stress = .true.
---
>     config_use_bulk_wind_stress = .false.
179c179
<     config_use_sgr_opt_kpp = .true.
---
>     config_use_sgr_opt_kpp = .false.
281c281
<     config_land_ice_rho_ocean = 1028.0
---
>       config_land_ice_rho_ocean = 1028.0
317c317
<     config_implicit_constant_bottom_drag_coeff = 1.0e-3
---
>     config_implicit_constant_bottom_drag_coeff = 0.0
324c324
<     config_explicit_bottom_drag_coeff = 1.0e-3
---
>     config_explicit_bottom_drag_coeff = 0.0
373c373
<     config_eos_type = 'linear'
---
>     config_eos_type = 'jm'
387,389c387,389
<     config_eos_linear_Tref = 5.0
<     config_eos_linear_Sref = 35.0
<     config_eos_linear_densityref = 1000.0
---
>     config_eos_linear_Tref = 0
>     config_eos_linear_Sref = 0
>     config_eos_linear_densityref = 973
401,402c401,403
<     config_btr_dt = '0000_00:00:15'
<     config_n_btr_cor_iter = 2
---
>     config_btr_dt = '0000_00:01:00.000'
>     !config_n_btr_cor_iter = 2
>     config_n_btr_cor_iter = 1
404c405
<     config_btr_subcycle_loop_factor = 2
---
>     config_btr_subcycle_loop_factor = 1
408c409
<     config_btr_solve_SSH2 = .false.
---
>     config_btr_solve_SSH2 = .true.
446c447
<     config_disable_thick_sflux = .false.
---
>     config_disable_thick_sflux = .true.
454,455c455,456
<     config_disable_vel_explicit_bottom_drag = .false.
<     config_disable_vel_vmix = .false.
---
>     config_disable_vel_explicit_bottom_drag = .true.
>     config_disable_vel_vmix = .true.
460,461c461,462
<     config_disable_tr_vmix = .false.
<     config_disable_tr_sflux = .false.
---
>     config_disable_tr_vmix = .true.
>     config_disable_tr_sflux = .true.
559c560
<     config_use_DMSTracers_exponential_decay = .false.
---
>       config_use_DMSTracers_exponential_decay = .false.
838c839
<     config_AM_timeSeriesStatsMonthly_restart_stream = 'timeSeriesStatsMonthlyRestart'
---
>       config_AM_timeSeriesStatsMonthly_restart_stream = 'timeSeriesStatsMonthlyRestart'

[xylar]

The file path here is missing the component name before subdir and is thus ending up in a spherical directory outside of ocean.

[xylar]

@hyungyukang, in line with what @cbegeman requested above, can you provide the full config files for the runs you did in #593 (comment)?

I am concerned that MPAS-Ocean may have been run with the linear EOS, not jm. If that is the case, the differences between the models might indicate that MPAS-Ocean is too-good-to-be-true with a nonlinear EOS, rather than that there is anything wrong on the Omega side, given the order of the HPG tendency.

I'm not seeing any evidence of any bugs in the centered HPG tendency so far. I won't dispute that a higher order HPG might help but I also wonder if the lack of monotonic advection might be responsible, and the HPG errors are only the mechanism that triggers the problem.

[hyungyukang]

@hyungyukang I'm having some trouble getting this test to work with MPAS-O. The error I get is 21: forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read. I think it would be helpful to have you compare what I have with your configuration. I did a diff on the namelist files and I didn't see the culprit:

@cbegeman , I have been running this test with a separately compiled version of MPAS-O. Since I wanted MPAS-O to match Omega as closely as possible, I made a few small changes to the MPAS-O code, such as removing the horizontal FCT, and then compiled that modified version separately.

I also tried running MPAS-O for this case through Polaris, but I encountered different errors from the ones you saw. I think they are mostly related to the time strings in namelist.ocean and streams.ocean.

In analysis_members/__init__.py,

            'run_duration': '0030_00-00-00',
            'output_interval': '0001_00-00-00',

should be

            'run_duration': '0030-00-00_00',
            'output_interval': '0001-00-00_00',

[hyungyukang]

@hyungyukang, in line with what @cbegeman requested above, can you provide the full config files for the runs you did in #593 (comment)?

I am concerned that MPAS-Ocean may have been run with the linear EOS, not jm. If that is the case, the differences between the models might indicate that MPAS-Ocean is too-good-to-be-true with a nonlinear EOS, rather than that there is anything wrong on the Omega side, given the order of the HPG tendency.

I'm not seeing any evidence of any bugs in the centered HPG tendency so far. I won't dispute that a higher order HPG might help but I also wonder if the lack of monotonic advection might be responsible, and the HPG errors are only the mechanism that triggers the problem.

@xylar , I'm out of town today, but I will upload them after back.

I have also been running MPAS-O with the linear Eos and without bottom drag to keep it consistent with Omega. With the linear Eos, the two models show quite similar results (no noise in KE at deep ocean). I will summarize everything together, comparing Omega linear and TEOS-10 with MPAS-O linear and JM, and post the results here.

[xylar]

Even with bottom drag on for both models and JM on the MPAS-Ocean side, I'm still seeing 30x the KE in the bottom layer in Omega. The horizontal pressure gradient doesn't seem like it explains it -- both implementations are the same to a good approximation and the pressure gradient at time step 1 is quite similar for both.

The shear mixing was quite different for the 2 but fixing that had little effect.

I'm still investigating.

[cbegeman]

I'm posting @hyungyukang 's results with the MontgomeryPotential_and_density because my understanding is that this is the pressure gradient implementation that's most consistent with Omega and it does show KE enhancement along boundaries. Caveat: MontgomeryPotential_and_density is recorded as not fully tested.

[xylar]

@hyungyukang and @cbegeman, how are you setting up your linear EOS runs? I'm seeing almost immediate crashes (NaNs) when I try to use the TEOS-10 initial condition with a linear EOS.

[hyungyukang]

@hyungyukang and @cbegeman, how are you setting up your linear EOS runs? I'm seeing almost immediate crashes (NaNs) when I try to use the TEOS-10 initial condition with a linear EOS.

@xylar , I created the Omega initial condition with the linear Eos using PR #562 , and then ran Omega with the linear Eos.

Here's the command I used to create the initial condition with linear EOS.

cd polaris
python utils/omega/convert_mpaso_ic_to_omega.py --input-file ocean.EC30to60E2r2.200908.nc --output-file ocean.EC30to60E2r2.200908.nc --include-wind-stress --eos-type linear

I downloaded ocean.EC30to60E2r2.200908.nc from https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/EC30to60E2r2/ocean.EC30to60E2r2.200908.nc

For quick access, I copied the linear-Eos initial condition to Frontier:

/lustre/orion/cli115/world-shared/hgkang/data4others/Omega/realistic_global/ocean.EC30to60E2r2.200908.lineareos.nc

[xylar]

Thanks @hyungyukang, that's very helpful!

[xylar]

Having worked on this in various ways for 2 days, I'm quite convinced that the pressure gradient for the sloped bathymetry is, indeed, responsible.

The medium-term solution is fairly clearly a different approach -- with quadrature -- for computing the HPG tendency.

The shorter-term fix isn't clear to me. Higher bottom drag seems like it's worth a try. I'm going to see how far I get with a substantially higher value.

[xylar]

With monotonic advection, explicit bottom drag of 1e-2, and del4 of 1.2e12, I've run for 15 days with no sign of instability. I'm going to run another 15 days and also try to lower the drag to 3e-3 and del4 to 4e11 and see how things go.

More Cd and more del4 are definitely reducing the KE at the bottom:

But it's also dissipating a bit with time:

[cbegeman]

@xylar Great! I'm glad del4 worked out. I agree with your strategy of trying to rely on this more than bottom drag. Can you output VelDel2DivCell and VelDel2RelVortVertex to help inform our choice of DivFactor? Those variables are both in VelocityDel2AuxVars.

[hyungyukang]

@xylar , thanks for testing those.

FYI, I have run Omega for more than a year using RK4 with noisy KE in the deep ocean.

I am testing the split-explicit time stepper. In Omega, the SE time stepper with the nonlinear Eos and centered PGrad runs much longer than MPAS-O with the same configuration: about 951 days in Omega VS 238 days in MPAS-O - while the noisy KE in the deep ocean still persists. Actually, this result does not really make sense to me, but Omega runs long enough to measure performance or to use it in coupling tests.

[xylar]

FYI, I have run Omega for more than a year using RK4 with noisy KE in the deep ocean.

Oh, that would be good to know more about. I assumed we were stuck. What features are you using for that run? What coefficients?

It sounds like I don't have to continue my debugging. It would be good if we can coordinate on this better in the future so we aren't doing redundant work.