Update MGBF to use 32 filtering levels and fix memory leak #525

SamuelDegelia-NOAA · 2026-01-23T01:11:21Z

Description

This PR updates the MGBF namelists for the fv3-jedi ctests to use 32 filtering levels instead of the previously configured 65 levels. This improves the computational efficiency when using MGBF for ensemble localization. Additionally, the MGBF workaround for saber is updated to resolve a memory leak. These changes together reduce the runtime for the rrfs_fv3jedi_2024052700_3denvar_refl ctest by a factor of ~2 (1200 to 600 seconds).

The ctest references are correspondingly updated.

Huge thanks to @TingLei-NOAA for helping to improve the efficiency and test these changes.

Issue(s) addressed

None

Dependencies (if applicable)

None

Checklist

I have performed a self-review of my own code.
I have run rrfs tests before creating the PR (if applicable).
Unit tests added/updated (if applicable).

…f_memory

rrfsbot · 2026-01-23T01:46:16Z

FAILED on hera

started build_and_test on hera at UTC time: Fri Jan 23 01:16:17 UTC 2026
finished at UTC time: Fri Jan 23 01:46:13 UTC 2026

Test project /scratch3/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/525/build/rrfs-test
      Start  6: rrfs_fv3jedi_2024052700_getkf_observer
      Start 15: rrfs_mpasjedi_2024052700_getkf_observer
      Start  1: rrfs_fv3jedi_2024052700_3dvar
      Start  2: rrfs_fv3jedi_2024052700_3denvar
      Start  3: rrfs_fv3jedi_2024052700_3denvar_mgbf
      Start  4: rrfs_fv3jedi_2024052700_hybrid3denvar
      Start  5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf
      Start  8: rrfs_fv3jedi_2024052700_3dvar_conv_surface
 1/17 Test  #8: rrfs_fv3jedi_2024052700_3dvar_conv_surface ....   Passed   92.21 sec
      Start  9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair
 2/17 Test  #1: rrfs_fv3jedi_2024052700_3dvar .................   Passed  101.90 sec
      Start 10: rrfs_fv3jedi_2024052700_3dvar_remote
 3/17 Test  #6: rrfs_fv3jedi_2024052700_getkf_observer ........   Passed  109.88 sec
      Start  7: rrfs_fv3jedi_2024052700_getkf_solver
 4/17 Test #10: rrfs_fv3jedi_2024052700_3dvar_remote ..........   Passed   29.80 sec
      Start 11: rrfs_fv3jedi_2024052700_3dvar_satrad
 5/17 Test  #9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair ...   Passed   48.86 sec
      Start 12: rrfs_fv3jedi_2024052700_3denvar_refl
 6/17 Test #11: rrfs_fv3jedi_2024052700_3dvar_satrad ..........   Passed   79.11 sec
      Start 13: rrfs_mpasjedi_2024052700_bumploc
 7/17 Test  #7: rrfs_fv3jedi_2024052700_getkf_solver ..........   Passed  133.37 sec
      Start 14: rrfs_mpasjedi_2024052700_3denvar
 8/17 Test  #2: rrfs_fv3jedi_2024052700_3denvar ...............   Passed  274.30 sec
      Start 17: rrfs_bufr2ioda_msonet
 9/17 Test #15: rrfs_mpasjedi_2024052700_getkf_observer .......   Passed  277.11 sec
      Start 16: rrfs_mpasjedi_2024052700_getkf_solver
10/17 Test  #4: rrfs_fv3jedi_2024052700_hybrid3denvar .........   Passed  280.90 sec
11/17 Test #17: rrfs_bufr2ioda_msonet .........................   Passed   27.11 sec
12/17 Test  #5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf ....   Passed  340.95 sec
13/17 Test  #3: rrfs_fv3jedi_2024052700_3denvar_mgbf ..........   Passed  343.70 sec
14/17 Test #16: rrfs_mpasjedi_2024052700_getkf_solver .........   Passed  194.86 sec
15/17 Test #14: rrfs_mpasjedi_2024052700_3denvar ..............   Passed  275.44 sec
16/17 Test #13: rrfs_mpasjedi_2024052700_bumploc ..............   Passed  311.66 sec
17/17 Test #12: rrfs_fv3jedi_2024052700_3denvar_refl ..........***Failed  410.14 sec

94% tests passed, 1 tests failed out of 17

Label Time Summary:
mpi            = 3331.29 sec*proc (17 tests)
rdas-bundle    = 3331.29 sec*proc (17 tests)
script         = 3331.29 sec*proc (17 tests)

Total Test time (real) = 551.26 sec

The following tests FAILED:
	 12 - rrfs_fv3jedi_2024052700_3denvar_refl (Failed)
Errors while running CTest
Output from these tests are in: /scratch3/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/525/build/rrfs-test/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

workdir: /scratch3/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/525

SamuelDegelia-NOAA · 2026-01-23T01:59:55Z

Forgot to sync my branch with develop before running the ctests. So the test references here were not generated using the recent updates from #522. That is causing the CI failures for the dbz test. Will update the test references again soon.

SamuelDegelia-NOAA · 2026-01-23T02:09:32Z

FAILED on wcoss2

started build_and_test on wcoss2 at UTC time: Fri Jan 23 01:13:54 UTC 2026
finished at UTC time: Fri Jan 23 02:09:31 UTC 2026

Test project /lfs/h2/emc/da/noscrub/samuel.degelia/rrfsbot/PRs_RDASApp/525/build/rrfs-test
      Start  6: rrfs_fv3jedi_2024052700_getkf_observer
      Start 15: rrfs_mpasjedi_2024052700_getkf_observer
      Start  1: rrfs_fv3jedi_2024052700_3dvar
      Start  2: rrfs_fv3jedi_2024052700_3denvar
      Start  3: rrfs_fv3jedi_2024052700_3denvar_mgbf
      Start  4: rrfs_fv3jedi_2024052700_hybrid3denvar
      Start  5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf
      Start  8: rrfs_fv3jedi_2024052700_3dvar_conv_surface
      Start  9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair
      Start 10: rrfs_fv3jedi_2024052700_3dvar_remote
 1/17 Test #10: rrfs_fv3jedi_2024052700_3dvar_remote ..........   Passed   79.75 sec
      Start 11: rrfs_fv3jedi_2024052700_3dvar_satrad
 2/17 Test  #1: rrfs_fv3jedi_2024052700_3dvar .................   Passed   91.79 sec
      Start 12: rrfs_fv3jedi_2024052700_3denvar_refl
 3/17 Test  #9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair ...   Passed  103.82 sec
      Start 13: rrfs_mpasjedi_2024052700_bumploc
 4/17 Test  #8: rrfs_fv3jedi_2024052700_3dvar_conv_surface ....   Passed  109.85 sec
      Start 14: rrfs_mpasjedi_2024052700_3denvar
 5/17 Test  #6: rrfs_fv3jedi_2024052700_getkf_observer ........   Passed  175.86 sec
      Start  7: rrfs_fv3jedi_2024052700_getkf_solver
 6/17 Test #11: rrfs_fv3jedi_2024052700_3dvar_satrad ..........   Passed  138.13 sec
      Start 17: rrfs_bufr2ioda_msonet
 7/17 Test #17: rrfs_bufr2ioda_msonet .........................   Passed   36.57 sec
 8/17 Test  #2: rrfs_fv3jedi_2024052700_3denvar ...............   Passed  271.84 sec
 9/17 Test  #4: rrfs_fv3jedi_2024052700_hybrid3denvar .........   Passed  275.73 sec
10/17 Test  #3: rrfs_fv3jedi_2024052700_3denvar_mgbf ..........   Passed  323.79 sec
11/17 Test  #5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf ....   Passed  332.74 sec
12/17 Test  #7: rrfs_fv3jedi_2024052700_getkf_solver ..........   Passed  208.33 sec
13/17 Test #13: rrfs_mpasjedi_2024052700_bumploc ..............   Passed  346.02 sec
14/17 Test #15: rrfs_mpasjedi_2024052700_getkf_observer .......   Passed  536.73 sec
      Start 16: rrfs_mpasjedi_2024052700_getkf_solver
15/17 Test #14: rrfs_mpasjedi_2024052700_3denvar ..............   Passed  493.87 sec
16/17 Test #12: rrfs_fv3jedi_2024052700_3denvar_refl ..........***Failed  728.07 sec
17/17 Test #16: rrfs_mpasjedi_2024052700_getkf_solver .........   Passed  499.00 sec

94% tests passed, 1 tests failed out of 17

Label Time Summary:
rdas-bundle    = 4751.87 sec*proc (17 tests)
script         = 4751.87 sec*proc (17 tests)

Total Test time (real) = 1035.75 sec

The following tests FAILED:
	 12 - rrfs_fv3jedi_2024052700_3denvar_refl (Failed)
Errors while running CTest
Output from these tests are in: /lfs/h2/emc/da/noscrub/samuel.degelia/rrfsbot/PRs_RDASApp/525/build/rrfs-test/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

workdir: /lfs/h2/emc/da/noscrub/samuel.degelia/rrfsbot/PRs_RDASApp/525

rrfsbot · 2026-01-23T04:25:25Z

PASSED on hera

started build_and_test on hera at UTC time: Fri Jan 23 03:55:15 UTC 2026
finished at UTC time: Fri Jan 23 04:25:24 UTC 2026

Test project /scratch3/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/525/build/rrfs-test
      Start  6: rrfs_fv3jedi_2024052700_getkf_observer
      Start 15: rrfs_mpasjedi_2024052700_getkf_observer
      Start  1: rrfs_fv3jedi_2024052700_3dvar
      Start  2: rrfs_fv3jedi_2024052700_3denvar
      Start  3: rrfs_fv3jedi_2024052700_3denvar_mgbf
      Start  4: rrfs_fv3jedi_2024052700_hybrid3denvar
      Start  5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf
      Start  8: rrfs_fv3jedi_2024052700_3dvar_conv_surface
 1/17 Test  #1: rrfs_fv3jedi_2024052700_3dvar .................   Passed   36.13 sec
      Start  9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair
 2/17 Test  #8: rrfs_fv3jedi_2024052700_3dvar_conv_surface ....   Passed   75.53 sec
      Start 10: rrfs_fv3jedi_2024052700_3dvar_remote
 3/17 Test  #9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair ...   Passed   58.17 sec
      Start 11: rrfs_fv3jedi_2024052700_3dvar_satrad
 4/17 Test #10: rrfs_fv3jedi_2024052700_3dvar_remote ..........   Passed   27.27 sec
      Start 12: rrfs_fv3jedi_2024052700_3denvar_refl
 5/17 Test  #6: rrfs_fv3jedi_2024052700_getkf_observer ........   Passed  104.62 sec
      Start  7: rrfs_fv3jedi_2024052700_getkf_solver
 6/17 Test #11: rrfs_fv3jedi_2024052700_3dvar_satrad ..........   Passed   75.56 sec
      Start 13: rrfs_mpasjedi_2024052700_bumploc
 7/17 Test  #7: rrfs_fv3jedi_2024052700_getkf_solver ..........   Passed  110.98 sec
      Start 14: rrfs_mpasjedi_2024052700_3denvar
 8/17 Test #15: rrfs_mpasjedi_2024052700_getkf_observer .......   Passed  227.87 sec
      Start 16: rrfs_mpasjedi_2024052700_getkf_solver
 9/17 Test  #2: rrfs_fv3jedi_2024052700_3denvar ...............   Passed  230.80 sec
      Start 17: rrfs_bufr2ioda_msonet
10/17 Test  #4: rrfs_fv3jedi_2024052700_hybrid3denvar .........   Passed  252.19 sec
11/17 Test #17: rrfs_bufr2ioda_msonet .........................   Passed   26.38 sec
12/17 Test  #5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf ....   Passed  270.99 sec
13/17 Test  #3: rrfs_fv3jedi_2024052700_3denvar_mgbf ..........   Passed  277.09 sec
14/17 Test #16: rrfs_mpasjedi_2024052700_getkf_solver .........   Passed  195.34 sec
15/17 Test #14: rrfs_mpasjedi_2024052700_3denvar ..............   Passed  268.24 sec
16/17 Test #13: rrfs_mpasjedi_2024052700_bumploc ..............   Passed  321.81 sec
17/17 Test #12: rrfs_fv3jedi_2024052700_3denvar_refl ..........   Passed  459.92 sec

100% tests passed, 0 tests failed out of 17

Label Time Summary:
mpi            = 3018.90 sec*proc (17 tests)
rdas-bundle    = 3018.90 sec*proc (17 tests)
script         = 3018.90 sec*proc (17 tests)

Total Test time (real) = 562.79 sec

workdir: /scratch3/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/525

SamuelDegelia-NOAA · 2026-01-23T04:44:21Z

PASSED on wcoss2

started build_and_test on wcoss2 at UTC time: Fri Jan 23 03:50:25 UTC 2026
finished at UTC time: Fri Jan 23 04:44:20 UTC 2026

Test project /lfs/h2/emc/da/noscrub/samuel.degelia/rrfsbot/PRs_RDASApp/525/build/rrfs-test
      Start  6: rrfs_fv3jedi_2024052700_getkf_observer
      Start 15: rrfs_mpasjedi_2024052700_getkf_observer
      Start  1: rrfs_fv3jedi_2024052700_3dvar
      Start  2: rrfs_fv3jedi_2024052700_3denvar
      Start  3: rrfs_fv3jedi_2024052700_3denvar_mgbf
      Start  4: rrfs_fv3jedi_2024052700_hybrid3denvar
      Start  5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf
      Start  8: rrfs_fv3jedi_2024052700_3dvar_conv_surface
      Start  9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair
      Start 10: rrfs_fv3jedi_2024052700_3dvar_remote
 1/17 Test #10: rrfs_fv3jedi_2024052700_3dvar_remote ..........   Passed   80.38 sec
      Start 11: rrfs_fv3jedi_2024052700_3dvar_satrad
 2/17 Test  #1: rrfs_fv3jedi_2024052700_3dvar .................   Passed   90.34 sec
      Start 12: rrfs_fv3jedi_2024052700_3denvar_refl
 3/17 Test  #9: rrfs_fv3jedi_2024052700_3dvar_conv_upperair ...   Passed  105.31 sec
      Start 13: rrfs_mpasjedi_2024052700_bumploc
 4/17 Test  #8: rrfs_fv3jedi_2024052700_3dvar_conv_surface ....   Passed  106.31 sec
      Start 14: rrfs_mpasjedi_2024052700_3denvar
 5/17 Test  #6: rrfs_fv3jedi_2024052700_getkf_observer ........   Passed  167.34 sec
      Start  7: rrfs_fv3jedi_2024052700_getkf_solver
 6/17 Test #11: rrfs_fv3jedi_2024052700_3dvar_satrad ..........   Passed  136.92 sec
      Start 17: rrfs_bufr2ioda_msonet
 7/17 Test #17: rrfs_bufr2ioda_msonet .........................   Passed   35.01 sec
 8/17 Test  #2: rrfs_fv3jedi_2024052700_3denvar ...............   Passed  263.46 sec
 9/17 Test  #4: rrfs_fv3jedi_2024052700_hybrid3denvar .........   Passed  268.33 sec
10/17 Test  #3: rrfs_fv3jedi_2024052700_3denvar_mgbf ..........   Passed  314.37 sec
11/17 Test  #5: rrfs_fv3jedi_2024052700_hybrid3denvar_mgbf ....   Passed  329.47 sec
12/17 Test  #7: rrfs_fv3jedi_2024052700_getkf_solver ..........   Passed  201.29 sec
13/17 Test #13: rrfs_mpasjedi_2024052700_bumploc ..............   Passed  349.10 sec
14/17 Test #15: rrfs_mpasjedi_2024052700_getkf_observer .......   Passed  549.36 sec
      Start 16: rrfs_mpasjedi_2024052700_getkf_solver
15/17 Test #14: rrfs_mpasjedi_2024052700_3denvar ..............   Passed  492.99 sec
16/17 Test #12: rrfs_fv3jedi_2024052700_3denvar_refl ..........   Passed  718.01 sec
17/17 Test #16: rrfs_mpasjedi_2024052700_getkf_solver .........   Passed  356.06 sec

100% tests passed, 0 tests failed out of 17

Label Time Summary:
rdas-bundle    = 4564.04 sec*proc (17 tests)
script         = 4564.04 sec*proc (17 tests)

Total Test time (real) = 905.44 sec

workdir: /lfs/h2/emc/da/noscrub/samuel.degelia/rrfsbot/PRs_RDASApp/525

TingLei-NOAA

@SamuelDegelia-NOAA Thanks for incorporating those changes to those RRFS ctests.
All look good to me.

Samuel Degelia added 8 commits January 22, 2026 21:46

update mgbf namelists

8d9a879

add memory leak fix to workaround code

2593215

add some comments about workaround status to build script

616ff12

move ps to last analysis variable in yaml

db81b58

set ctest wall clock time to original setting

e942c94

update test references

f210fc4

Merge remote-tracking branch 'origin/develop' into feature/bugfix_mgb…

777d455

…f_memory

revert some accidental changes to jcb templates

df1081a

SamuelDegelia-NOAA requested review from ShunLiu-NOAA, TingLei-NOAA and delippi January 23, 2026 01:11

SamuelDegelia-NOAA added test_hera test_wcoss2 running_wcoss2 and removed test_wcoss2 labels Jan 23, 2026

rrfsbot added running_hera Failed_hera and removed test_hera running_hera labels Jan 23, 2026

SamuelDegelia-NOAA added Failed_wcoss2 and removed running_wcoss2 labels Jan 23, 2026

update refl test output for earlier PR

5836da2

ShunLiu-NOAA approved these changes Jan 23, 2026

View reviewed changes

SamuelDegelia-NOAA added test_hera test_wcoss2 and removed Failed_hera Failed_wcoss2 labels Jan 23, 2026

SamuelDegelia-NOAA added running_wcoss2 and removed test_wcoss2 labels Jan 23, 2026

rrfsbot added running_hera Passed_hera and removed test_hera running_hera labels Jan 23, 2026

SamuelDegelia-NOAA added Passed_wcoss2 and removed running_wcoss2 labels Jan 23, 2026

TingLei-NOAA approved these changes Jan 23, 2026

View reviewed changes

ShunLiu-NOAA merged commit beb30f2 into NOAA-EMC:develop Jan 23, 2026
1 check passed

SamuelDegelia-NOAA deleted the feature/bugfix_mgbf_memory branch January 23, 2026 20:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update MGBF to use 32 filtering levels and fix memory leak #525

Update MGBF to use 32 filtering levels and fix memory leak #525

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026 •

edited

Loading

Uh oh!

rrfsbot commented Jan 23, 2026

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026 •

edited

Loading

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026

Uh oh!

rrfsbot commented Jan 23, 2026

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026

Uh oh!

TingLei-NOAA left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Update MGBF to use 32 filtering levels and fix memory leak #525

Update MGBF to use 32 filtering levels and fix memory leak #525

Uh oh!

Conversation

SamuelDegelia-NOAA commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue(s) addressed

Dependencies (if applicable)

Checklist

Uh oh!

rrfsbot commented Jan 23, 2026

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026

Uh oh!

rrfsbot commented Jan 23, 2026

Uh oh!

SamuelDegelia-NOAA commented Jan 23, 2026

Uh oh!

TingLei-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SamuelDegelia-NOAA commented Jan 23, 2026 •

edited

Loading

SamuelDegelia-NOAA commented Jan 23, 2026 •

edited

Loading