Link to NorESM-FATES historical run crash (NorESSM/CTSM issue 218)

[rosiealice]

Describe the issue

Hi all,

Just to let you know that @JessicaNeedham ran into a water balancce error in the soil column towards the end of an offline FATES-NOCOMP LUC historical simulation. Link to the issue is here:

NorESMhub/CTSM#218

I just wanted to reach out to see if anyone has seen anything similar. We think there might have been similar behavious in TRENDY as Kjetil (and maybe @ckoven?) changed the error tolerances to get that to finish.

@JessicaNeedham reports that turning off the endrun here creates a series of occasional where the tolerance is exceeded for this one gridcell, but not for any others.

It's somewhat hard to recreate this as it comes as the end of the whole AD/pAD/hist workflow...
we are running new spinups now with a clean version of. the recently created NorESM beta16 tag which may or may not encounter the same issues.

Relevant log output

FATES tag

see issue description for documentation.

Host land model tag

see issue description for documentation.

Machine

other HLM supported machine

Other supported machine name

Betzy

Additional context

No response

[JessicaNeedham]

We see a lot of warnings like this, always in the same location:

WARNING: column-level water balance error nstep= 1766844 local indexc= 878: 3 global indexc= 4579 errh2o= 4.687651641854806E-009

But the conditions for endrun were only hit one time in that single column (at least running for another 8 years).