Some implementation details for Castro retries

Author: maximumcats

retries/paper.tex

@@ -62,7 +62,7 @@ \section{Introduction}\label{Sec:Introduction}
   \item Zones whose local velocity exceeds the CFL hydrodynamic stability criterion given the selected timestep
   \item Negative density in a zone after a hydrodynamic update
 \end{itemize}
-All of these failure modes (and more) are represented in the simulation code Castro \citep{castro_joss}.
+All of these failure modes (and more) are represented in the simulation code \castro\ \citep{castro_joss}.
 In our experience, any useful science simulation will almost inevitably hit one of these failure modes.
 
 There are (at least) three possible responses to such failures. First, the application can abort and tell the user
@@ -78,7 +78,35 @@ \section{Introduction}\label{Sec:Introduction}
 that encounters one of these failure modes. Typically one will use a smaller $\Delta t$ on the retry, although
 other options exist, such as modifying the spatial resolution when using an adaptive mesh. One astrophysics
 application that successfully does this, which we drew inspiration from for this work, is the stellar evolution
-code MESA \citep{MESA}.
+code MESA \citep{MESA}. MESA also has the notion of a ``backup'': a retry can be defined as a second attempt at
+the current timestep, while a backup involves returning to an earlier timestep.
+
+\section{Implementation Details}\label{Sec:Implementation}
+
+A functional retry mechanism requires a decision scheme and associated code infrastructure for when to perform
+a second attempt at the timestep. A retry scheme effectively turns a timestep into a \texttt{while} loop: attempt
+an advance by $\Delta t$, and if it fails then attempt an advance by $\Delta t / 2$, and keep going until you have
+reached $t_{i} + \Delta t$ where $t_{i}$ is the starting simulation time. So in both principle and practice,
+implementing a retry mechanism is actually quite straightforward, with two complications as noted next.
+
+First, depending on context, this scheme may require saving application state and then restoring that application
+state at a later point. For example, \castro\ is a multi-level mesh refinement code where the fine levels are
+subcycled with respect to the coarse levels \citep{castro,berger_colella}, and so fine levels typically need to
+interpolate coarse data between time levels $t_{i}$ and $t_{i} + \Delta t$, where $\Delta t$ is the coarse grid
+timestep. \castro\ does this by maintaining both an ``old'' state and a ``new'' state corresponding to those two
+time levels. But subcycling within a given level due to retries will mean that at the end of the coarse step, the
+``old'' state data will be at some later time, such as $t_{i} + 3\Delta t / 4$ in the case of two failures. This
+needs to be reset back to the initial ``old'' state at $t_{i}$ for valid interpolation onto the fine grid. In \castro\
+we handle this by making a backup copy of the data\footnote{For the GPU build of \castro, we save this backup copy
+in host pinned memory since we are typically GPU memory constrained} at $t_{i}$ if (and only if) we encounter an
+advance failure on the first try.
+
+Second, this scheme may involve refactoring physics modules that perform irrevocable changes to application
+state. One way to guarantee that the scheme works is for the sequence of physics modules corresponding to an
+advance to be written as an idempotent operator: given an input ``old'' state at $t_{i}$, the advance updates
+only the ``new'' state at $t_{i} + \Delta t$, and only in a way that if you re-run the timestep you get the same
+result in the ``new'' state. This makes a retry scheme trivial: if you detect a failure, exit from the advance
+immediately, and then retry the advance with a smaller timestep.
 
 %======================================================================
 % References

retries/ws.bib

@@ -12,6 +12,27 @@ @article{castro_joss
   journal = {Journal of Open Source Software}
 }
 
+@article{castro,
+ author = {Almgren, A. S. and Beckner, V. E. and Bell, J. B. and Day, M. S. and Howell, L. H. and Joggerst, C. C. and Lijewski, M. J. and Nonaka, A. and Singer, M. and Zingale, M.},
+ title = {Castro: A New Compressible Astrophysical Solver. I. Hydrodynamics And Self-Gravity},
+ journal = {ApJ},
+ archiveprefix = {arXiv},
+ eprint = {1005.0114},
+ keywords = {equation of state, gravitation, hydrodynamics, methods: numerical, nuclear reactions, nucleosynthesis, abundances},
+ year = {2010},
+ month = may,
+ volume = {715},
+ pages = {1221-1238},
+ doi = {10.1088/0004-637x/715/2/1221},
+ adsurl = {http://adsabs.harvard.edu/abs/2010ApJ...715.1221A},
+ adsnote = {Provided by the SAO/NASA Astrophysics Data System},
+ number = {2},
+ source = {Crossref},
+ url = {https://doi.org/10.1088/0004-637x/715/2/1221},
+ publisher = {American Astronomical Society},
+ issn = {0004-637X, 1538-4357},
+}
+
 @ARTICLE{positivity_preserving,
 author = {{Hu}, Xiangyu Y. and {Adams}, Nikolaus A. and {Shu}, Chi-Wang},
 title = "{Positivity-preserving method for high-order conservative schemes solving compressible Euler equations}",
@@ -47,3 +68,18 @@ @ARTICLE{MESA
 adsurl = {https://ui.adsabs.harvard.edu/abs/2011ApJS..192....3P},
 adsnote = {Provided by the SAO/NASA Astrophysics Data System}
 }
+
+@ARTICLE{berger_colella,
+author = {{Berger}, M.~J. and {Colella}, P.},
+title = "{Local Adaptive Mesh Refinement for Shock Hydrodynamics}",
+journal = {Journal of Computational Physics},
+keywords = {Conservation Laws, Grid Generation (Mathematics), Hydrodynamics, Shock Waves, Algorithms, Boundary Integral Method, Discontinuity, Error Analysis, Euler Equations Of Motion, Numerical Analysis, Fluid Mechanics and Heat Transfer},
+year = 1989,
+month = may,
+volume = {82},
+number = {1},
+pages = {64-84},
+doi = {10.1016/0021-9991(89)90035-1},
+adsurl = {https://ui.adsabs.harvard.edu/abs/1989JCoPh..82...64B},
+adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}