Input file parsing

[wpbonelli]

We have discussed/prototyped using lark to decode MF6 input files into Python containers and primitives (before passing these to cattrs for structuring).

The prototype is currently (mostly) functional. But we still need to determine how performance compares to manual decoding. This will involve benchmarking/profiling the lark prototype against flopy3. If it becomes evident that flopy3 can be optimized in fairly straightforward ways, maybe it will be useful to add a minimal hand-written decoder prototype here for better comparison.

This is related to the performance testing mentioned in #49 (comment), but the focus areas are slightly different:

• here we are only concerned with load performance, and can ignore write performance
• in the other issue, we are most concerned with IO for data structures (arrays/tables/lists) specifically, where here we consider the full contents of input files, including internal and/or external data

[deltamarnix]

Sometimes I could prefer readability over performance.

• Are we concerned that load times will increase significantly?
• Are we only speaking of MF6 input files, or also MF6 output files?

[wpbonelli]

Are we concerned that load times will increase significantly?

This will need benchmarking to make sure they don't, and rather decrease.

Are we only speaking of MF6 input files, or also MF6 output files?

Lark is more suitable for input files than output files IMO. Input files share the same format, so we only have to write 1 grammar. Output files have a variety of formats. Maybe at some point it's worth considering for output files too, but maybe not yet.

[wpbonelli]

The main benefit provided by lark is maintainability. Instead of hand-writing a parser, a parser is generated from a grammar.

We can move forward with lark, and if performance turns out to be unacceptable, bite the bullet and hand-write something. To maximize performance the grammar should be kept as simple as possible (e.g., injecting component-specific keywords into the grammar rather than defining a grammar that is generic over all components).

Perhaps consider a prospective back-compat policy for flopy4 (e.g. dfns will be supported X years or Y releases). We can be more lenient with retrospective back-compat — upon competing an initial prototype, we can test against historic dfn versions and see how far back we can support.

In the long run, pursue standardizing input files with

• netcdf
• toml/etc

[wpbonelli]

The reason the proof of concept is so slow is the grammar is ambiguous. So lark has to use the Earley parser. For LALR the grammar can't have ambiguities. If we can get them out it should run in linear time.

[wpbonelli]

#74

[wpbonelli]

I think we can implement a custom xarray backend to read an input file into a dataset/tree. Internally open_dataset can invoke the lark reader.