Optimize parser performance for large library loads

[nlothian]

Summary

Even after caching redundant imports, the Earley parser remains the dominant cost when consulting large libraries. Profiling library/format.pl shows ~642 million function calls with 171 seconds of CPU time spent inside lark/parsers/earley.py:78(predict_and_complete) and friends (grammar.__eq__, earley_common.__eq__, hashing). A plain consult (no profiler) still takes ~48 seconds on a modern machine. We need to optimize or replace the parsing approach so that loading a file with a few use_module/1 directives doesn’t explode in Earley work.

Steps to Reproduce

1. From the repo root run:

   python - <<'PY'
   import cProfile, pstats
   from vibeprolog import PrologInterpreter
   prof = cProfile.Profile()
   prolog = PrologInterpreter(builtin_conflict="skip")
   prof.enable()
   prolog.consult("library/format.pl")
   prof.disable()
   prof.dump_stats("format_profile.pstats")
   pstats.Stats(prof).sort_stats("cumulative").print_stats(25)
   PY

2. Observe that nearly all time is consumed by the Earley parser regardless of repeated imports.

Expected Behavior

Consulting a few library files should complete quickly (on the order of seconds, not nearly a minute). The parser should avoid massive Earley state explosions, either by memoizing per-statement parses, reusing intermediate Earley states, switching to a more appropriate parser backend for deterministic input, or precompiling library ASTs.

Proposed Work

1. Investigate why Earley generates so many states for our grammar (642 million calls) when parsing deterministic Prolog code. Identify hotspots such as repeated equality checks (grammar.__eq__, earley_common.__eq__) and the state explosion around DCGs and operator-heavy clauses.
2. Explore optimizations such as caching parsed statements within a file, reusing Earley states between statements, or switching common paths to the LALR backend while keeping Earley only for constructs that require it.
3. Consider pre-parsing bundled libraries (format, pio, dcgs, etc.) into serialized ASTs that can be loaded quickly at runtime when the source hasn’t changed.
4. Implement the chosen optimization(s) and ensure they integrate cleanly with existing features (operator directives, DCGs, conditional compilation).
5. Document the new parser behavior in docs/FEATURES.md (performance section) and docs/SYNTAX_NOTES.md if there are any implications for syntax or library authors.

Testing

• Add regression tests that guard the new optimization (e.g., verifying that a representative library consult completes within a reasonable time budget, or that certain cached states are reused). Where measuring time directly is flaky, count parser invocations or hook into the optimizer’s instrumentation.
• Add tests ensuring the parser still handles edge cases covered by Earley today (operators, DCGs, conditional directives) after the optimization.
• Include tests proving that precompiled ASTs (if implemented) round-trip correctly when the source matches and that modifications trigger a re-parse.
• Ensure coverage includes docs/FEATURES.md/docs/SYNTAX_NOTES.md documentation updates.