Skip to content

feat(generator): clickable citations, parser and LLM arbitration - #120

Merged
kdroidFilter merged 3 commits into
masterfrom
feat/ref-links
Oct 11, 2026
Merged

kdroidFilter merged 3 commits into
masterfrom
feat/ref-links

Conversation

@kdroidFilter

@kdroidFilter kdroidFilter commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Groundwork for #111: make citations like (בראשית א, ד) clickable. This PR adds the :reflinks generator module that finds the citations, decides their targets, and writes them into the text as <a class="ref" href="seforim://line/<id>">(בראשית א, ד)</a> (data-llm="1" when the LLM decided). The app side (rendering and opening the link) comes in a follow-up.

How it works

The scanner reads every line, finds the ( … ) / [ … ] groups and resolves them through line.heRef.

Deterministic (≈ 90 %, no LLM):

  • explicit citations: Tanakh, Bavli (folio + side), Mishna, Yerushalmi, Tosefta, Rambam, Shulchan Arukh, Tur, Midrash Rabba, Rashi/Tosafot on Bavli, Zohar (volume + folio through the alt TOC)…
  • a book name written before the parentheses: בביצה (דף יז)
  • a Bavli chapter name: בפ' מרובה (דף עד) (chapter names from the Chapters alt TOC)
  • a folio alone inside a commentary on a tractate
  • (שם …) whose antecedent is in the same line
  • links already in the texts: the 19,296 otzaria://inline-link anchors (path + line index) and the 151 Sefaria refLinks (English data-ref translated through the export schemas), all resolved

On the current corpus: 1,588,668 deterministic links (explicit 1,145,154 · name before 315,228 · ibid in the line 56,346 · chapter name 33,090 · commentary's tractate 19,403 · Otzaria anchors 19,296 · Sefaria anchors 151).

Ambiguous (≈ 153k distinct cases), arbitrated by an LLM over a closed list of candidates:
שם across lines or bare, או"ח סי' without a book (SA / Tur / MB / the commentary named before), מ"א/מ"ב (Kings or MA/MB), פ"X after a tractate (chapter or folio 80+X), Tanakh name before the parentheses.

Measured on a held-out set of 420 annotated cases (Muse Spark 1.3, prompt v4, one prompt for ibid and one for references): 94.8 % accuracy, 98.5 % of the links it creates are right, 91 % recall. When unsure it answers "no link".

Decision cache

Decisions live in generator/data/ref-arbitration.jsonl, keyed by a SHA-256 of what the answer depends on (context, candidates as title + heRef + snippet hash, prompt version, model). A rebuild reuses them without any network call; only new or changed passages need a new run. Targets are stored as title|heRef, never as ids. ref-overrides.jsonl (manual corrections) always wins.

Token safety (:reflinks:arbitrateRefs)

  • estimate only unless -Pconfirm; the budget is min(-PmaxSpend, remaining credit − 0.25 USD)
  • every paid answer is fsynced as it arrives; a line cut by a crash is skipped on load
  • 402 / 401 stop every worker; 12 consecutive server failures open a circuit breaker; 429 pauses all workers without tripping it
  • a decided case is never sent again; an unparseable paid reply is stored as INVALID and only retried with -PretryInvalid
./gradlew :reflinks:arbitrateRefs -PllmKeyFile=~/key.txt -PmaxSpend=10 -Pconfirm

Writing the links

./gradlew :reflinks:generateRefLinks writes the deterministic links and the cached decisions into a copy of the database (build/reflinks/seforim.db, the source is never modified), without network access. Links live in the HTML itself, so the app needs no offset mapping: its HTML parser only has to read the href. On the current corpus: 1,596,063 links in 800,404 lines, visible text unchanged (checked on 3,000 random lines).

Not in this PR

  • the app side (HtmlParser keeping the href, click → open the target line)
  • the remaining LLM decisions (14,224 of 152,457 cases are decided and committed here)

Test plan

  • ./gradlew :reflinks:jvmTest (31 tests): parser, Sefaria refs (real citations and the pitfalls found on the corpus), commentator mapping, store durability, and the arbitrator under 402, down server, budget, 429, transient errors, rejected and invalid replies
  • dry run on the full corpus: 7.1M lines scanned in ~4 min, 152,713 distinct ambiguous cases
  • paid run in progress with the protections above

New :reflinks module that finds the citations written in the texts,
e.g. (בראשית א, ד), (ב"ק כב:), (רמב"ם פ"ב מהל' שבת ה"ג), and resolves
them to the cited line through line.heRef.

Deterministic links (about 90%): explicit citations, a book or Bavli
chapter name written before the parentheses, a folio inside a
commentary on its tractate, and ibid (שם) within the same line.

The ambiguous rest (שם across lines or bare, או"ח סי' without a book,
מ"א/מ"ב, פ"X chapter or folio, Tanakh name before the parentheses) is
arbitrated by an LLM over a closed list of candidates. Decisions are
cached in generator/data/ref-arbitration.jsonl, keyed by a content
hash of the case, so a rebuild never pays twice. The run fsyncs every
paid answer, stops on no credit or a down server, honours a budget and
the rate limits, and resumes where it stopped.

./gradlew :reflinks:arbitrateRefs only estimates the cost unless
-Pconfirm is given.
- otzaria://inline-link anchors (path + 1-based line index): 19,296
  resolved out of 19,296, all in שמירת שבת כהלכתה - מפתחות.
- Sefaria refLink anchors: the English data-ref is translated through
  the titles and node names of the Sefaria export schemas (151/151).
  An anchor wins over the citation written inside it.
- Bavli chapter names now come from the Bavli only: the Mishna and the
  Yerushalmi share them, which made every name ambiguous. A chapter
  name only resolves an explicit folio, so "(ב)" after it stays a note
  marker.
- arbitrateRefs logs the deterministic links by method.
generateRefLinks scans a seforim.db, takes the deterministic links and
the LLM decisions already cached, and writes them into a copy as
<a class="ref" href="seforim://line/<id>"> around the citation, the
Otzaria and Sefaria anchors being rewritten the same way. Links
decided by the LLM carry data-llm="1". No network access; the source
database is never modified.

On the current corpus: 1,596,063 links in 800,404 lines; the visible
text of the lines is unchanged.

Also commits the decision cache (14,224 paid decisions).
@kdroidFilter
kdroidFilter marked this pull request as ready for review October 11, 2026 11:28
@kdroidFilter
kdroidFilter merged commit 0073ded into master Oct 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant