Skip to content

feat(search): one highlighting logic, word-exact and trustworthy - #116

Merged
kdroidFilter merged 1 commit into
masterfrom
fix/search-highlighting
Oct 10, 2026
Merged

kdroidFilter merged 1 commit into
masterfrom
fix/search-highlighting

Conversation

@kdroidFilter

Copy link
Copy Markdown
Owner

One highlighting logic for the search, used by the results' snippets, their preview and the reader's smart find (Zayit PR to follow).

Rules

  • Whole words, diacritics included: never part of a word.
  • A query word counts as itself (behind Hebrew prefixes ו ה ב כ ל מ ש ד, or inside a longer word from 4 letters, as the n-gram matching allows) or as one of its inflections, from the dictionary entries the word truly is a form of.
  • Never a dictionary variant: variants hold loose associations (עם for כל, יום for הוא) that help find results but aren't the word.
  • Acronyms with any quote marks (רמב״ם, רמב"ם, הרמב״ם); a lone ה only with its geresh (ה׳); quoted phrases as their exact words in a row.
  • Consecutive matched words form one span (also across a maqaf).
  • Snippets lead with their matches: results show two lines, so the snippet starts just before the first matched word (or its sentence), in the hit line, cut between words.

API

  • SearchEngine.highlightRanges(text, query): the words.
  • SearchEngine.highlights(texts, query): the words, else (dense engines) the passage closest in meaning; batched.
  • SearchEngine.rangeSnippet(text, ranges). Replaces semanticSpan / passageSnippet.
  • SnippetProvider returns a SnippetSource with the hit line's position, built by the shared SnippetSources.build (the CLI provider no longer loads every line between two hits).

Dictionary entry choice

A token's entry is one it truly is a form of, before a larger one reached through a variant only: לעולם → עולם (was יום), להעמידה → עמד (was קום), ולדעת → דעת. ~5% of tokens change.

Validation

  • 113 search tests, 18 on the snippet builder (one per rule).
  • Real index, 31 queries × 25 results, lexical and hybrid: 0 result without bold, 0 cut word; first bold at the 13th visible character (median).
  • Results page latency unchanged (sum over the bench ~515–530 ms hybrid, was 594 before this work).

Pairs with seforim-semantic 0.4.1 (published).

- QueryHighlighter: whole words (diacritics included) standing for the query's
  words: the word itself (behind Hebrew prefixes, or inside a longer word from
  4 letters, as the n-gram matching allows) or its inflections, from the
  dictionary entries the word truly is a form of; never a dictionary variant
  (loose associations: עם for כל). Acronyms match with any quote marks; a lone
  ה needs its geresh; quoted phrases as their words in a row; consecutive
  matches as one span
- SearchEngine.highlightRanges / highlights / rangeSnippet: the one entry point
  results' snippets, their preview and the reader's smart find all use
  (highlights falls back to the passage closest in meaning on dense engines)
- SnippetBuilder: the snippet leads with its matches (results show two lines),
  starting in the hit line, cut between words
- SnippetSources: snippet sources built once for every store, with the hit
  line's position (the CLI provider also loaded every line between hits)
- MagicDictionaryIndex: a token's entry is one it is a form of before a larger
  one reached through a variant only (לעולם to עולם, not יום)
- seforim-semantic 0.4.1 (book-filtered dense search)

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant