Repository navigation
feat(search): one highlighting logic, word-exact and trustworthy - #116
Merged
Merged
Conversation
- QueryHighlighter: whole words (diacritics included) standing for the query's words: the word itself (behind Hebrew prefixes, or inside a longer word from 4 letters, as the n-gram matching allows) or its inflections, from the dictionary entries the word truly is a form of; never a dictionary variant (loose associations: עם for כל). Acronyms match with any quote marks; a lone ה needs its geresh; quoted phrases as their words in a row; consecutive matches as one span - SearchEngine.highlightRanges / highlights / rangeSnippet: the one entry point results' snippets, their preview and the reader's smart find all use (highlights falls back to the passage closest in meaning on dense engines) - SnippetBuilder: the snippet leads with its matches (results show two lines), starting in the hit line, cut between words - SnippetSources: snippet sources built once for every store, with the hit line's position (the CLI provider also loaded every line between hits) - MagicDictionaryIndex: a token's entry is one it is a form of before a larger one reached through a variant only (לעולם to עולם, not יום) - seforim-semantic 0.4.1 (book-filtered dense search) Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One highlighting logic for the search, used by the results' snippets, their preview and the reader's smart find (Zayit PR to follow).
Rules
API
SearchEngine.highlightRanges(text, query): the words.SearchEngine.highlights(texts, query): the words, else (dense engines) the passage closest in meaning; batched.SearchEngine.rangeSnippet(text, ranges). ReplacessemanticSpan/passageSnippet.SnippetProviderreturns aSnippetSourcewith the hit line's position, built by the sharedSnippetSources.build(the CLI provider no longer loads every line between two hits).Dictionary entry choice
A token's entry is one it truly is a form of, before a larger one reached through a variant only: לעולם → עולם (was יום), להעמידה → עמד (was קום), ולדעת → דעת. ~5% of tokens change.
Validation
Pairs with seforim-semantic 0.4.1 (published).