feat(retrievers): add Keenable web search backend - #862
Open
ilya-bogin-keenable wants to merge 1 commit into
Open
feat(retrievers): add Keenable web search backend#862ilya-bogin-keenable wants to merge 1 commit into
ilya-bogin-keenable wants to merge 1 commit into
Conversation
Add kotaemon.indices.retrievers.keenable_web_search.WebSearch as a third option for KH_WEB_SEARCH_BACKEND, alongside Tavily and Jina. Unlike the existing two backends it works without an API key: with no KEENABLE_API_KEY set it calls the public endpoint, and setting a key only lifts the rate limits. Results are returned in the same shape as the Tavily retriever (one RetrievedDocument with URL, title and page text per result). A 429 from the API raises a clear error carrying the retry-after value. Uses requests, which the package already imports. Documented in flowsettings.py, .env.example and README, with unit tests against a mocked response.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
kotaemon.indices.retrievers.keenable_web_search.WebSearchas a third option forKH_WEB_SEARCH_BACKEND, next to the Tavily and Jina retrievers.The difference from the two existing backends is that it works without an API key. With no
KEENABLE_API_KEYset it calls the public endpoint (POST https://api.keenable.ai/v1/search/public); setting a key switches to the authenticated endpoint and only lifts the rate limits. So the@WebSearchchat command can work on a fresh install.The module follows the shape of
tavily_web_search.py:RetrievedDocumentwith the results concatenated as URL / title / page text, and the samefile_name/type/llm_trulens_scoremetadata the reasoning pipelines expect;max_results,snippet_max_lengthandtimeoutexposed asParams;RuntimeErrorwith the retry-after value.It uses
requests, which the package already imports elsewhere; no new dependencies.Also documented: the new option is listed (commented) in
flowsettings.py,.env.examplegains a web-search block with the three key variables, andREADME.mdmentionsKH_WEB_SEARCH_BACKENDunder notable settings. Tavily stays the default.Tested with
libs/kotaemon/tests/test_web_search.py(mockedrequests.post: keyless endpoint and headers, keyed endpoint, 429 handling),pre-commiton the changed files, and a live keyless query through the retriever.Disclosure: I work at Keenable.
Type of change
Checklist