Skip to content

feat(retrievers): add Keenable web search backend - #862

Open
ilya-bogin-keenable wants to merge 1 commit into
Cinnamon:mainfrom
keenableai:feat/keenable-web-search
Open

feat(retrievers): add Keenable web search backend#862
ilya-bogin-keenable wants to merge 1 commit into
Cinnamon:mainfrom
keenableai:feat/keenable-web-search

Conversation

@ilya-bogin-keenable

Copy link
Copy Markdown

Description

Adds kotaemon.indices.retrievers.keenable_web_search.WebSearch as a third option for KH_WEB_SEARCH_BACKEND, next to the Tavily and Jina retrievers.

The difference from the two existing backends is that it works without an API key. With no KEENABLE_API_KEY set it calls the public endpoint (POST https://api.keenable.ai/v1/search/public); setting a key switches to the authenticated endpoint and only lifts the rate limits. So the @WebSearch chat command can work on a fresh install.

The module follows the shape of tavily_web_search.py:

  • same class name and module layout, so the dotted-string setting works unchanged;
  • one RetrievedDocument with the results concatenated as URL / title / page text, and the same file_name / type / llm_trulens_score metadata the reasoning pipelines expect;
  • max_results, snippet_max_length and timeout exposed as Params;
  • a 429 from the API raises a RuntimeError with the retry-after value.

It uses requests, which the package already imports elsewhere; no new dependencies.

Also documented: the new option is listed (commented) in flowsettings.py, .env.example gains a web-search block with the three key variables, and README.md mentions KH_WEB_SEARCH_BACKEND under notable settings. Tavily stays the default.

Tested with libs/kotaemon/tests/test_web_search.py (mocked requests.post: keyless endpoint and headers, keyed endpoint, 429 handling), pre-commit on the changed files, and a live keyless query through the retriever.

Disclosure: I work at Keenable.

Type of change

  • New features (non-breaking change).
  • Bug fix (non-breaking change).
  • Breaking change (fix or feature that would cause existing functionality not to work as expected).

Checklist

  • I have performed a self-review of my code.
  • I have added thorough tests if it is a core feature.
  • There is a reference to the original bug report and related work.
  • I have commented on my code, particularly in hard-to-understand areas.
  • The feature is well documented.

Add kotaemon.indices.retrievers.keenable_web_search.WebSearch as a third
option for KH_WEB_SEARCH_BACKEND, alongside Tavily and Jina.

Unlike the existing two backends it works without an API key: with no
KEENABLE_API_KEY set it calls the public endpoint, and setting a key
only lifts the rate limits. Results are returned in the same shape as
the Tavily retriever (one RetrievedDocument with URL, title and page
text per result). A 429 from the API raises a clear error carrying the
retry-after value. Uses requests, which the package already imports.

Documented in flowsettings.py, .env.example and README, with unit tests
against a mocked response.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant