Skip to content

feat(demos): add keyless Keenable MCP web search and screenshots - #191

Open
ilya-bogin-keenable wants to merge 1 commit into
StarTrail-org:mainfrom
keenableai:feat/keenable-search-demo
Open

ilya-bogin-keenable wants to merge 1 commit into
StarTrail-org:mainfrom
keenableai:feat/keenable-search-demo

Conversation

@ilya-bogin-keenable

Copy link
Copy Markdown

Adds an optional Keenable Search MCP demo next to demos/parallel_search/, with the same shape: find pages on the open web, save source URLs and snippets, add --fetch for page text, and opt into --render for PixelRAG screenshot tiles. Search and fetch work without an account or API key.

What changed

  • demos/keenable_search/: isolated uv project (run.py, test_run.py, README.md, pyproject.toml, uv.lock, .gitignore). It calls search_web_pages and fetch_page_content on https://api.keenable.ai/mcp over Streamable HTTP and hands selected URLs to the public render_urls API.
  • Root README.md: one line under "Render a page programmatically", after the Parallel demo link.
  • Nothing else is touched: no default, search path, renderer setting or agent config changes, and no index build or answer generation.

What is sent to Keenable: the query, the selected URLs when --fetch is used, and an X-Keenable-Title: PixelRAG header. An X-API-Key header is sent only if the user sets KEENABLE_API_KEY (for higher rate limits).

Following the review notes on #175, results on localhost or loopback, private and link-local IP hosts are skipped before fetching or rendering, and the tests run the MCP transport against both a structuredContent result and a JSON text result.

Tested

  • pytest demos/keenable_search/test_run.py: 6 passed (Python 3.12, mcp 1.30.0, httpx 0.28.1, macOS). The tests run the real MCP client over httpx.MockTransport, check the exact tool arguments, the title header, no key header by default, the opt-in key, a failed fetch, URL filtering and the renderer handoff. demos/parallel_search tests still pass (3).
  • ruff check and ruff format --check on demos/keenable_search pass (demos/ is excluded from the root lint and test jobs, so these were run by hand).
  • uv lock --check passes. The lock is the feat(demos): add free Parallel MCP web search and screenshots #175 lock with only the project name changed (same mcp, httpx and pixelrag 0.4.0 entries).
  • Live, keyless, through run.search_pages: 3 queries returned 10 results each; 0.8 to 1.2 s end to end per run including MCP initialize, and 1.7 to 1.9 s with --fetch of 3 pages.
  • Live run.py "Find explanations of retrieval augmented generation" --limit 2 --fetch --render on macOS with Google Chrome: 2 arXiv pages selected and fetched, 3 screenshot tiles written, manifest saved.

Disclosure: I work at Keenable, which runs this search API.

@vercel

vercel Bot commented Oct 26, 2026

Copy link
Copy Markdown

@ilya-bogin-keenable is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

@KGKallasmaa

Copy link
Copy Markdown

@ASuresh0524 and @andylizf, could one of you take a look at the code here when you have a moment? Thanks!

@ASuresh0524

Copy link
Copy Markdown
Collaborator

Yes @ilya-bogin-keenable. Will take a look at this by EOD today.

@ilya-bogin-keenable

Copy link
Copy Markdown
Author

@ASuresh0524 @andylizf gentle ping on this one. The branch is still MERGEABLE against main, and I'm happy to change anything in the demo if you'd like it shaped differently.

@ASuresh0524

Copy link
Copy Markdown
Collaborator

Status update — this is now green and mergeable, and the only thing left on it is a decision no amount of code review settles.

What I changed: merged current main in (CI had never run here; the original workflow run had aged out of retention) and fixed the lint errors that appeared when #209 removed demos/ from the ruff exclude list. That was the goalposts moving under you, not a defect in the PR. Against current main: ruff check and ruff format --check clean, and the change is still confined to its own directory.

What is left: the positioning question @andylizf raised — whether a managed/commercial backend demo belongs in this repo — has never been answered, and it is the only thing holding this. There is precedent in both directions: #175 merged a vendor demo into demos/, and #192 established a deliberately restrictive policy for hosted readers in eval/. demos/ has no written policy at all, which is the actual gap.

I have drafted one for @ASuresh0524 and @andylizf to settle, proposing the bar be about upkeep rather than commerce: the demo has to show something about PixelRAG rather than just the vendor, stay self-contained, pass lint, and carry a named owner and a Last verified date, with removal after 30 days of being broken not treated as a negotiation.

You will get a yes or a no against that, not more silence. Apologies that it has taken this long to get to a straight answer.

@ilya-bogin-keenable

Copy link
Copy Markdown
Author

Thanks for merging main and fixing the lint. That bar works for us: I'll be the named owner, add a Last verified date to the demo README, and I'm fine with removal if it stays broken for 30 days.

Sibling of demos/parallel_search: discover pages with Keenable's keyless MCP
(search_web_pages, optional fetch_page_content), save a results.json manifest,
and optionally render the selected pages with PixelRAG's render_urls.

Skips localhost and loopback, private and link-local IP hosts before fetching
or rendering, and tests both the structured-content and JSON-text tool result
paths.
@KGKallasmaa
KGKallasmaa force-pushed the feat/keenable-search-demo branch from 78ce1ad to d14cad9 Compare November 17, 2026 16:54
@KGKallasmaa

Copy link
Copy Markdown

I work with Ilya at Keenable. Rebased this onto current main, including #209 and #214. pytest demos/keenable_search/test_run.py still passes (6 tests), and ruff is clean on the demo files.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants