Add swappable websearch backend, make Tako search default

docs/pages/reference/tool-directory.mdx

@@ -33,7 +33,7 @@ These are broadly useful across most deployments and are good candidates to conf
 | `notion` | Search and update Notion pages, databases, blocks, and comments | `NOTION_API_KEY` |
 | `slack` | Search Slack, read threads, inspect channels/users, and send or upload messages | `SLACK_BOT_TOKEN`; optional: `SLACK_SEARCH_TOKEN`, `SLACK_ETL_TOKEN` |
 | `gsuite` | Use Gmail, Calendar, Drive, Docs, Sheets, Slides, and Google Analytics | `GOOGLE_TOKEN_JSON` |
-| `websearch` | Free web search via Parallel and deep research | None; `PARALLEL_API_KEY` for `deep_research`; `ANTHROPIC_API_KEY` for search synthesis |
+| `websearch` | Web search over Tako's data graph and the web (default) or Parallel, plus deep research | None for `search`; `TAKO_API_KEY` for Tako REST and `deep-research`; `PARALLEL_API_KEY` when `WEBSEARCH_BACKEND=parallel`; `ANTHROPIC_API_KEY` for search synthesis |
 | `company_context` | Search indexed company history, private Slack conversations, and Google Docs | None |
 | `grafana` | Query dashboards, alerts, VictoriaMetrics, VictoriaLogs, and annotations | `GRAFANA_URL`, `GRAFANA_API_KEY` |
 | `posthog` | Query product analytics, events, pageviews, breakdowns, and user agents | `POSTHOG_API_KEY`, `POSTHOG_PROJECT_ID` |
@@ -104,7 +104,7 @@ These are broadly useful across most deployments and are good candidates to conf
 | `plural` | State legislation, legislators, committees, events, and jurisdictions | `PLURAL_API_KEY` |
 | `sensortower` | Mobile app analytics, publisher data, charts, and sales estimates | `SENSOR_TOWER_AUTH_TOKEN` |
 | `similarweb` | Web traffic, rankings, referrals, keywords, geography, and app data | `SIMILARWEB_API_KEY` |
-| `websearch` | Free web search via Parallel and deep research | None; `PARALLEL_API_KEY` for `deep_research`; `ANTHROPIC_API_KEY` for search synthesis |
+| `websearch` | Web search over Tako's data graph and the web (default) or Parallel, plus deep research | None for `search`; `TAKO_API_KEY` for Tako REST and `deep-research`; `PARALLEL_API_KEY` when `WEBSEARCH_BACKEND=parallel`; `ANTHROPIC_API_KEY` for search synthesis |
 | `youtube` | YouTube video, channel, transcript, and search data | `YOUTUBE_API_KEY`, `GOOGLE_API_KEY` |
 
 ## Media

tools/research/websearch/.env.example

@@ -1,10 +1,17 @@
-# `search` works anonymously through the hosted Parallel Search MCP without
-# any credentials. Each capability layers on:
-#   PARALLEL_API_KEY     — unlocks paid Search REST path and `deep_research`
-#   ANTHROPIC_API_KEY    — enables Claude synthesis on top of search results
+# `search` works with no credentials: anonymous Tako search by default, or
+# Parallel's free Search MCP when WEBSEARCH_BACKEND=parallel. Each key layers on:
+#   TAKO_API_KEY         — Tako REST search with filters, and `deep-research`
+#                          on the Tako Answer Agent (WEBSEARCH_BACKEND=tako)
+#   PARALLEL_API_KEY     — Parallel Search REST, and `deep-research` on the
+#                          Parallel Task API (WEBSEARCH_BACKEND=parallel)
+#   ANTHROPIC_API_KEY    — Claude synthesis on top of `search` results
 #
-# These are the only env-driven knobs. Non-secret config (synthesis model,
-# default Task processor, REST/MCP base URLs) is set at WebSearchClient
-# construction time via kwargs.
-PARALLEL_API_KEY=
-ANTHROPIC_API_KEY=
+# WEBSEARCH_BACKEND is deployment config (set it in sandbox.extraEnv), not a
+# secret. Other non-secret config (synthesis model, base URLs) is set on
+# WebSearchClient(...) at construction time.
+# Leave a key commented out to run without it. An exported-but-empty value and
+# an absent one both fall back to anonymous search.
+WEBSEARCH_BACKEND=tako
+# TAKO_API_KEY=
+# PARALLEL_API_KEY=
+# ANTHROPIC_API_KEY=

tools/research/websearch/README.md

@@ -1,81 +1,96 @@
 # Websearch Plugin
 
-Web search and deep research via [Parallel Web Systems](https://docs.parallel.ai),
-with optional Claude synthesis on top of search results.
+Web search and deep research over one of two backends, with optional Claude
+synthesis on top of `search` results.
 
-| Capability                       | No credentials                                  | + `PARALLEL_API_KEY`                                  | + `ANTHROPIC_API_KEY`                                 |
-| -------------------------------- | ----------------------------------------------- | ----------------------------------------------------- | ----------------------------------------------------- |
-| `search` (sources + excerpts)    | Free hosted MCP, lower rate limits              | Parallel Search REST, higher limits + filters         | (same; synthesis layered on top — see below)          |
-| `search(synthesize=True)`        | Raw excerpts only, skipped synthesis flagged    | Raw excerpts only, skipped synthesis flagged          | Claude reviewer → writer → citation-repair pipeline   |
-| `deep_research`                  | not available (key required)                    | Parallel Task API (default processor `ultra-fast`)    | (same)                                                |
+| Backend (`WEBSEARCH_BACKEND`) | `search`, no key | `search`, keyed | `deep-research` (keyed only) |
+| --- | --- | --- | --- |
+| `tako` (default) | Anonymous `tako_search` on `mcp.tako.com`: Tako data cards plus web results | `POST https://tako.com/api/v3/search` with filters and `--effort` (`TAKO_API_KEY`) | Tako Answer Agent, `POST /api/v1/agent/answer/runs` (`TAKO_API_KEY`) |
+| `parallel` | Parallel's free Search MCP | Parallel Search REST (`PARALLEL_API_KEY`) | Parallel Task API (`PARALLEL_API_KEY`) |
 
-Each credential is additive: drop in only what you need.
+`ANTHROPIC_API_KEY` adds the Claude reviewer → writer → citation-repair
+pipeline over whichever backend retrieved (`search` only). Without it, `search`
+returns raw results and records the skipped synthesis in `meta.partial_failures`.
 
-The free MCP path is provided by Parallel; on that path the response includes a `meta.attribution` string ("Search powered by the free Parallel Web Search MCP …"). The CLI `--pretty` mode surfaces it. Please retain or display this attribution when redistributing free-tier results. See <https://parallel.ai/customer-terms>.
+## How the backend is chosen
+
+Two facts decide a call:
+
+- **Vendor** comes from the `WEBSEARCH_BACKEND` environment variable (set it
+  in the chart's `sandbox.extraEnv`). Unknown values fail at startup with the
+  accepted values in the message.
+- **Keyed or anonymous** comes from the principal's grant. The tool sends the
+  injected placeholder to the vendor's REST endpoint; a 401 means the key isn't
+  granted, and the tool falls back to that vendor's anonymous path for the rest
+  of the process. Nothing probes the other vendor.
+
+`meta.backend` reports which path served the call: `tako:api`,
+`tako:anonymous`, `tako:agent`, `parallel:api`, `parallel:mcp`, or
+`parallel:task:<processor>`.
+
+Anonymous Tako search is rate limited per client IP (10 calls per minute) and
+platform-wide. A limited call fails with the server's message. A real
+deployment brings a `TAKO_API_KEY`.
+
+On the Parallel free-MCP path the response carries a `meta.attribution` string
+("Search powered by the free Parallel Web Search MCP …") that `--pretty`
+surfaces. Retain or display it when you redistribute free-tier results. See
+<https://parallel.ai/customer-terms>.
 
 ## Quickstart
 
 ```python
 from websearch.client import WebSearchClient
 
 client = WebSearchClient()
-result = await client.search("Parallel Web Systems funding")
-# meta.backend will be 'parallel:mcp' (no key) or 'parallel:api' (with key)
+result = await client.search("US GDP growth since 2020")
+# meta.backend is 'tako:anonymous' with no key, 'tako:api' with a granted TAKO_API_KEY
 ```
 
-That's the minimum — works with zero credentials via the free MCP. Add a `PARALLEL_API_KEY` to use the paid REST and unlock `deep_research`; add `ANTHROPIC_API_KEY` to get cited markdown synthesis on top of the results.
-
 ## Secrets
 
 Set in root `.env` (preferred) or `tools/research/websearch/.env`.
 
-- `PARALLEL_API_KEY` — optional. Get one at <https://platform.parallel.ai>. Unlocks the paid Search REST path (with source filters) and enables `deep_research` via the Task API.
-- `ANTHROPIC_API_KEY` — optional. Enables the Claude-backed synthesis pipeline on `search(synthesize=True)`.
+- `TAKO_API_KEY` — optional. Unlocks Tako REST search (filters, `--effort deep`) and `deep-research` on the Answer Agent. Injected as `X-API-Key` on `tako.com` only.
+- `PARALLEL_API_KEY` — optional, used when `WEBSEARCH_BACKEND=parallel`. Get one at <https://platform.parallel.ai>.
+- `ANTHROPIC_API_KEY` — optional. Enables the Claude synthesis pipeline on `search`.
 
-Non-secret config (synthesis model, default Task processor, REST/MCP base URLs) is configured via `WebSearchClient(...)` constructor kwargs at instantiation time. Defaults: `synthesis_model="claude-opus-4-6"`, `parallel_deep_research_processor="ultra-fast"`, `parallel_api_base_url="https://api.parallel.ai"`, `parallel_mcp_url="https://search.parallel.ai/mcp"`.
+Non-secret config (synthesis model, base URLs, the default Parallel processor) is set with `WebSearchClient(...)` kwargs. Defaults: `synthesis_model="claude-opus-4-6"`, `tako_api_base_url="https://tako.com"`, `tako_mcp_url="https://mcp.tako.com/mcp"`, `parallel_api_base_url="https://api.parallel.ai"`, `parallel_mcp_url="https://search.parallel.ai/mcp"`, `parallel_deep_research_processor="ultra-fast"`.
 
 ## Tools
 
 ### `search`
 
 ```python
-await client.search(
-    "How should a fintech startup evaluate MPC vs HSM in 2026?",
-    num_results=10,
-)
+await client.search("How should a fintech startup evaluate MPC vs HSM in 2026?", num_results=10)
 ```
 
-Arguments:
+- `query: str` — required.
+- `effort: "instant" | "fast" | "deep"` — default `fast`. On Tako, `deep` widens retrieval and adds a rerank at a higher price. On Parallel, `instant` maps to `basic` and the others to `advanced`. Anonymous paths record it in `meta.partial_failures`.
+- `include_domains`, `exclude_domains: list[str]`, `max_age_hours: int` — keyed paths only. `max_age_hours` rounds down to a UTC calendar date on both vendors.
+- `client_model`, `max_chars_total`, `session_id` — Parallel REST knobs. Tako ignores `client_model` and `session_id` and notes `max_chars_total` in `meta.partial_failures`.
+- `synthesize: bool` — default `True`. Needs `ANTHROPIC_API_KEY`.
 
-- `query: str` — **required** positional. The user's question or topic. Forwarded to Parallel as both the `objective` and the sole `search_queries` entry.
-- `client_model: str` — identifier of the LLM consuming the excerpts (e.g. `claude-opus-4-7`). Enables per-model retrieval/excerpt tuning.
-- `session_id: str` — stable UUID reused across related Search/Extract calls; used for free-tier rate limiting on the MCP.
+Results are `SourceDocument`s (`source_id`, `title`, `url`, `snippet`, `published_date`, `domain`). From Tako, data cards come first and web results follow. A card's `url` is its Tako page (the chart), its `snippet` is the data-bound description plus metric definitions and methodology, and its `domain` is the publisher (`International Monetary Fund`), not `tako.com`.
 
-REST-only (silently warned via `meta.partial_failures` when used over the free MCP):
-
-- `mode: "basic" | "advanced"` — default `advanced` on REST. MCP forces `basic`.
-- `max_chars_total: int` — hard cap on total excerpt characters.
-- `include_domains`, `exclude_domains: list[str]` — domain filters.
-- `max_age_hours: int` — recency filter, rounded **down to a UTC calendar-date** cutoff (Parallel's `source_policy.after_date` is date-granular, not hour-precise — `max_age_hours=6` becomes "published on or after today's date").
-
-Set `synthesize=True` (default) to also run the Claude reviewer → writer → citation-repair pipeline against the retrieved sources. Without `ANTHROPIC_API_KEY` the call still returns raw Parallel sources and records the skipped synthesis in `meta.partial_failures`.
-
-### `deep_research`
+### `deep-research`
 
 ```python
 await client.deep_research(
-    "How should a fintech startup evaluate MPC vs HSM in 2026?",
-    processor="pro-fast",  # optional
+    "How should a fintech startup evaluate MPC vs HSM in 2026?", effort="high"
 )
 ```
 
-Creates a Parallel Task API run with auto schema, polls to completion, and renders the structured JSON output as cited markdown.
+- `effort: "medium" | "high"` — default `medium`. Tako passes it to the Answer Agent. Parallel maps `medium` to `ultra-fast` and `high` to `ultra`.
+- `timeout_seconds` — default 600 s on Tako, per-processor on Parallel. Neither vendor has a cancel endpoint: a run that outlives the budget keeps running and keeps costing.
 
-**Requires `PARALLEL_API_KEY`** — the free MCP does not include deep-research. Restricted to the `pro/ultra` processor family (`lite`/`base`/`core` raise a clear error pointing at the docs).
+On Tako, the report is the agent's answer with `[n]` citations, then `## Charts` (one link per card), `## Definitions`, `## Assumptions`, `## Methodology` (each only when present), and `## Sources`. `meta.estimated_cost_usd` is the run's actual billed cost from `usage.total_cost_usd`. A query Tako declines (`refusal_code`) fails with the code instead of returning an empty report.
 
-### Processor cheatsheet
+On Parallel, the run goes through the Task API with auto schema and is restricted to the `pro`/`ultra` processor family (`lite`/`base`/`core` raise a clear error pointing at the docs).
 
-Pick a `processor` based on the depth and latency you need (cost is per 1 000 runs):
+#### Parallel processor cheatsheet
+
+The hidden `--processor` flag overrides the `effort` mapping on Parallel only. Cost is per 1,000 runs.
 
 | Processor       | Cost  | Latency band  | Use case |
 | --------------- | -----:| -------------:| -------- |
@@ -90,27 +105,26 @@ Pick a `processor` based on the depth and latency you need (cost is per 1 000 ru
 ## CLI
 
 ```bash
-# Works with zero credentials (free MCP)
-ai-v2 tools run websearch search "Recent funding for AI search startups"
+# No credentials: anonymous Tako search
+websearch search "Recent funding for AI search startups"
 
-# With PARALLEL_API_KEY: REST path + filters
-PARALLEL_API_KEY=... ai-v2 tools run websearch search \
-  "Recent funding for AI search startups" \
+# With a granted TAKO_API_KEY: REST path with filters
+websearch search "Recent funding for AI search startups" \
   --include-domain techcrunch.com --include-domain reuters.com \
-  --max-age-hours 720 --pretty
+  --max-age-hours 720 --effort deep --pretty
+
+# Deep research on the Tako Answer Agent (requires TAKO_API_KEY)
+websearch deep-research "comparison of L2 rollup economics" --effort high --pretty
 
-# Deep research (requires PARALLEL_API_KEY)
-PARALLEL_API_KEY=... ai-v2 tools run websearch deep-research \
-  "How should a fintech startup evaluate MPC vs HSM in 2026?" \
-  --processor pro-fast --pretty
+# Parallel instead, for the whole deployment
+WEBSEARCH_BACKEND=parallel websearch search "..." --pretty
 ```
 
 ### Backward-compatibility notes
 
-- Hidden CLI flags `--search-type`, `--max-iterations`, `--num-queries-per-iteration`, `--num-results-per-query` are accepted for prod-shaped invocations but emit a deprecation notice and are ignored under Parallel retrieval.
-- `meta.exa_request_ids` is retained as an alias of `meta.request_ids` so existing consumers of the original Exa-backed response shape still work.
-- `DeepResearchResponse.iterations` is retained as a single-element list per call (Parallel Task API is single-call rather than iterative).
-
-`meta.backend` in the JSON payload reports `parallel:mcp`, `parallel:api`, or `parallel:task:<processor>` so you can confirm which path served the call.
+- `--mode basic|advanced` still parses as a hidden flag and maps to `--effort instant|fast` with a deprecation note. Passing both is an error.
+- `--processor` still parses as a hidden flag. On Parallel it overrides `--effort`; on Tako it's recorded in `meta.partial_failures`.
+- Hidden flags `--search-type`, `--max-iterations`, `--num-queries-per-iteration`, `--num-results-per-query` are accepted, warn, and are ignored.
+- `meta.exa_request_ids` mirrors `meta.request_ids`. `DeepResearchResponse.iterations` stays a single synthetic entry.
 
-`meta.estimated_cost_usd` is a best-effort estimate from Parallel's published list prices ([pricing](https://docs.parallel.ai/getting-started/pricing)) — `0.0` on the free MCP path, `$0.005`+ per REST search, and the per-run processor price for `deep_research`. The API does not return billed cost, so treat this as an estimate that can drift if prices change.
+`meta.estimated_cost_usd`: on Tako REST it's the response's `usage.total_cost_usd` when present, else the rate table ($0.007 for `instant`/`fast`, $0.012 for `deep`); `0.0` on anonymous paths; the run's actual cost for the Answer Agent. On Parallel it's an estimate from list prices.

tools/research/websearch/__init__.py

@@ -1 +1 @@
-"""Web search and deep research via Parallel Web Systems."""
+"""Web search and deep research via Tako (default) or Parallel."""

tools/research/websearch/_common.py

@@ -0,0 +1,90 @@
+"""Constants and text helpers shared by every websearch backend."""
+
+from __future__ import annotations
+
+import json
+from importlib.metadata import PackageNotFoundError, version
+from typing import Any
+
+import httpx
+
+from .models import SourceDocument
+
+TOOL_NAME = "centaur-websearch"
+DISTRIBUTION_NAME = "websearch"
+FALLBACK_VERSION = "0.3.0"
+
+
+def _installed_version() -> str:
+    try:
+        return version(DISTRIBUTION_NAME)
+    except PackageNotFoundError:
+        return FALLBACK_VERSION
+
+
+TOOL_VERSION = _installed_version()
+USER_AGENT = f"{TOOL_NAME}/{TOOL_VERSION}"
+
+
+def append_within_budget(body: str, trailer: str, max_chars: int) -> str:
+    """Append `trailer` to `body` within `max_chars`, truncating only the body.
+
+    The trailer carries citation integrity (a `## Sources` block or an
+    attribution footer), so it is never sliced. If the trailer alone exceeds
+    the budget, the cap is exceeded rather than the citation map corrupted.
+    """
+    body = body.rstrip()
+    if len(body) + len(trailer) <= max_chars:
+        return body + trailer
+    body_budget = max(0, max_chars - len(trailer))
+    return body[:body_budget].rstrip() + trailer
+
+
+def render_sources_block(sources: list[SourceDocument]) -> str:
+    if not sources:
+        return ""
+    lines = [f"[{source.source_id}] {source.title} — {source.url}" for source in sources]
+    return "\n\n## Sources\n" + "\n".join(lines)
+
+
+def decode_jsonrpc_response(
+    response: httpx.Response, request_id: str | None = None
+) -> dict[str, Any]:
+    """Decode a JSON-RPC reply that arrived as JSON or as an SSE body.
+
+    For SSE, the reply is the event whose `id` matches `request_id`; a server
+    that interleaves notifications sends them on the same stream. When no frame
+    carries the id, the last event that parses as a JSON object wins.
+    """
+    content_type = response.headers.get("content-type", "")
+    text = response.text
+    if "text/event-stream" not in content_type:
+        if not text.strip():
+            return {}
+        try:
+            return json.loads(text)
+        except json.JSONDecodeError:
+            return {}
+    latest: dict[str, Any] | None = None
+    matched: dict[str, Any] | None = None
+    for event_block in text.split("\n\n"):
+        data_lines = [
+            line[len("data:") :].lstrip(" ")
+            for line in event_block.splitlines()
+            if line.startswith("data:")
+        ]
+        payload_text = "\n".join(data_lines).strip()
+        if not payload_text:
+            continue
+        try:
+            parsed = json.loads(payload_text)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(parsed, dict):
+            latest = parsed
+            if request_id is not None and parsed.get("id") == request_id:
+                matched = parsed
+    reply = matched if matched is not None else latest
+    if reply is None:
+        raise RuntimeError("JSON-RPC endpoint returned an empty SSE stream.")
+    return reply