ADR-0086: WebReaper.Mcp.AspNetCore Streamable HTTP MCP transport (n8n / remote clients)

[alex-on-ai]

Design of record: ADR-0086 (PR #238). This PRD operationalizes it. Related: ADR-0049 (MCP satellite, stdio), ADR-0073 (MCP bakes Cdp), ADR-0083 (escalating page loader), ADR-0084 (WebReaper.AI.Http client + WEBREAPER_LLM_*), ADR-0085 (climb-progress observer).

Problem Statement

I run n8n and want to call WebReaper's scrape / map / extract tools from my workflows. WebReaper ships an MCP server, but I cannot connect n8n to it. n8n's built-in MCP Client Tool node only accepts a server URL (Streamable HTTP; its SSE option is deprecated), while WebReaper's MCP server (WebReaper.Mcp, ADR-0049) speaks stdio only, a locally spawned child process. n8n running in Docker or n8n Cloud has no way to spawn that binary, so the server is unreachable. The same wall hits any hosted or remote MCP client. The earlier-noted missing tools (whole-site crawl, stealth) are a secondary concern; the binding constraint is that the server cannot be reached at all.

Solution

WebReaper ships a second MCP host that listens over HTTP. I point n8n's MCP Client node at its URL with a bearer token, and the same tools (scrape, map, extract, extract_with_prompt) are available, plus a bounded whole-site crawl. I run it as a container next to n8n; it either has Chromium baked in or I point it at a browser sidecar via an environment variable for JS-rendered pages. My LLM key lives in the server's environment, so prompt-based extraction works without putting secrets in the workflow. The existing stdio server is unchanged for local agents (Cursor, Claude Desktop).

User Stories

1. As an n8n workflow builder, I want to connect n8n's MCP Client node to WebReaper via a URL, so that I can use its tools without running a local stdio process.
2. As an n8n workflow builder, I want to authenticate with a bearer token, so that my scraper endpoint is not open to the internet.
3. As an n8n workflow builder, I want to call scrape and get a page as clean Markdown, so that I can feed it into downstream nodes.
4. As an n8n workflow builder, I want to call map and get a site's URLs, so that I can iterate over them with n8n's Split Out / Loop nodes.
5. As an n8n workflow builder, I want to call extract with a CSS schema, so that I get structured JSON from a known page shape.
6. As an n8n workflow builder, I want to call extract_with_prompt with a natural-language instruction, so that I get structured data without writing selectors.
7. As an n8n workflow builder, I want to override the model on a single extract_with_prompt call, so that I can use a cheap model for easy pages and a stronger one for hard pages without restarting the server.
8. As an n8n workflow builder, I want a bounded crawl for a small site in one node, so that I do not have to build a map-and-loop for a trivial sweep.
9. As an n8n workflow builder, I want the crawl tool to state plainly that it is a single long call with no progress feedback, so that I set my node timeout correctly or choose map-and-scrape instead.
10. As an n8n workflow builder, I want the documented map then Split then scrape pattern, so that large sweeps stay within node timeouts and get per-URL retries inside the workflow.
11. As an n8n workflow builder, I want browser=true for JS-rendered pages, so that single-page-app content is captured.
12. As an n8n workflow builder, I want tool descriptions and errors that tell me exactly what to set, so that I can self-serve without reading source.
13. As a self-hosting operator, I want to run the HTTP MCP server as a Docker container, so that it sits in the same compose network as my n8n.
14. As a self-hosting operator, I want Chromium baked into the image, so that browser=true works out of the box.
15. As a self-hosting operator, I want to point the server at an external CDP endpoint via WEBREAPER_CDP_URL, so that I can share a browserless pool instead of launching Chromium per call.
16. As a self-hosting operator, I want to cap concurrent browser launches via an environment variable, so that parallel workflow executions do not exhaust host memory.
17. As a self-hosting operator, I want to configure the LLM endpoint, model, and key via environment, so that prompt extraction is available to all workflows without per-call secrets.
18. As a self-hosting operator, I want an actionable startup error if I bind a non-loopback interface without a token, so that I cannot accidentally expose an unauthenticated scraper.
19. As a self-hosting operator, I want an actionable error when an LLM tool is called without LLM config, so that misconfiguration is obvious rather than silent.
20. As a self-hosting operator, I want a /health endpoint, so that my orchestrator can health-check the container.
21. As a self-hosting operator, I want to run bound to loopback with no token for local dev, so that I can try it in seconds.
22. As a self-hosting operator, I want the server to tear down any spawned Chromium on shutdown and per call, so that I do not leak browser processes.
23. As an MCP client developer, I want any Streamable-HTTP MCP client to reach the URL, so that WebReaper is usable beyond local stdio.
24. As an MCP client developer, I want tools/list to enumerate all tools, so that my client auto-populates them.
25. As a Cursor / Claude Desktop user, I want the stdio server to keep working unchanged, so that the local-spawn path is not regressed.
26. As a security-conscious operator, I want auth required off-loopback, so that an open scraper is not an SSRF amplifier on my network.
27. As an n8n workflow builder, I want a browser scrape to auto-climb to a stronger tier on a detected block (server-side escalating loader), so that a bot-checked page can still come back.
28. As a WebReaper maintainer, I want one shared tools class feeding both hosts, so that a new tool appears on stdio and HTTP at once.
29. As a WebReaper maintainer, I want the new package on the release CANDIDATES list and a container image published per release, so that versions stay aligned and operators can pull a tagged server.
30. As a WebReaper maintainer, I want the transport, auth, and config logic gated by tests in CI, so that it cannot regress.

Implementation Decisions

• New satellite WebReaper.Mcp.AspNetCore. References WebReaper.Mcp (to reuse the WebReaperTools [McpServerToolType] class) and ModelContextProtocol.AspNetCore. Two hosts, one tools class: the stdio console host (ADR-0049, unchanged) and this ASP.NET Core HTTP host. The host is mostly glue: build a web application, AddMcpServer().WithHttpTransport(stateless).WithTools<WebReaperTools>(), map the MCP endpoint at root, add auth middleware and a health endpoint.
• Transport. Streamable HTTP with Stateless = true. Legacy SSE is not enabled (deprecated in both the C# SDK and n8n; stateful-only with backpressure caveats). The MCP endpoint is mapped at the root path; n8n points its MCP Endpoint URL at the server root.
• Auth. A bearer token read from WEBREAPER_MCP_TOKEN, enforced by middleware that returns 401 on a missing or wrong token. The accept-or-reject logic is a pure decision module the middleware calls. Startup refuses to bind a non-loopback interface when no token is set; loopback with no token is allowed for local dev.
• Config module (deep). A pure parser and validator: environment values into a validated options record, or an actionable error. Owns WEBREAPER_MCP_TOKEN, WEBREAPER_CDP_URL, WEBREAPER_MCP_MAX_CONCURRENT_BROWSERS, the bind address, and the WEBREAPER_LLM_* set. Mirrors the CLI's ScrapeContext / CrawlContext parsing seam. Holds the non-loopback-without-token rule.
• Tool surface. The four existing single-page tools stay the documented default. Add a fifth tool crawl(url, max_pages, browser): a bounded whole-site sweep mirroring the CLI CrawlCommand shape (Crawl(url).Sweep(...).PageCrawlLimit(max_pages) then collect records). Default max_pages is 50 (not the CLI's 1000) with a hard clamp; the tool description states it is one long blocking call with no progress feedback and recommends map-and-scrape for large sweeps.
• extract_with_prompt per-call model. Gains an optional model parameter; the API key stays environment-only via the ADR-0084 OpenAiCompatibleChatClient from WebReaper.AI.Http. The key is never a tool parameter.
• Browser-transport selection (deep, shared). A pure closed-sum decision keyed on the browser argument and WEBREAPER_CDP_URL: NoBrowser | LaunchManagedChromium | ConnectToCdp(url). The tool layer turns the decision into the page-loader wiring. Per the PRD sign-off this lives in the shared WebReaper.Mcp, so the stdio host also gains CDP-sidecar support and both hosts share one browser-wiring path. (This widens ADR-0086, which framed WEBREAPER_CDP_URL as HTTP-host-only; the ADR will be amended.)
• Concurrency. A semaphore caps managed-Chromium launches via WEBREAPER_MCP_MAX_CONCURRENT_BROWSERS. When WEBREAPER_CDP_URL is set no launch happens, so the sidecar's own pool governs concurrency instead.
• Engine lifecycle. Each tool call builds and runs its own engine and disposes it on scope exit (ADR-0058 teardown chain), so a spawned Chromium dies with the call; this matches the stateless transport and the ADR-0049 no-shared-state property.
• Server-side escalation. Browser calls run through the escalating page loader (ADR-0083), so a blocked page climbs within the call.
• Infra. A Chromium-baked Dockerfile modeled on the playground Tier-B lean image, plus a docker-compose example showing the server next to n8n with an optional browserless sidecar. A /health endpoint. Add WebReaper.Mcp.AspNetCore to the release CANDIDATES list and publish a versioned container image. n8n wiring docs (URL, Bearer token, transport = Streamable HTTP).
• Not AOT. ASP.NET Core host plus MCP SDK reflection paths; consistent with ADR-0049's no-PublishAot stance for the MCP satellite.

Testing Decisions

A good test here exercises external behavior, not implementation: the pure decision modules are closed truth tables with no IO; the host smoke is a real protocol handshake against the deterministic local test site. All four targets below are in scope (confirmed with the developer).

• Config parser/validator. Valid environment into a populated options record; missing or malformed values into an actionable error; non-loopback bind without a token is refused; loopback with no token is allowed.
• Bearer-auth decision. Correct token into Authorized; missing or wrong token into Unauthorized; comparison is fixed-time.
• Browser-transport selection plus crawl-arg bounding. browser=false into NoBrowser; browser=true with no CDP url into LaunchManagedChromium; browser=true with WEBREAPER_CDP_URL set into ConnectToCdp(url). Crawl bounding: unset max_pages defaults to 50; a value above the cap clamps; a non-positive value is rejected.
• HTTP host smoke (end to end). Host the HTTP server and drive a real initialize then tools/list (sees all five tools) then tools/call for scrape / map / extract against the local test site. Auth path: a call without the token is rejected; with the token it succeeds.

Prior art to follow:

• WebReaper.Cli.Tests/EscalationPlanTests.cs: a pure flag-to-decision truth table (the model for the config, auth, and transport-selection modules).
• WebReaper.Cli.Tests/ScrapeContextTests.cs and CrawlContextTests.cs: context parsing tests (the model for the config validator).
• WebReaper.IntegrationTests/McpServerTests.cs: spawns the real MCP server and drives a true initialize / tools/list / tools/call handshake with the official SDK client against the LocalTestSite; the HTTP smoke is its Streamable-HTTP analog (SDK HTTP client transport or an in-process test host), tagged [Trait("Category","Mcp")] so the fast gate skips the subprocess/network cost.

Out of Scope

• Multi-tenant auth and per-tenant LLM or browser. That is the WebReaper Cloud milestone; this server is single-tenant and self-run.
• Streaming or progress notifications over MCP. The crawl tool stays a single blocking call; surfacing the ADR-0085 climb-progress observer as MCP progress notifications is a possible later ADR.
• The stealth tier in MCP. The satellite bakes only WebReaper.Cdp (ADR-0073); stealth Chromium forks remain library and CLI only, so a site that defeats a vanilla browser is not served by MCP browser mode.
• Authenticated or session scraping (cookies) over MCP.
• An --infer-style learned-schema MCP tool. extract_with_prompt covers schema-free extraction; an inferred-schema tool can be a later addition.
• Auto-configuration of n8n or other MCP clients. Manual URL-and-token wiring is documented.
• Rate limiting. Flagged as an operational concern, not built here.
• A dotnet tool packaging of the server (permanently deferred per project stance).

Further Notes

• ADR-0086 is the design of record; this PRD operationalizes it. One refinement against the ADR text: the WEBREAPER_CDP_URL and browser-transport-selection logic lands in the shared WebReaper.Mcp (so the stdio host gains it), not HTTP-host-only as the ADR first framed. The ADR should be amended to reflect this once the PRD is accepted.
• The crawl default of 50 pages is deliberate given there is no streaming (ADR-0049 bounded scope plus ADR-0086); it keeps a one-node sweep from silently running past an n8n node timeout.
• n8n specifics, verified during the ADR-0086 grilling against current n8n and C# SDK docs: the built-in MCP Client Tool node, Server Transport set to Streamable HTTP (not the deprecated SSE), MCP Endpoint URL set to the server root, Authentication set to Bearer.
• Security: a network-bound scraper is an SSRF amplifier, hence the mandatory-token-off-loopback rule. The playground's network-layer SSRF guard (cloud/WebReaper.PlaygroundApi) is related prior art for server-side scrape safety; this PRD does not require an SSRF allowlist for a self-run single-tenant server, but notes it as a possible future hardening.