Skip to content

agent-eval: capture the Storybook startup log when the MCP readiness check fails #35961

Description

@kasperpeulen

Describe the problem

When start-storybook-mcp.mjs times out ("Storybook MCP server did not become ready … within 60000ms. See /tmp/storybook-mcp.log"), the eval run fails during sandbox npm install and the sandbox is torn down — but dumpMcpDebug() (which snapshots the landing page, MCP initialize response, and /tmp/storybook-mcp.log into .storybook/mcp-debug/) only runs on the success path. The one file that explains the failure is unreachable from the saved results.

Concrete case: a stale local addon-mcp dist (importing a since-renamed storybook/open-service export, MCP_TOOL_NAMES) made Storybook crash on preset load in every sandbox. The result snapshot only said "npm install failed / not ready within 60000ms"; diagnosing required rebuilding the whole sandbox workspace locally to read the log.

Suggested fix

On the timeout path, copy /tmp/storybook-mcp.log (and a best-effort fetch of the landing page) into the workspace before throwing, so the eval result snapshot includes it. A staleness guard would also help: the harness could warn when the packed local-packages dist imports names the pinned npm core does not export, or simply recommend yarn nx run-many -t compile -p addon-mcp mcp in the README's local-run steps.

Metadata

Metadata

Assignees

No one assigned

    Labels

    internal-qaIssues reported by the Storybook team, potentially on unreleased or prerelease software.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions