[bug] preVisit accesses the story store before the index is ready

[sergei-maertens]

Describe the bug

We have the test-runner.js implement with the preVisit to apply the different viewports when running tests, per the README.

After upgrading to Storybook 8, our tests fail in a flaky way, e.g.:

 FAIL  src/components/PostCompletionViews/PostCompletionView.stories.js
  ● Private API / Post completion views  › Generic › smoke-test

    page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.

    It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
    remove access to the store entirely

      at Object.get (http:/127.0.0.1:6006/sb-preview/runtime.js:118:5542)
      at __getContext (<anonymous>:215:54)
      at eval (eval at evaluate (:226:30), <anonymous>:1:29)
      at UtilityScript.evaluate (<anonymous>:228:17)
      at UtilityScript.<anonymous> (<anonymous>:1:44)
      at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:15306:15)
      at getStoryContext (.storybook/test-runner.js:9:27)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:151:64
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at src/components/PostCompletionViews/PostCompletionView.stories.js:19:16
      at testFn (src/components/PostCompletionViews/PostCompletionView.stories.js:211:49)
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:224:33)
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:19:16)

In the browser, there are no problems, only happens when using the test-runner.

To Reproduce

Unfortunately I don't see a way to reproduce this reliably, but our repository is public: https://github.com/open-formulieren/open-forms-sdk/tree/chore/662-sb-upgrade

# clone repo
npm ci
npm run storybook

and in a different tab/shell:

npm run test:storybook

Expected behavior

Tests only run when the story story is ready

Screenshots
n/a

System

Storybook Environment Info:

  System:
    OS: Linux 6.7 Arch Linux
    CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.21 - /usr/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm <----- active
    pnpm: 8.15.2 - /usr/bin/pnpm
  npmPackages:
    @storybook/addon-actions: ^8.0.0 => 8.0.0 
    @storybook/addon-coverage: ^1.0.1 => 1.0.1 
    @storybook/addon-essentials: ^8.0.0 => 8.0.0 
    @storybook/addon-interactions: ^8.0.0 => 8.0.0 
    @storybook/addon-links: ^8.0.0 => 8.0.0 
    @storybook/addon-webpack5-compiler-babel: ^3.0.3 => 3.0.3 
    @storybook/components: ^8.0.0 => 8.0.0 
    @storybook/manager-api: ^8.0.0 => 8.0.0 
    @storybook/preview-api: ^8.0.0 => 8.0.0 
    @storybook/react: ^8.0.0 => 8.0.0 
    @storybook/react-webpack5: ^8.0.0 => 8.0.0 
    @storybook/test: ^8.0.0 => 8.0.0 
    @storybook/test-runner: ^0.17.0 => 0.17.0 
    @storybook/types: ^8.0.0 => 8.0.0 
    msw-storybook-addon: ^1.10.0 => 1.10.0 
    storybook: ^8.0.0 => 8.0.0 
    storybook-addon-remix-react-router: ^3.0.0 => 3.0.0 
    storybook-addon-themes: ^6.1.0 => 6.1.0 
    storybook-react-intl: ^3.0.1 => 3.0.1 

Additional context

• I have an outdated theme addon that needs to be replaced still, no idea if that could be messing things up 🤔

• The relevant code throwing the error appears to be this: https://github.com/storybookjs/storybook/blob/718ae4ad3352d2bc69f95c0cfab25b4b075e2308/code/lib/preview-api/src/modules/preview-web/Preview.tsx#L106

[sergei-maertens]

Interestingly enough, this doesn't seem to happen with a production build since our CI is passing: https://github.com/open-formulieren/open-forms-sdk/actions/runs/8340149970/job/22823603724?pr=663

[AlmarAubel]

We have the same problem. If I add await waitForPageReady(page); i runs less flakky.

const config: TestRunnerConfig = {
  async preVisit(page, story) {
    await waitForPageReady(page);
    }
}

[ItaiCovver]

We have the same problem but adding waitForPageReady stalls indefinitely on this line

await page.waitForLoadState("networkidle")

when running locally and in our CI flow it does not seem to do anything (get the SB_PREVIEW_API_0011 errors). In both we run identical docker containers

Edit: Figured out the discrepancy. In local storybook is running via storybook dev and in CI its building storybook and running npx http-server storybook-static. Seems linked to this issue

[VizTheWiz]

I'm having the same issue, it happens when running storybook dev with the --ci flag (Without the CI flag, when the playground webpage is opened, this does not occur).
However, it seems to resolve if I immediately run test-runner after this fail, but it takes longer than usual (10s as opposed to around 4s). Another consecutive call to test-runner takes the expected duration.

This only seems to happen in one specific instance, which also relies on Apollo GraphQL client mocked provider and is overall more complex. Smaller standalone packages with simple components do not seem to have this issue, and the tests run successfully.

For reference, this is the output from storybook info, using Vite:

Storybook Environment Info:

  System:
    OS: Linux 6.5 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
    CPU: (12) x64 12th Gen Intel(R) Core(TM) i7-1265U
    Shell: 5.1.4 - /bin/bash
  Binaries:
    Node: 18.19.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.2.4 - /usr/local/bin/npm <----- active
    pnpm: 8.15.4 - /usr/local/share/npm-global/bin/pnpm
  npmPackages:
    @storybook/addon-essentials: ^8.0.5 => 8.0.5 
    @storybook/addon-interactions: ^8.0.5 => 8.0.5 
    @storybook/addon-links: ^8.0.5 => 8.0.5 
    @storybook/addon-onboarding: ^8.0.5 => 8.0.5 
    @storybook/blocks: ^8.0.5 => 8.0.5 
    @storybook/react: ^8.0.5 => 8.0.5 
    @storybook/react-vite: ^8.0.5 => 8.0.5 
    @storybook/test: ^8.0.5 => 8.0.5 
    @storybook/test-runner: ^0.17.0 => 0.17.0 
    eslint-plugin-storybook: ^0.8.0 => 0.8.0 
    storybook: ^8.0.5 => 8.0.5 

[Mil4n0r]

Any updates on this? I am having the exact same problem.

[noranda]

Same here! Same problem.

I'm doing the same thing where I'm using preVisit to set the viewport in some of our tests. In development mode, we see these errors when using test-runner. When building statically, we don't see the errors.

[kdrbek06]

Same issue here in postVisit hook

async postVisit(page, context) {
    await waitForPageReady(page);
    ...
}

[dorsharonfuse]

any update on this?
We're getting the same problem, and the tests run flaky because of it.

[shilman]

@yannbf Have you had a chance to take a look at this?

[Jero786]

Any update on this? I encountered the same issue too.

[jorgeortega]

With a very small set of stories, test-storybook --maxWorkers=1 worked for me locally. Haven't tried yet on CI.