Script timeout in Chrome Canary with `selenium_jspi` fixture

[illia-v]

Hi there,

In the urllib3 CI, we have a job that tests our library against Chrome Canary. For the past few weeks, this job has been consistently failing due to a script timeout. The last successful run was with Chrome 139.0.7249.0, and the first failure we observed was with Chrome 139.0.7254.0.

The timeout appears to be specific to the selenium_jspi fixture, as the tests pass when using the plain selenium fixture. I was able to reproduce this issue by running the pytest-pyodide test suite with Google Chrome 140.0.7301.0.

test_jspi[chrome] details of pytest -v --runtime=chrome

__________________________________________________________________________________ test_jspi[chrome] ___________________________________________________________________________________

self = <pytest_pyodide.runner.SeleniumChromeRunner object at 0x7f24f91be350>
code = '\n        async def __tmp():\n            __tracebackhide__ = True\n\n            from pytest_pyodide.decorator impor...n\n        try:\n            result = await __tmp()\n        finally:\n            del __tmp\n        result\n        '

    def run_async(self, code):
>       return self.run_js(
            f"""
            await pyodide.loadPackagesFromImports({code!r})
            let result = await pyodide.runPythonAsync({code!r});
            return pyodide.$handleTestResult(result);
            """
        )

pytest_pyodide/runner.py:235: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pytest_pyodide/runner.py:264: in run_js
    return self.run_js_inner(code, check_code)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pytest_pyodide/runner.py:364: in run_js_inner
    retval = self.driver.execute_async_script(wrapper % (code, check_code))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../.virtualenvs/pytest-pyodide/lib/python3.13/site-packages/selenium/webdriver/remote/webdriver.py:426: in execute_async_script
    return self.execute(command, {"script": script, "args": converted_args})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../.virtualenvs/pytest-pyodide/lib/python3.13/site-packages/selenium/webdriver/remote/webdriver.py:347: in execute
    self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7f24f91bec50>
response = {'status': 500, 'value': '{"value":{"error":"script timeout","message":"script timeout\\n  (Session info: chrome=140.0...3Cunknown>\\n#15 0x5599747d1c13 \\u003Cunknown>\\n#16 0x7f0b05368724 start_thread\\n#17 0x7f0b053ec80c __clone3\\n"}}'}

    def check_response(self, response: Dict[str, Any]) -> None:
        """Checks that a JSON response from the WebDriver does not have an
        error.
    
        :Args:
         - response - The JSON response from the WebDriver server as a dictionary
           object.
    
        :Raises: If the response contains an error message.
        """
        status = response.get("status", None)
        if not status or status == ErrorCode.SUCCESS:
            return
        value = None
        message = response.get("message", "")
        screen: str = response.get("screen", "")
        stacktrace = None
        if isinstance(status, int):
            value_json = response.get("value", None)
            if value_json and isinstance(value_json, str):
                import json
    
                try:
                    value = json.loads(value_json)
                    if len(value) == 1:
                        value = value["value"]
                    status = value.get("error", None)
                    if not status:
                        status = value.get("status", ErrorCode.UNKNOWN_ERROR)
                        message = value.get("value") or value.get("message")
                        if not isinstance(message, str):
                            value = message
                            message = message.get("message")
                    else:
                        message = value.get("message", None)
                except ValueError:
                    pass
    
        exception_class: Type[WebDriverException]
        e = ErrorCode()
        error_codes = [item for item in dir(e) if not item.startswith("__")]
        for error_code in error_codes:
            error_info = getattr(ErrorCode, error_code)
            if isinstance(error_info, list) and status in error_info:
                exception_class = getattr(ExceptionMapping, error_code, WebDriverException)
                break
        else:
            exception_class = WebDriverException
    
        if not value:
            value = response["value"]
        if isinstance(value, str):
            raise exception_class(value)
        if message == "" and "message" in value:
            message = value["message"]
    
        screen = None  # type: ignore[assignment]
        if "screen" in value:
            screen = value["screen"]
    
        stacktrace = None
        st_value = value.get("stackTrace") or value.get("stacktrace")
        if st_value:
            if isinstance(st_value, str):
                stacktrace = st_value.split("\n")
            else:
                stacktrace = []
                try:
                    for frame in st_value:
                        line = frame.get("lineNumber", "")
                        file = frame.get("fileName", "<anonymous>")
                        if line:
                            file = f"{file}:{line}"
                        meth = frame.get("methodName", "<anonymous>")
                        if "className" in frame:
                            meth = f"{frame['className']}.{meth}"
                        msg = "    at %s (%s)"
                        msg = msg % (meth, file)
                        stacktrace.append(msg)
                except TypeError:
                    pass
        if exception_class == UnexpectedAlertPresentException:
            alert_text = None
            if "data" in value:
                alert_text = value["data"].get("text")
            elif "alert" in value:
                alert_text = value["alert"].get("text")
            raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
>       raise exception_class(message, screen, stacktrace)
E       selenium.common.exceptions.TimeoutException: Message: script timeout
E         (Session info: chrome=140.0.7301.0)
E       Stacktrace:
E       #0 0x5599747d2c3a <unknown>
E       #1 0x559974278673 <unknown>
E       #2 0x559974317524 <unknown>
E       #3 0x5599742f0222 <unknown>
E       #4 0x559974315ec2 <unknown>
E       #5 0x5599742efff3 <unknown>
E       #6 0x5599742bcbe8 <unknown>
E       #7 0x5599742bd861 <unknown>
E       #8 0x559974797548 <unknown>
E       #9 0x55997479b282 <unknown>
E       #10 0x55997477ea49 <unknown>
E       #11 0x55997479bdc5 <unknown>
E       #12 0x5599747637cf <unknown>
E       #13 0x5599747bfee8 <unknown>
E       #14 0x5599747c00c2 <unknown>
E       #15 0x5599747d1c13 <unknown>
E       #16 0x7f0b05368724 start_thread
E       #17 0x7f0b053ec80c __clone3

../../.virtualenvs/pytest-pyodide/lib/python3.13/site-packages/selenium/webdriver/remote/errorhandler.py:229: TimeoutException

It seems like there might have been a change in recent Chrome Canary 139 & 140 versions that affects the selenium_jspi fixture.

Thanks for your help!

[hoodmane]

Thanks for the report @illia-v. The change that jspi makes is here:
https://github.com/pyodide/pytest-pyodide/blob/main/pytest_pyodide/runner.py?plain=1#L452-L454

It's possible that they removed --enable-features=WebAssemblyExperimentalJSPI or that --enable-experimental-webassembly-features picked up some new feature that is breaking selenium.

JSPI is on by default since Chrome 137 so if you are only testing against versions newer than that, you can replace it with selenium fixture and add the runtime fixture and the following skip to the test:

    if runtime in ["firefox", "safari"]:
        pytest.skip(f"jspi not supported in {runtime}")

[illia-v]

JSPI is on by default since Chrome 137 so if you are only testing against versions newer than that, you can replace it with selenium fixture

Thanks for the suggestion, it helped a lot!

[hoodmane]

That looks more annoying than I thought since you still need jspi fixture for node. I think I'll update pytest-pyodide and release an update to reduce the effort for you.

[hoodmane]

Okay opened #167. We can make a new release with this change and that should make life easier.

[illia-v]

That looks more annoying than I thought since you still need jspi fixture for node.

Yes, I had to make additional changes to make it work with just one selenium fixture, but I feel like the result reduced technical debt for the integration.
Additionally, time to run Chrome tests in urllib3 drops from ~5m to ~3m when only one fixture is used for both JSPI and non-JSPI tests in urllib3.

[hoodmane]

Surprising that it uses such a large amount of time starting the browser.