Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_get_platform is quite flaky #2033

Closed
1 task done
booxter opened this issue Jan 18, 2025 · 2 comments
Closed
1 task done

test_get_platform is quite flaky #2033

booxter opened this issue Jan 18, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@booxter
Copy link

booxter commented Jan 18, 2025

Confirm this is an issue with the Python library and not an underlying OpenAI API

  • This is an issue with the Python library

Describe the bug

The test case often fails when I try to run it while building the nixpkgs package for the library, as follows:

 nix build --file . python313Packages.openai  --rebuild
error: builder for '/nix/store/28s74skxza9qfcqa1a92dvfz00q6capd-python3.13-openai-1.59.8.drv' failed with exit code 1;
       last 25 log lines:
       > tests/test_streaming.py ..................                               [ 85%]
       > tests/test_transform.py ................................................ [ 97%]
       >                                                                          [ 97%]
       > tests/test_utils/test_logging.py .....                                   [ 98%]
       > tests/test_utils/test_proxy.py .                                         [ 98%]
       > tests/test_utils/test_typing.py .....                                    [100%]
       >
       > =================================== FAILURES ===================================
       > ______________________ TestAsyncOpenAI.test_get_platform _______________________
       > tests/test_client.py:1801: in test_get_platform
       >     process.wait(2)
       > /nix/store/wn0n52lnysbscn702gfp20sx96cryiwp-python3-3.13.1/lib/python3.13/subprocess.py:1274: in wait
       >     return self._wait(timeout=timeout)
       > /nix/store/wn0n52lnysbscn702gfp20sx96cryiwp-python3-3.13.1/lib/python3.13/subprocess.py:2052: in _wait
       >     raise TimeoutExpired(self.args, timeout)
       > E   subprocess.TimeoutExpired: Command '['/nix/store/wn0n52lnysbscn702gfp20sx96cryiwp-python3-3.13.1/bin/python3.13', '-c', '\nimport asyncio\nimport nest_asyncio\nimport threading\n\nfrom openai._utils import asyncify\nfrom openai._base_client import get_platform \n\nasync def test_main() -> None:\n    result = await asyncify(get_platform)()\n    print(result)\n    for thread in threading.enumerate():\n        print(thread.name)\n\nnest_asyncio.apply()\nasyncio.run(test_main())\n']' timed out after 2seconds
       >
       > The above exception was the direct cause of the following exception:
       > tests/test_client.py:1806: in test_get_platform
       >     raise AssertionError("calling get_platform using asyncify resulted in a hung process") from e
       > E   AssertionError: calling get_platform using asyncify resulted in a hung process
       > =============================== inline snapshot ================================
       > =========================== short test summary info ============================
       > FAILED tests/test_client.py::TestAsyncOpenAI::test_get_platform - AssertionError: calling get_platform using asyncify resulted in a hung process
       > ================= 1 failed, 406 passed, 5 deselected in 20.23s =================
       For full logs, run 'nix log /nix/store/28s74skxza9qfcqa1a92dvfz00q6capd-python3.13-openai-1.59.8.drv'.

I believe the 2-second timeout is too short for the machine. This is M3 Pro btw, (though it often runs other builds in parallel, so it may be somewhat short on CPU.) The nature of the test case doesn't allow for an easy solution; perhaps a simple timeout bump to e.g. 10 seconds could be helpful? I understand this locks a python test thread for sleeping though - not a great use of a core.

To Reproduce

  1. Run test suite on a loaded machine.

Code snippets

OS

macOS

Python version

Python 3.13.1

Library version

openai 1.59.8

@booxter booxter added the bug Something isn't working label Jan 18, 2025
@Programmer-RD-AI

This comment has been minimized.

@RobertCraigie
Copy link
Collaborator

@booxter this should be fixed in the next release https://github.com/openai/openai-python/pull/2040/files, I bumped the timeout to 10 seconds and moved to a polling based approach instead of just using an arbitrary timeout. Let me know if you're still running into this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants