[Bug]: can't run tests with pytest from repo root

[jamesbraza]

What happened?

This should work:

git clone [email protected]:BerriAI/litellm.git
cd litellm
python -m venv venv && source venv/bin/activate
pip install poetry
poetry install --with dev
pytest tests # Fails
pytest litellm/tests # Fails

Second to last line, I get the below error:

Test session starts (platform: darwin, Python 3.11.8, pytest 8.1.1, pytest-sugar 1.0.0)
rootdir: /Users/jamesbraza/code/litellm
configfile: pyproject.toml
plugins: sugar-1.0.0, anyio-4.3.0, time-machine-2.14.0, timer-1.0.0, asyncio-0.23.5.post1
asyncio: mode=Mode.STRICT
collecting ...
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_keys.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_keys.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_keys.py:15: in <module>
    from litellm.proxy._types import LitellmUserRoles
E   ModuleNotFoundError: No module named 'litellm.proxy'

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_ratelimit.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_ratelimit.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_ratelimit.py:17: in <module>
    from litellm import utils, Router
E   ImportError: cannot import name 'utils' from 'litellm' (unknown location)

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_whisper.py ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_whisper.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_whisper.py:26: in <module>
    from litellm import Router
E   ImportError: cannot import name 'Router' from 'litellm' (unknown location)

Last line, I get the below error:

ImportError while loading conftest '/path/to/litellm/litellm/tests/conftest.py'.
litellm/tests/conftest.py:12: in <module>
    import litellm
E   ModuleNotFoundError: No module named 'litellm'

Can we fix the Poetry --> pytest workflow

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

@jamesbraza what needs to be changed here?

cd litellm/tests
pytest . -x -vv

works

cd litellm/litellm/tests
pytest . -x -vv

works

[jamesbraza]

So check this:

git clone [email protected]:BerriAI/litellm.git
cd litellm
python -m venv venv && source venv/bin/activate
pip install poetry
poetry install --with dev
pytest tests # Fails with ImportError: cannot import name 'Router' from 'litellm' (unknown location)
cd tests
pytest # Fails with ModuleNotFoundError: No module named 'openai.types.beta.threads.message_content'
cd ../litellm/tests
pytest # Fails with ModuleNotFoundError: No module named 'openai.types.beta.threads.message_content'
pytest . # Fails with ModuleNotFoundError: No module named 'fastapi', 'opentelemetry', 'backoff'

I am more just sharing that LiteLLM is somewhat sensitive to starting directory because pytest only works when run from specific directories with specific parameterizations (e.g. specifying a file_or_dir of .).

If one sets things up nicely (e.g. specifying testpaths in a config), one should just be able to invoke pytest from the repo root and have everything work.

One other issue is that poetry install --with dev doesn't seem to install everything necessary to run tests. Maybe there should be more Poetry groups made, or the dev group should be expanded

---

Also, feel free to close this out. I am more just pointing out the "getting started" developing has opportunities for improvement

[krrishdholakia]

making this a discussion, as this is useful feedback

Would welcome a pr here @jamesbraza

[jamesbraza]

To share, my pytest config in the repo root's pyproject.toml:

[tool.pytest.ini_options]
addopts = "--doctest-modules"
filterwarnings = [
    ...
]
norecursedirs = [
    ...
]
session_timeout = 2400  # From pytest-timeout
# a and b are for tests
testpaths = ["a", "b", "tests"]
timeout = 600  # From pytest-timeout

This can enable you to just type pytest and have it run everything.

---

One question is why is there two tests folders? It would be useful for LiteLLM to consolidate them imo because then there's less ways to need to invoke tests

[krrishdholakia]

why is there two tests folders

needs to be refactored. It just started with the nested test and we needed one outside to test the proxy.

We should probably move all tests to the top level

[jamesbraza]

One other thing is, I advise you:

• Consolidate your mypy.ini and ruff.toml into the repo root's pyproject.toml, just to keep file count down at repo root
• Remove the log.txt in the repo root (seems to have been an accidental commit)