Skip to content

tox is testing the source code instead of the package? #2359

Description

@galsasi1989

Hi
I have a project called 'sdk':

sdk/
    sdk/
       __init__.py
       file1.py
       file2.py
    tests
    setup.py
    tox.ini
    pytest.ini

This is the content of the tox.ini:

[tox]
envlist = py{37,38}
requires = tox-wheel
sitepackages = true

[testenv]
wheel = true
setenv =
    PIP_INDEX_URL = https://<artifactory_url>/artifactory/api/pypi/pypi/simple
    PACKAGE_VERSION = {env:PACKAGE_VERSION:}
    PYTHON_VERSION = {envname}
    JENKINS_URL = {env:JENKINS_URL:}
deps =
    pytest >=6.1.2,<7
    moto[s3,sqs] >=2.0.7,<3

whitelist_externals = python

commands =
    python -m pytest --color=yes --show-capture=no -v --junitxml=pytest-results-pypi-package-{envname}-unitests.xml

These are first 4 lines in init.py:

import os
import logging
import structlog
from sdk.file1 import validate  # validate function is existed in file1.py

I was wondering whether tox is testing the package which is created and installed in a new virtual environment or the source code in the file system
I would expect that the package will be tested and not the source code. So in order to be sure, I did a test with the following tox-test.ini:

[tox]
envlist = py{37,38}
sitepackages = true
skipsdist = true

[testenv]
setenv =
    PIP_INDEX_URL = https://<artifactory_url>/artifactory/api/pypi/pypi/simple
    PACKAGE_VERSION = {env:PACKAGE_VERSION:}
    PYTHON_VERSION = {envname}
    JENKINS_URL = {env:JENKINS_URL:}
deps =
    pytest >=6.1.2,<7
    moto[s3,sqs] >=2.0.7,<3
    sdk ==1.19.0

whitelist_externals = python

commands =
    python -m pytest --color=yes --show-capture=no -v --junitxml=pytest-results-{envname}-unitests.xml

In the above tox-test.ini, I am just asking tox to skip on creating the package, but only to create a virtualenv with all packages mentioned in deps and run the tests .

But, before running tox using the command: tox -c tox-test.ini,` I renamed the sdk inner directory to be sdk-back and I created a new sdk directory with the same init.py file as in sdk-backup but file1.py and file2.py are now empty files. This is how my project look:

sdk/
    sdk/
       __init__.py
       file1.py    # empty file, the validate function **does not** exist in this file anymore
       file2.py    # empty file
   sdk-backup/
      __init__.py
     file1.py  # the original file1.py
     file2.py  # the original file2.py
    tests
    setup.py
    tox.ini
    tox-test.ini
    pytest.ini

Now, when I am running tox -c tox-test.ini, I would expect that the tests will be run on the installed package sdk ==1.19.0 but then all the tests fail because the validate function from file1.py does not exist anymore, which is not the case in the sdk ==1.19.0.
As I understand it, the installed package sdk ==1.19.0 is not tested but only the source code

Am I missing something? How can I be sure that only the package which tox creates is tested and source code is ignored?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug:normalaffects many people or has quite an impact

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions