Conversation
dinhngtu
force-pushed
the
dnt/enable-marker
branch
from
c726566 to
1cef705
Compare
|
|
parametrize often looks like a good idea until we actually try to use it :) |
glehmann
left a comment
glehmann
left a comment
There was a problem hiding this comment.
Looks mostly good to me, but it is missing some documentation in the README.md.
If we add more excluded makers by default, we might want to add a --enable-all, but I don't think that's necessary for now.
Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
dinhngtu
force-pushed
the
dnt/enable-marker
branch
from
1cef705 to
8557c2f
Compare
|
Added documentation. Also rearranged pytest.ini to explicitly list the non-default markers. |
This avoids us having to specify 'not reboot/flaky' everywhere in jobs.py by adding an implicit skip marker if the corresponding arguments are not specified in the pytest call. Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
The "slow" marker designates tests that may take a long time and are designed to catch rare issues that may need large datasets or multiple repetitions to be reproduced. Add the following tests to reproduce the following rare issues: * test_migrate_repeat: Windows migration issues associated with the XenVbd driver * test_create_and_destroy_sr_repeat: NFS deadlock issue Add a new `multi/slow` VM list for this purpose. Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
dinhngtu
force-pushed
the
dnt/enable-marker
branch
from
8557c2f to
77aeed5
Compare
| "paths": ["tests/limits"], | ||
| } | ||
| }, | ||
| "slow": { |
There was a problem hiding this comment.
We can start with that, but I think we'll quickly see the need for a way to be more specific about which slow tests to run, depending on their requirements.
There was a problem hiding this comment.
Indeed I ran into this when trying to add tests/guest_tools/win into the slow suite.
stormi
left a comment
stormi
left a comment
There was a problem hiding this comment.
I have doubts about some of the choices, but nothing that we can't improve later.
glehmann
left a comment
glehmann
left a comment
There was a problem hiding this comment.
Please make sure to update vm_data.py in the jenkins-ci/ci-configuration repository before merging this PR
There was a problem hiding this comment.
On second thought I don't like the implementation. It will cause annoyances when once wants to run a specific test from a failed job, or execute tests manually, only to discover that pytest skipped parts of the tests when it's too late already.
To me, the implementation should be limited to jobs.py: automatically adding "not XXX" to the -m switch, in order to exclude some markers by default, unless they're specified in the job definition or manually via -m on the command line.
|
Using a for loop made per-test setup/teardown more complicated compared to using a fixture, so I'm having second thoughts... |
3 participants
Some markers (e.g. reboot/flaky) represent tests that shouldn't be run in the default suite. As a result, we have to negate them over and over in jobs.py. Add a mechanism to skip them by default. These tests can be manually enabled using a pytest argument.
Add a "slow" marker to represent slow tests for detecting rare issues.
Add a couple initial slow tests to catch previously-encountered issues.
Add some typing to conftest.py while we're at it.