Writes "ok" instead of "not ok" for an expected failure

[lazka]

import unittest

class Test(unittest.TestCase):

    @unittest.expectedFailure
    def test_should_fail(self):
        assert 0

> python3 -m pytest test.py --tap-stream
1..1
ok 1 test.py::Test.test_should_fail # TODO expected failure: reason:

I would have expected it to write out "not ok", since the the TAP spec says for TODO: "They are not expected to succeed".

The problem is that I have a TAP consumer which complains that this test succeeded unexpectedly.

Am I missing something?

[mblayman]

No, you've got a good eye and are not missing anything. I had to make a judgement call between two competing behaviors.

In the unittest docs, expectedFailure says:

Mark the test as an expected failure. If the test fails it will be considered a success. If the test passes, it will be considered a failure.

https://docs.python.org/3/library/unittest.html#unittest.expectedFailure

Continuing the TAP spec, it says:

They are not expected to succeed. Should a todo test point begin succeeding, the harness should report it as a bonus. This indicates that whatever you were supposed to do has been done and you should promote this to a normal test point.

Since Python wants expected failures to count as success (when the assertion fails) and the TAP spec says that unexpected passes are a bonus, I chose to represent expected failures in line with the Python definition.

I hope that clears things up.

[mblayman]

Hmmm... maybe there's something else going on. I dug into the code a bit deeper and it appears that the code that I should be running actually matched what you stated. https://github.com/python-tap/tappy/blob/master/tap/runner.py#L59

But I guess that's not the code that's running. It's probably in this area of the plugin: https://github.com/python-tap/pytest-tap/blob/master/src/pytest_tap/plugin.py#L98

I've not looked at this code in a long time so I don't know if anything has changed about the pytest_runtest_logreport hook. Of course, it's also possible that I didn't write this correctly in the first place. 😁

[lazka]

I've looked a bit into this, but various things don't make sense to me (like the Non-strict xfails will include 'wasxfail' while strict xfails won't. is exactly the other way around here) so I've stopped for now.

The test I looked at:

import unittest
import pytest

class Test(unittest.TestCase):

    @unittest.expectedFailure
    def test_unittest_xfail_fail(self):
        assert 0

    @pytest.mark.xfail(strict=False, reason="foo")
    def test_xfail_nonstrict_fail(self):
        assert 0

    @pytest.mark.xfail(strict=True, reason="foo")
    def test_xfail_strict_fail(self):
        assert 0

    @unittest.expectedFailure
    def test_unittest_xfail_nofail(self):
        pass

    @pytest.mark.xfail(strict=False, reason="foo")
    def test_xfail_nonstrict_nofail(self):
        pass

    @pytest.mark.xfail(strict=True, reason="foo")
    def test_xfail_strict_nofail(self):
        pass