mypy behavior differs when run on a file directly vs a test

[sterliakov]

Bug Report

Likely due to some test stubs inconsistency, mypy behaviour in test deviates from results of running full mypy on a file.

This is a follow-up for #18675.

To Reproduce

Add the following test to check-enum.test:

[case testDemo]
# flags: --python-version 3.13 --warn-unreachable
from enum import Enum, member, nonmember
from typing import Literal, Never

def assert_never(_: Never) -> Never: ...

class E2(Enum):
    @member
    def C() -> None: ...  # E: Method must have at least one argument. Did you forget the "self" argument?  [misc]

c: Literal[E2.C]  # E: Parameter 1 of Literal[...] is invalid  [valid-type]

def check_2(e: E2) -> None:
    match e:
        case E2.C:
            pass
        case other:
            assert_never(other)  # E: Argument 1 to "assert_never" has incompatible type "E2"; expected "Never"  [arg-type]

    if e is E2.C:
        pass
    else:
        assert_never(e)  # E: Argument 1 to "assert_never" has incompatible type "<subclass of "enum.member[Callable[[], None]]" and "__main__.E2">"; expected "Never"  [arg-type]
[builtins fixtures/enum.pyi]

Now copy the test code to a separate file and run mypy on it.

Expected Behavior

Test and mypy outputs are identical.

Actual Behavior

$ pytest mypy/test/testcheck.py::TypeCheckSuite::check-enum.test -k testDemo 
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.12.6, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/stas/Documents/Work/mypy
configfile: pyproject.toml
plugins: cov-5.0.0, xdist-3.6.1
2 workers [1 item]      
F                                                                                                                                                                                                                                    [100%]
================================================================================================================= FAILURES =================================================================================================================
_________________________________________________________________________________________________________________ testDemo _________________________________________________________________________________________________________________
[gw0] linux -- Python 3.12.6 /home/stas/Documents/Work/mypy/.venv12/bin/python3
data: /home/stas/Documents/Work/mypy/test-data/unit/check-enum.test:2516:
Failed: Unexpected type checker output (/home/stas/Documents/Work/mypy/test-data/unit/check-enum.test, line 2516)
----------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------
Expected:
  main:9: error: Method must have at least one argument. Did you forget the "self" argument?  [misc] (diff)
  main:11: error: Parameter 1 of Literal[...] is invalid  [valid-type] (diff)
  main:18: error: Argument 1 to "assert_never" has incompatible type "E2"; expected "Never"  [arg-type] (diff)
  main:23: error: Argument 1 to "assert_never" has incompatible type "<subclass of "enum.member[Callable[[], None]]" and "__main__.E2">"; expected "Never"  [arg-type] (diff)
Actual:
  main:9: error: Method must have at least one argument. Did you forget the "self" argument? (diff)
  main:11: error: Parameter 1 of Literal[...] is invalid (diff)
  main:18: error: Argument 1 to "assert_never" has incompatible type "Union[member[Callable[[], None]], E2]"; expected "Never" (diff)

Alignment of first line difference:
  E: ...the "self" argument?  [misc]
  A: ...the "self" argument?
                            ^
Update the test output using --update-data (implies -n0; you can additionally use the -k selector to update only specific tests)
========================================================================================================= short test summary info ==========================================================================================================
FAILED mypy/test/testcheck.py::TypeCheckSuite::check-enum.test::testDemo
============================================================================================================ 1 failed in 0.64s =============================================================================================================

$ python -m mypy --config-file= --warn-unreachable  --python-version 3.13 /tmp/a.py

/tmp/a.py:5: error: Implicit return in function which does not return  [empty-body]
/tmp/a.py:9: error: Method must have at least one argument. Did you forget the "self" argument?  [misc]
/tmp/a.py:11: error: Parameter 1 of Literal[...] is invalid  [valid-type]
/tmp/a.py:18: error: Argument 1 to "assert_never" has incompatible type "E2"; expected "Never"  [arg-type]
/tmp/a.py:23: error: Argument 1 to "assert_never" has incompatible type "<subclass of "member" and "E2">"; expected "Never"  [arg-type]
Found 5 errors in 1 file (checked 1 source file)

Your Environment

• Mypy version used: Always use .enum_members to find enum members #18675 head
• Mypy command-line flags: --warn-unreachable --config-file= --python-version=3.13
• Mypy configuration options from mypy.ini (and other config files): N/A
• Python version used: 3.12