Add support for literal addition

[cdce8p]

Add support for addition of Literal values and str, int and bytes expressions. This is especially useful when working with TypedDicts. Previously this code would have emitted a literal-required error.

from typing import Literal, TypedDict

class LookupDict(TypedDict):
    top_var: str
    bottom_var: str
    var: str

def func(d: LookupDict, pos: Literal["top_", "bottom_", ""]) -> str:
    return d[pos + "var"]

Refs #11616

[JukkaL]

Can you look at the new errors reported by mypy_primer? If we infer literal types more eagerly, they could generate new false positives.

[cdce8p]

Can you look at the new errors reported by mypy_primer?

The primer results look quite promising IMO. For starlette the issue is something like this:

from typing import Literal

class SessionMiddleware:
    def __init__(
        self,
        same_site: Literal["lax", "strict", "none"] = "lax",
        https_only: bool = False,
    ) -> None:
        ...
        self.security_flags = "httponly; samesite=" + same_site
        if https_only:  # Secure flag can be used with HTTPS only
            self.security_flags += "; secure"  # <-- error

It can be resolved by annotation self.security_flags as str.

--
For AutoSplit the issue looks like this

class CaptureMethodBase:
    description = ""

class BitBltCaptureMethod(CaptureMethodBase):
    description = (
        "\nThe best option when compatible. But it cannot properly record "
        + "\nOpenGL, Hardware Accelerated or Exclusive Fullscreen windows. "
        + "\nThe smaller the selected region, the more efficient it is. "
    )

class ForceFullContentRenderingCaptureMethod(BitBltCaptureMethod):
    description = (
        "\nUses BitBlt behind the scene, but passes a special flag "
        + "\nto PrintWindow to force rendering the entire desktop. "
        + "\nAbout 10-15x slower than BitBlt based on original window size "
        + "\nand can mess up some applications' rendering pipelines. "
    )

Just removing the + symbols and using implicit string concatenation would resolve the issue here.

--

Not completely sure whats going on with trio why the type ignore there is no longer required. However, it's in a test file.
https://github.com/python-trio/trio/blob/3d592af32b4a6dae78525fb5103ac2e58020774a/src/trio/_tests/test_fakenet.py#L185-L189

[JukkaL]

Just removing the + symbols and using implicit string concatenation would resolve the issue here.

I think implicit and explicit string concatenation should behave the same. Otherwise it would seem inconsistent. I also know that some people prefer explicit string concatenation as a style thing, since it's easy to accidentally use implicit string concatenation (e.g. omit a comma at the end).

[JukkaL]

What about inferring a literal addition only if one of the operands is not a literal expression? We have tried to adhere to an informal policy where literal types are not inferred for variables "out of thin air", i.e. an explicit Literal[...], Final or similar needs to be somewhere.

[JukkaL]

In particular, this feels inconsistent to me:

class A:
    a = ""

class B(A):
    a = "a" + "b"

class C:
    a = "a" + "b"

reveal_type(A.a)  # Revealed type is "builtins.str"
reveal_type(B.a)  # Revealed type is "Literal['ab']"  (why literal here?)
reveal_type(C.a)  # Revealed type is "builtins.str"

[cdce8p]

What about inferring a literal addition only if one of the operands is not a literal expression? We have tried to adhere to an informal policy where literal types are not inferred for variables "out of thin air", i.e. an explicit Literal[...], Final or similar needs to be somewhere.

Makes sense. I missed the "a" + "b" case when I was refining the implementation. Should be fixed with the latest commits.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

starlette (https://github.com/encode/starlette)
+ starlette/middleware/sessions.py:34: error: Incompatible types in assignment (expression has type "Literal['httponly; samesite=lax; secure', 'httponly; samesite=none; secure', 'httponly; samesite=strict; secure']", variable has type "Literal['httponly; samesite=lax', 'httponly; samesite=none', 'httponly; samesite=strict']")  [assignment]
+ starlette/middleware/sessions.py:36: error: Incompatible types in assignment (expression has type "str", variable has type "Literal['httponly; samesite=lax', 'httponly; samesite=none', 'httponly; samesite=strict']")  [assignment]

[brianschubert]

Related issue: #11616

[JukkaL]

Did a quick pass, mainly to look at the big picture. I can also test this manually later.

[JukkaL]

Calling accept here can be inefficient, since the caller may also call accept on the expression at some point, causing duplicate work. This could cause exponential time complexity in nested expression trees (we already have examples of this and it can cause extremely slow type checking). Maybe the caller can pass the expression type, so that we only call accept once per expression? Maybe the new logic should be called from check_op, when we have the right operand type?

[JukkaL]

Can you avoid the type: ignore here? In the future we will probably disallow many ignores in code compiled with mypyc, since some type ignores can cause erratic behavior, such as C compilation errors.

[JukkaL]

Can you add the interesting bits of the inheritance example I gave in an earlier comment?

[JukkaL]

Check a.__add__(a). It would be nice if it was consistent with a + a, though it's probably not important.

[JukkaL]

Test union with literal type and non-literal type -- e.g. user-defined class that defines __add__ that accepts str. Also test with union of literal type and Any.