Add flag to allow more flexible variable redefinition

[JukkaL]

Infer union types for simple variables from multiple assignments, if the variable isn't annotated. The feature is enabled via --allow-redefinition-new. --local-partial-types must also be enabled.

This is still experimental and has known issues, so it's not documented anywhere. It works well enough that it can be used for non-trivial experimentation, however.

Closes #6233. Closes #6232. Closes #18568. Fixes #18619.

In this example, the type of x is inferred as int | str when using the new behavior:

def f(i: int, s : str) -> int | str:
    if i > 5:
        x = i
    else:
        x = s  # No longer an error
    reveal_type(x)  # int | str
    return s

Here is a summary of how it works:

• Assignment widens the inferred type of a variable and always narrows (when there is no annotation).
• Simple variable lvalues are put into the binder on initial assignment when using the new feature. We need to be able to track whether a variable is defined or not to infer correct types (see Binder loses narrowed type of a variable if variable may be uninitialized #18619).
• Assignment of None values are no longer special, and we don't use partial None if the feature is enabled for simple variables.
• Lvalues other than simple variables (e.g. self.x) continue to work as in the past. Attribute types can't be widened, since they are externally visible and widening could cause confusion, but this is something we might relax in the future. Globals can be widened, however. This seems necessary for consistency.
• If a loop body widens a variable type, we have to analyze the body again. However, we only do one extra pass, since the inferred type could be expanded without bound (consider x = 0 outside loop and x = [x] within the loop body).
• We first infer the type of an rvalue without using the lvalue type as context, as otherwise the type context would often prevent redefinition. If the rvalue type isn't valid for inference (e.g. list item type can't be inferred), we fall back to the lvalue type context.

There are some other known bugs and limitations:

• Annotated variables can't be freely redefined (but they can still be narrowed, of course). I may want to relax this in the future, but I'm not sure yet.
• If there is a function definition between assignments to a variable, the inferred types may be incorrect.
• There are few tests for nonlocal and some other features. We don't have good test coverage for deferrals, mypy daemon, and disabling strict optional.
• Imported names can't be redefined in a consistent way. This needs further analysis.

In self check the feature generates 6 additional errors, which all seem correct -- we infer more precise types, which will generate additional errors due to invariant containers and fixing false negatives.

When type checking the largest internal codebase at Dropbox, this generated about 700 new errors, the vast majority of which seemed legitimate. Mostly they were due to inferring more precise types for variables that used to have Any types. I used a recent but not the latest version of the feature to type check the internal codebase.

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅