Generate the Python stubs automatically

[dgelessus]

Closes #1660. Adds a script to automatically generate the Python "stub" files under Scripts/Python/plasma. The script must be run from the in-game Python console:

>>> __import__("generate_stubs").run()

This will generate stubs for all Plasma built-in modules into a subfolder "plasma_stubs_generated" under the game folder.

Everything happens fully automatically using runtime introspection. Only one part is "semi-manual": because Python cannot introspect the signatures of C-defined functions, all function parameters must be specified manually in the C++-defined function docstrings, on a line starting with Params: . This convention was apparently used by Cyan originally, so most function docstrings already follow this format.

The generator itself is almost completely finished, but this PR still needs a bit of work:

• The generator currently removes all type annotations.
  • This will be mostly fixed by copying the parameter types from the existing stubs into the Params: lines in the C++-defined docstrings.
  • There's no way to specify type annotations for function return types and attribute/property types. Most likely, I'll implement this using another docstring line prefix, such as Returns: .
• Some C++-defined docstrings are outdated, incorrect, slightly misformatted, or don't match the stubs, leading to spurious and incorrect changes.
• The existing stubs are sometimes not sorted alphabetically. I'll fix this in a separate PR (Sort Python stubs consistently #1662), so that the diff of this PR will become more readable/meaningful.

With those things in mind, early feedback is welcome. I'm particularly interested in the opinion of @Hoikas, who has recently hand-written some stubs that will be overwritten by this script.

[dpogue]

The generated signature for PtLocalizedYesNoDialog is resulting in Python syntax errors (it has a / as an argument, which I think is supposed to have special meaning and not show up as an argument itself?)

[dgelessus]

I think / in parameter lists is allowed in real Python code since... some recent-ish version of Python 3. The actual syntax error is that the / appears after a *args (/ must come before *). This should get fixed along the way once I get type annotations working, because the manually type-annotated version of PtLocalizedYesNoDialog in the stub doesn't have this syntax error.

[dgelessus]

Type annotations are now supported by the stub generator. I've introduced a new Type: docstring prefix, which works similarly to the existing Params: prefix, but allows specifying a full function signature including a return type annotation (example: Type: (x: int, y: int) -> int) or the type of a property getter (example: Type: Tuple[str, int]).

I've also brought all of the C++-defined docstrings up-to-date with the manual changes and type annotations in the stubs. This should resolve all of the spurious changes.

This PR is basically finished now. I'll un-draft it once #1662 is merged and I've rebased this PR onto that (so that this PR's diff is more readable for reviewing).

[dpogue]

Previously this was -1, but I'm guessing it's handled as a uint internally so it keeps the positive value?

[dpogue]

I guess a good question is whether a comparison against -1 would ever work in Python, maybe this MAX_UINT32 is the only correct option

[dgelessus]

Yeah, it's declared in C++ as kGameJoinPending = (uint32_t)-1, so it gets output as unsigned.

Also worth noting that I used a 64-bit client to generate the stubs. Plasma's custom EnumValues store the numeric value as Py_ssize_t, so it's possible that a 32-bit client would see this constant as -1 on the Python side. That seems like something we should fix...

[Hoikas]

I wonder if would be clearer to use 0xFFFFFFFF?

[dgelessus]

Hmm, how would we detect this in the generator though? Add a special case for 0xffffffff, or just use hex for all values larger than some number?

In any case, I've now regenerated the stubs using a 32-bit client, so we can ignore this until the underlying problem with kGameJoinPending is fixed.

[Hoikas]

I wonder if would be clearer to use 0xFFFFFFFF?

[Hoikas]

This could be replaced with textwrap.indent().

[dgelessus]

This is a good idea, but I would prefer to do this in a later PR, because it changes the formatting of generated docstrings that contain newlines. It turns out we have quite a few of those and it would clutter up the diff with even more whitespace changes.

[Hoikas]

Interesting. I thought based on the documentation that it would give the same result, especially if we provided a predicate, eg

add_indents = functools.partial(textwrap.indent, predicate=lambda line: bool(line))

(The call to bool() is for the sake of clarity and could be omitted.)

[Hoikas]

With that being said, with the need for functools.partial(), we might be reaching too far into the bag of tricks. I think I'd be okay with keeping add_indents() if we remove the string concat in favor of an f-string 😉

[dgelessus]

The "problem" is that textwrap.indent will indent every line, whereas my add_indents lets you insert newlines that will not cause an indent, e. g. add_indents(" ", ["indented\nnot indented", "indented again"]).

This behavior was by accident, but it turned out to be necessary for matching Cyan's original formatting, where multiline docstrings only have their first line indented and not the following lines.

[Hoikas]

Makes sense. Thanks for the clarification

[Hoikas]

(Can you tell that I'm not a fan of string concats?)

[Hoikas]

This will make maintaining the stubs significantly easier. Thanks!