[mypyc] feat: new primitive for int.to_bytes

[BobTheBuidler]

This PR adds a new primitive for all arg combinations of int.to_bytes

[ilevkivskyi]

Nice! I will keep this open for a day or two in case @JukkaL has some comments.

[ilevkivskyi]

On the second look I think I have some more questions.

[ilevkivskyi]

On the second thought, all these type checks look unnecessary, normally Python wrappers should do them. You can probably verify this by adding some run tests with Anys in them.

[BobTheBuidler]

what, like this?

def f(x: Any) -> bytes:
    return int.to_bytes(x)

[BobTheBuidler]

based on @JukkaL response to a similar question on #19673 I think we can safely remove this check since CPyTagged_StealAsObject guarantees the type

[JukkaL]

I think CPyTagged_StealAsObject is not correct there, since it will transfer the ownership of the parameter, and this can cause a double free. CPyTagged_AsObject will return a new reference which you can decref at the end of the function.

[ilevkivskyi]

@BobTheBuidler

what, like this?

First, not just the self, second I think you should try more something like this

def to_bytes(n: int, length: int, byteorder: str = "little", signed: bool = False) -> bytes:
    return n.to_bytes(length, byteorder, signed=signed)

x: Any = "no"
bad: Any = "way"
to_bytes(x, bad)

and check that a TypeError will be given even before getting to your specialized code.

[BobTheBuidler]

I could implement this test, but wouldn't we then just be testing the standard python-wrapper type validation functionality, as opposed to some specific functionality related to this PR?

I can still add the tests accordingly, I just want to make sure we have the same understanding of things before I proceed.

[ilevkivskyi]

Maybe also test calling to_bytes() function from interpreted code.

[BobTheBuidler]

@ilevkivskyi how would I implement that? Is there a good example I can look from?

[JukkaL]

I measured the performance impact using a micro-benchmark, and the performance was pretty similar to master on Python 3.13:

import time

def bench(n: int) -> None:
    for i in range(n):
        i.to_bytes(4, "little", signed=True)

bench(10 * 1000 * 1000)
t0 = time.time()
bench(30 * 1000 * 1000)
print(time.time() - t0)

It's possible that the operation is expensive enough that the call overhead when there's no primitive isn't significant. In any case, since maintaining a new primitive takes some effort, it's important that there's some measurable performance improvement first. Have you been able to measure a performance improvement?

[BobTheBuidler]

what if you assign the result to a value? That way we can include the interpreted version's unboxing penalty in our benchmark.

import time

def bench(n: int) -> None:
    for i in range(n):
        x = i.to_bytes(4, "little", signed=True)

bench(10 * 1000 * 1000)
t0 = time.time()
bench(30 * 1000 * 1000)
print(time.time() - t0)

[BobTheBuidler]

Separately, I might be able to slightly optimize the little/big check by only checking the length of the input string once, since we already know the length of 'little' and 'big'.

Or alternatively, I could make a bespoke version for each of the two cases and eliminate the check entirely. This would be fastest but I'm not sure if there's currently a clean way to implement the method_op

[JukkaL]

Having a specializer function that choosen a little/big endian variant of the primitive if the argument is a string literal could help (in mypyc.irbuild.specialize). We do a similar thing for encode and decode calls.

[BobTheBuidler]

That's odd. I fixed the kw-only arg in to_bytes stub, but doing so breaks the IR. Is there a way for me to cover this case with a method_op?

[BobTheBuidler]

1.18 vs this branch

There was a ~5-8% increase in speed for various permutations of the int_to_big_endian benchmark which includes not only a call to to_bytes but a decent bit of other stuff as well