Implement true parallelization in `qpy.load` and `qpy.dump`

[ihincks]

What should we add?

qpy.dump allows one to serialize multiple QuantumCircuit objects into one binary, and qpy.load can likewise instantiate multiple QuantumCircuit instances. Services that use QPY as the circuit transport format often need to load multiple circuits, and the de/serialization process can add non-negligible overhead to the round-trip-time of the service when there are many large circuits. If qpy.dump implemented rust-space parallelization of the de/serialization process, this would have the potential of reducing these round-trip times.

I would myself place more priority on the load side because that's the side that lives inside the service, rather than in client land.

[jakelishman]

This is certainly something we're interested in in the intermediate term. It's not immediately useful for your use case even if we flicked a flag and turned it on today, because we can only load QPY 17 through the (mostly) pure-Rust path, whereas qiskit-ibm-runtime is configured to submit jobs with QPY 13 by default.

That said, #15663 should add the missing compatibility, and then the story gets a lot better. We'll still likely have some small GIL contention, even for IBM ISA circuits, at the point of constructing the Python-space objects, but that's solvable.

[ihincks]

I wrote the issue generically, but of course you're right that I have the IBM service in mind, in particular. The default QPY version for executor is a bit different than sampler/estimator, and what I'd be targeting.

If you see the 0.2 executor schema here and how it gets used here you can see that it's trying to use as new a version as possible because the schema now defines valid ranges the service is willing to accept

The idea is that we'd introduce a 0.X executor schema that puts all of the circuits into one field to leverage a parallel dump/load. This could be done ahead of time, too, though we'd need to iterate to 0.(X+1) if the parallelization for some reason required a QPY version bump.

Edit: Actually, it would probably need to be a 1.X schema version, but same deal. The service accepts a range of schema versions, and dropping an old schema version must follow cloud deprecation guidelines, so we'd support 0.1, 0.2, 1.0 all simultaneously, for example.

[jakelishman]

Ah good to know thanks. I think in practice we'll be able to get Rust QPY loading for all meaningful versions and so enable the parallelism for everything by Qiskit 2.5 anyway, so the timelines are probably not altered.

[gadial]

I'll make this one top priority after we're finished with QPY13 compatibility for Rust and switching to full Rust flow (#15662) which both seem like prerequisites for this. If you know of a standard way to test/benchmark parallel Rust code I'll be happy to hear.

[ihincks]

If you know of a standard way to test/benchmark parallel Rust code I'll be happy to hear.

As part of qiskit testing benchmarks, or just as a one-off on your computer locally?

[gadial]

Both, but for now locally is more important since I want to make sure it works before I start pushing it.