WIP: Expose QkObs borrowing from Python-space SparseObservable

[jakelishman]

Borrowing QkObs from the Python pointer is particularly tricky because the Python-space object has internal shared ownership and runtime read/write locking. At present the locks should never fail to acquire (as long as we aren't accidentally cloning a PySparseObservable anywhere) because there's no situation where Python space can hold multiple views onto the same backing observable, but we do expect that to become possible in the future. More importantly, we expect that to happen with PyCircuitData and PyDAGCircuit in order to expose control-flow blocks to Python space without having to copy them out all the time; control-flow blocks will all be shared ownership with their containing circuit and the Python-space views.

Here, we introduce several locking primitives for use with QkObs, but intended to be re-used when PyCircuitData and PyDAGCircuit similarly involve internal shared ownership. For consistency with DAGCiruit and Target, I also included the simple-case qk_obs_borrow_from_python in cases where the caller can know that the observable is unique (or is prepared to accept a Python exception in those situations).

Summary

Details and comments

Built on #15337

WIP because I haven't written any tests yet, or otherwise verified in any way that this actually works!

I also am not at all convinced about the try_lock APIs I've written - I want to come back to those and see if there's a better way to indicate the locking failure without:

• using ExitCode
• requiring C to initialise a non-null pointer QkObsGuard *lock; before calling one of the functions
• forcing the caller to call PyErr_Occurred()

If I have to go for any of them, I'm likely to go for the last one (instead of the first one, which this commit currently has).

[eliarbel]

I tested it with a toy Python c-extension (managed to avoid writing something like print!("{}", "%d", num_qubits, end="\n") 😉). I mostly tested the C API in a sequential code setting, in which it seems to be working as expected. The tests validated things like borrowing (and mutating) from Python, creating and returning to Python, locking for read/write, blocking on a locked object, failing on try-lock on a locked object etc. I plan to test this a bit further with Python multithreading, but I feel confident already that for the basic use-case the implementation is good.

Some general comments:

1. We definitely need a way to test this, or at least the locking aspect in a more general way (that is, not necessarily thorough a particular API like QkObs's). Maybe we should have something like a dummy C Extension module which which runs outside of the regular unit testing, like in the vein of the QPY compatibility testing?
2. Regarding the try_lock APIs, how about extending the regular lock functions with a parameter, e.g. something like:

    pub unsafe extern "C" fn qk_obs_lock_read_from_python(
        ob: *mut ::pyo3::ffi::PyObject,
        lock: *mut *mut ObsGuard,
        blocking: bool,
    ) -> *const SparseObservable {

and return NULL if locking fails?

3. We're missing code examples for the new Python-binding C API functions. I realize it's tricky because there are several approaches for building C extensions for Python, so I don't have a good answer.

In addition I've also left a few concrete inline comments and questions.

[eliarbel]

To be in line with the enum and CGuard, maybe call this CReadGuard (and the write one CWriteGuard)?

[jakelishman]

This shouldn't actually ever be accessible or even visible in the C header file (unless I've made a mistake) because it's a private-only implementation detail of the Rust backing. Fwiw, we use the C prefix on C-exposed things as a sort of dummy for cbindgen because it's purely syntactical and can't resolve namespaces, so we give C-exposed symbols uglier names, since we're going to rename them to QkName stuff anyway.

[eliarbel]

Suggested change

	// Python object and `address` points to anough space to write a pointer.
	// Python object and `address` points to enough space to write a pointer.

[eliarbel]

Why is this not part of the py module?

[jakelishman]

I think I left it out here because it doesn't actually depend on access to the Python symbols or libraries in any way, so it didn't need protecting in the header file. It might not be useful without Python, but I don't think there's a harm in failing to declare it.

[eliarbel]

Suggested change

	// SAFETY: per doecumentation, `lock` points to valid owned data returned by a locking function.
	// SAFETY: per documentation, `lock` points to valid owned data returned by a locking function.

[eliarbel]

I'm wondering if we need this and qk_obs_convert_from_python, given the qk_obs_*lock* functions.

If there is a use-case for having both variants, we should be clearer about semantics of using this along with the *lock* functions and vice versa. For example, using this function on a locked object makes it behave like a try-lock function which returns NULL on failure.

[jakelishman]

You can certainly get the same functionality as this by using the locking functions, but I included these as a convenience for C APIs that do not expect to be called in threaded or aliasing contexts at all - it's still safe, it just raises a Python exception in situations that actually can be handled if you use the more complex settings.

The difference here roughly is that it's a non-blocking function, but rather than returning a "wait" status, it sets the Python exception state.

[jakelishman]

Following discussion with Eli, I've opened #15821 to split off the simpler components of this into a separate PR that will be easier to merge for 2.4, and deferred this one to 2.5.