Support for free-threaded build of CPython #504
Draft
+209
−15
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a draft PR. I'll need to make some additional updates before it's ready for review but I thought it's useful to share already.
Summary of changes:
Add mechanism to set the Py_MOD_GIL_NOT_USED. I have a patch that does something similar to what pybind11 does: add a variadic macro with an optional flag so extensions can opt-in to free-threaded support
Replace borrow based APIs with strong reference ones (e.g. PyDict_GetItemRef). That's not included yet and for now Py_MOD_GIL_NOT_USED is always set (makes testing a bit easier).
Replace borrow based APIs with strong reference ones (e.g. PyDict_GetItemRef)
Use a mutex to protect global state (e.g. the registry). To avoid deadlocks and keep the code simple, there is a single global recursive lock. We can refine this later (split it into the multiple locks, use critical sections, etc).
Fix references to ob_refcnt, don't use Py_REFCNT() as an l-value.
Ugly work-around for crash in
~object_base(). Needs more investigation for a better fix.Building this requires a fix to faber. I found building Boost.Python very tricky (likely because I'm not familiar with b2 and faber). I'll share my build scripts in case they help someone else (need to tidy them first).
There is another branch that includes doctests converted to use pytest. I only did a subset of the tests (using Claude to assist in conversion). The goal is to use pytest-run-parallel and try to trigger data races or other issues.