Concept: Rust database access via Python database connection pool (via runInteraction) (LLM splat v1)

[MadLittleMods]

Note: This iteration isn't meant to be merged at all and is just meant to see if it's possible. Just splatting this here for the reference. It does seem to work (see testing strategy) although I haven't inspected the code. I just asked an LLM to take my previous non-working structure and make it work.

---

Rust database access via Python database connection pool.

This is a stepping stone before we can go full Rust everywhere. We're providing a generic interface as we want database access to work in Synapse and synapse-rust-apps. In synapse-rust-apps, we will use a tokio-postgres based database connection pool so it's full Rust.

We want to avoid the situation where we have two database connection pools (one for Python, one for Rust) as we've run into connection exhaustion problems on Matrix.org before.

Why runInteraction(...)?

This is a variation of #19824 instead using the runInteraction(...) pattern we already have in Synapse. This means everything that uses the transaction happens in the function callback which enables retries for serialization/deadlock errors when using repeatable-read isolation.

Using the same runInteraction pattern also means we can port over existing Synapse code/endpoints without much thought.

This strategy is less of an impedance mismatch (aligns more closely) with Synapse so the glue code should also be simpler.

Testing strategy

Added some tests that exercise some async Rust handlers for the /versions endpoint:

SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_versions.VersionsTestCase

Real-world:

1. poetry run synapse_homeserver --config-path homeserver.yaml
2. GET http://localhost:8008/_matrix/client/versions

Logging

We already support normal logging from Rust -> Python with pyo3-log and log:

For example, if you add this inside async fn build_versions_response(...) { ... }

log::info!(
    "asdf msc3881_enabled={:?} msc3575_enabled={:?}",
    msc3881_enabled,
    msc3575_enabled
);

It's logged out on the Python side although it's logged against the sentinel LoggingContext because we're crossing a thread boundary and losing the association to the LoggingContext on the main thread:

2026-06-22 18:00:57-0500 [-] synapse.handlers.versions - 124 - INFO - sentinel - asdf msc3881_enabled=false msc3575_enabled=true

But if you use create_deferred(...) to start running some async Rust function, we will capture the calling logcontext as a Tokio task local which can be referenced by any code running on that other thread.

For example, when using run_python_awaitable(...) we activate the logcontext stored in the Tokio task local. This is why we get database metrics tracked when using run_interaction(...).

Similarly, we can add some sort of helper to wrap any place we want to log that will activate the correct logcontext for that block in Rust. I think this is best left as a follow-up where we can specifically focus on the approach.

For the Processed request log lines:

1. Does it count CPU time from things run in Rust?
   • No
2. Does it count database time from things run in Rust?
   • Yes! Because run_interaction(...) activates the correct logcontext and calls runInteraction(...) on the Python side, everything will be counted like normal

2026-06-23 21:20:31-0500 [-] synapse.access.http.fake - 643 - INFO - GET-6 - 127.0.0.1 - test - {@user1:test} Processed request: 0.619sec/0.000sec ru=(0.000sec, 0.000sec) db=(0.000sec/0.601sec/3) 383B 200 "GET /_matrix/client/versions 1.1" "-" [0 dbevts]

Dev notes

LLM Prompt

The goal is to achieve Rust database access via the Python database connection pool.

We're providing a generic interface as we want database access to work in Synapse and a separate project synapse-rust-apps. In synapse-rust-apps, we will use a tokio-postgres based database connection pool so it's full Rust. This is what's being prototyped in rust/src/storage/db/rust_db_pool.rs vs the actual implementation we'll keep here in rust/src/storage/db/python_db_pool.rs.

We want to stick with the runInteraction(...) pattern from Synapse as it allows us to retry transactions in a world where we're using repeatable-read isolation level (serialization and deadlock errors).

I'm in the middle of getting it working and most of it is just non-working structure. Please finish the rest.

As an example, we're doing some Rust database access in rust/src/handlers/versions.rs -> rust/src/storage/store.rs

---

Transaction retries were originally introduced into Synapse for deadlocks which makes me question why they're even happening in the first place. How are other people dealing with this kind of problem? (poorly, not a problem, retries like us, etc)

@reivilibre points out that we're more likely to hit retries because of serialization errors which aligns with the Postgres docs (we use the repeatable read isolation level in Synapse):

When an application receives this error message, it should abort the current transaction and retry the whole transaction from the beginning. The second time through, the transaction will see the previously-committed change as part of its initial view of the database, so there is no logical conflict in using the new version of the row as the starting point for the new transaction's update.

Note that only updating transactions might need to be retried; read-only transactions will never have serialization conflicts.

-- https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ

---

Usage:

db_pool
    .run_interaction(|txn| {
        async move {
            /* do stuff with txn */
        }
        .boxed()
    })

Or equivalent (if preferred):

db_pool
    .run_interaction(|txn| {
        Box::pin(async move {
            /* do stuff with txn */
        })
    })

---

AsyncFn (Async closure) with Send trait doesn't seem very viable:

• https://stackoverflow.com/questions/79884006/how-do-i-require-an-async-closure-types-future-to-be-send
• RFC 3668: Async closures, Async closures rust-lang/rfcs#3668
• RFC 3654: Return Type Notation, RFC: Return Type Notation rust-lang/rfcs#3654
• Feature Request: Support for async closures SeaQL/sea-orm#2749
  • Pointed to https://blog.weiznich.de/blog/diesel-async-0-9/

---

Other PR where I dealt with deferred (make_deferred_yieldable) and async in Rust: #18903

Fix MemoryReactor.callWhenRunning not invoking callbacks if already started, twisted/twisted#12514 - this is relevant because we need to start the tokio thread pool for these tests.

---

PyO3 Rust -> Python logging: https://pyo3.rs/main/ecosystem/logging

Todo

• See how LoggingContext (log context) interact
  • Logging
  • Tracing
• See how metrics work

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

From my high-level reading of things, the type erasure stuff here is because we want DatabasePool to by dyn-compatible because we currently store it like this:

pub struct Store {
    pub config: SynapseConfig,
    pub db_pool: Box<dyn DatabasePool>,
}

If we drop that constraint and instead store the DatabasePool like the following, all of erasure complexity can fall away.

pub struct Store<P: DatabasePool> {
    pub config: SynapseConfig,
    pub db_pool: P,
}

I think the answer depends on whether we care about the Store being Synapse specific or if that also should be shared with synapse-rust-apps.

I think the synapse-rust-apps might have its own Store implementation given we already store things that are Synapse specific in it like SynapseConfig. Although that could be refactored to be different 🤔

[MadLittleMods]

Added d9a111b to see what this looks like

[erikjohnston]

You could probably relax this to a FnMut?

[MadLittleMods]

Perhaps it's better if we don't allow FnMut as the func can be retried multiple times or even run and then fail to commit leaving the outside mutated state.

For example with the following example where we do this kind of FnMut thing in Python, this seems flawed when the transaction is retried. When retried, it will use min_stream_id mutated from the previous attempt.

synapse/synapse/storage/databases/main/devices.py

Lines 2557 to 2596 in 4e9f775

	# The minimum stream ID to delete in the next batch, c.f. comment above.
	# We default to 0 here as that is less than all possible stream IDs.
	min_stream_id = 0
	def prune_device_lists_changes_in_room_txn(txn: LoggingTransaction) -> int:
	nonlocal min_stream_id
	delete_sql = """
	DELETE FROM device_lists_changes_in_room
	WHERE stream_id IN (
	SELECT stream_id FROM device_lists_changes_in_room
	WHERE ? < stream_id AND stream_id <= ?
	ORDER BY stream_id ASC
	LIMIT ?
	)
	RETURNING stream_id
	"""
	txn.execute(
	delete_sql,
	(min_stream_id, prune_before_stream_id, PRUNE_DEVICE_LISTS_BATCH_SIZE),
	)
	# We can't use rowcount as that is incorrect on SQLite when using
	# RETURNING.
	num_deleted = 0
	for row in txn:
	num_deleted += 1
	min_stream_id = max(min_stream_id, row[0])
	if num_deleted:
	# Update the max pruned stream ID tracking table so that the
	# safety check knows data up to this point has been deleted.
	self.db_pool.simple_update_one_txn(
	txn,
	table="device_lists_changes_in_room_max_pruned_stream_id",
	keyvalues={},
	updatevalues={"stream_id": min_stream_id},
	)
	return num_deleted

And as a further example that doesn't exist here but if we tried to use min_stream_id outside of that function and ignored the runInteraction(...) error, we could continue with some flawed data.

[MadLittleMods]

As an update, twisted/twisted#12514 is finally part of a Twisted release 26.4.0 (2026-05-11). If we updated our Twisted version, we could probably get rid of all of this ugliness.

But that may be a few years away given our deprecation policy considers a "no-brainer" upgrade once it's available in both the latest Debian Stable (currently Twisted 24.11.0) and Ubuntu LTS repositories (currently Twisted 25.5.0)

[MadLittleMods]

This is just here to prove it's possible (no scrutiny given to this)

We can care about the implementation in synapse-rust-apps when we actually care about using it

[MadLittleMods]

Perhaps this will get fixed by #19876 (comment)

[MadLittleMods]

Made these a set to better represent what it is and more efficient lookups. Touched this because I had to model it in SynapseHomeServerConfig on the Rust side

[MadLittleMods]

Extracted things into a reviewable PR in #19878