Rust database access via Python database connection pool#19824
Draft
MadLittleMods wants to merge 25 commits into
Draft
Rust database access via Python database connection pool#19824MadLittleMods wants to merge 25 commits into
MadLittleMods wants to merge 25 commits into
Conversation
From `rei/exp_db_rust_via_shim`, e19dfa1
```
error[E0277]: the size for values of type `(dyn storage::db::DatabasePool + 'static)` cannot be known at compilation time
--> rust/src/handlers/mod.rs:53:30
|
53 | let store = Arc::new(Store { config, db_pool });
| ^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: within `storage::store::Store`, the trait `std::marker::Sized` is not implemented for `(dyn storage::db::DatabasePool + 'static)`
note: required because it appears within the type `storage::store::Store`
--> rust/src/storage/store.rs:37:12
|
37 | pub struct Store {
| ^^^^^
= note: structs must have a statically known size to be initialized
```
…t fine
```
error[E0277]: `(dyn storage::db::DatabasePool + 'static)` cannot be shared between threads safely
--> rust/src/handlers/mod.rs:32:8
|
32 | struct RustHandlers {
| ^^^^^^^^^^^^ `(dyn storage::db::DatabasePool + 'static)` cannot be shared between threads safely
|
= help: the trait `std::marker::Sync` is not implemented for `(dyn storage::db::DatabasePool + 'static)`
= note: required for `std::ptr::Unique<(dyn storage::db::DatabasePool + 'static)>` to implement `std::marker::Sync`
note: required because it appears within the type `std::boxed::Box<(dyn storage::db::DatabasePool + 'static)>`
--> /home/eric/.rustup/toolchains/1.91.1-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:231:12
|
231 | pub struct Box<
| ^^^
note: required because it appears within the type `storage::store::Store`
--> rust/src/storage/store.rs:37:12
|
37 | pub struct Store {
| ^^^^^
= note: required for `std::sync::Arc<storage::store::Store>` to implement `std::marker::Send`
note: required because it appears within the type `handlers::versions::VersionsHandler`
--> rust/src/handlers/versions.rs:30:12
|
30 | pub struct VersionsHandler {
| ^^^^^^^^^^^^^^^
note: required because it appears within the type `handlers::RustHandlers`
--> rust/src/handlers/mod.rs:32:8
|
32 | struct RustHandlers {
| ^^^^^^^^^^^^
note: required by a bound in `pyo3::impl_::pyclass::assertions::assert_pyclass_send_sync`
--> /home/eric/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.28.3/src/impl_/pyclass/assertions.rs:6:8
|
4 | pub const fn assert_pyclass_send_sync<T>()
| ------------------------ required by a bound in this function
5 | where
6 | T: Send + Sync,
| ^^^^ required by this bound in `assert_pyclass_send_sync`
```
We can use `FromPyObjectOwned` (instead of `FromPyObject`)
because we don't borrow anything
```
error[E0107]: trait takes 2 lifetime arguments but 1 lifetime argument was supplied
--> rust/src/storage/db/python_db_pool.rs:220:24
|
220 | pub fn fetchall<T: FromPyObject<'py> + ValidDatabaseReturnType>(
| ^^^^^^^^^^^^ --- supplied 1 lifetime argument
| |
| expected 2 lifetime arguments
|
note: trait defined here, with 2 lifetime parameters: `'a`, `'py`
--> /home/eric/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.28.3/src/conversion.rs:401:11
|
401 | pub trait FromPyObject<'a, 'py>: Sized {
| ^^^^^^^^^^^^ -- ---
help: add missing lifetime argument
|
220 | pub fn fetchall<T: FromPyObject<'py, 'py> + ValidDatabaseReturnType>(
| +++++
```
…al `py`
```
error[E0277]: the trait bound `std::vec::Vec<T>: pyo3::FromPyObject<'_, '_>` is not satisfied
--> rust/src/storage/db/python_db_pool.rs:228:30
|
228 | Ok(fetch_fn.call0()?.extract()?)
| ^^^^^^^ the trait `pyo3::FromPyObject<'_, '_>` is not implemented for `std::vec::Vec<T>`
|
note: required by a bound in `pyo3::types::PyAnyMethods::extract`
--> /home/eric/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.28.3/src/types/any.rs:881:12
|
879 | fn extract<'a, T>(&'a self) -> Result<T, T::Error>
| ------- required by a bound in this associated function
880 | where
881 | T: FromPyObject<'a, 'py>;
| ^^^^^^^^^^^^^^^^^^^^^ required by this bound in `PyAnyMethods::extract`
help: consider introducing a `where` clause, but there might be an alternative better way to express this requirement
|
223 | ) -> anyhow::Result<Vec<T>> where std::vec::Vec<T>: pyo3::FromPyObject<'_, '_> {
| ++++++++++++++++++++++++++++++++++++++++++++++++++
```
```
error[E0308]: mismatched types
--> rust/src/storage/db/python_db_pool.rs:236:35
|
236 | self.execute(py, sql, args)?;
| ------- ^^^^ expected `&Bound<'_, PyAny>`, found `&[&str]`
| |
| arguments to this method are incorrect
|
= note: expected reference `&pyo3::Bound<'_, pyo3::PyAny>`
found reference `&'life2 [&'life3 str]`
note: method defined here
--> rust/src/storage/db/python_db_pool.rs:206:12
|
206 | pub fn execute<'py>(
| ^^^^^^^
...
210 | args: &Bound<'py, PyAny>,
| ------------------------
```
…-db-pool Conflicts: rust/src/lib.rs synapse/storage/util/id_generators.py
3 tasks
Contributor
Author
|
See v2 of this PR: #19878 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust database access via Python database connection pool. This is meant to be a
tokio-postgresfeeling interface.This is a stepping stone before we can go full Rust everywhere. We're providing a generic interface as we want database access to work in Synapse and
synapse-rust-apps. Insynapse-rust-apps, we will use atokio-postgresbased database connection pool so it's full Rust.We want to avoid the situation where we have two database connection pools (one for Python, one for Rust) as we've run into connection exhaustion problems on Matrix.org before.
(non-working)
Dev notes
Some exploratory work from @reivilibre in e19dfa1 (although this is framed from the perspective of passing the
txnfrom Python to Rust)Other recent Rust work for reference: #19701
PyO3 (Rust bindings for Python) docs: https://pyo3.rs/
Example of a API endpoint handler written in Rust:
rust/src/rendezvousPull Request Checklist
EventStoretoEventWorkerStore.".code blocks.