Skip to content

Conversation

@Kerollmops
Copy link
Member

@Kerollmops Kerollmops commented Jan 23, 2025

This PR is an attempt at patching LMDB to be capable of initiating multiple RoTxn from an RwTxn and, therefore, be able to read the content of an uncommitted RwTxn in parallel. You can find the LMDB patch in the dedicated fork. We can run many algorithms in Meilisearch to process the uncommitted changes and generate new data structures before committing the transaction.

This PR is open, but we also have it on crates.io under the v0.22.1-nested-rtxns-6 version. This version is experimental and will not be maintained in this state until we merge the changes into upstream LMDB. For the time being, we will continue to maintain it in this manner.

// opening a write transaction
let mut wtxn = env.write_txn()?;

// [use the RwTxn to write into the env]

// opening multiple read-only transactions without committing beforehand
let rtxns = (0..1000).map(|_| env.nested_read_txn(&wtxn)).collect::<heed::Result<Vec<_>>>()?;

// [use the RoTxns and move them onto different threads]

@Kerollmops Kerollmops added the experimental Is unstable and must be tested more thoroughly label Mar 6, 2025
@Kerollmops Kerollmops force-pushed the allow-nested-rtxn-from-wtxn branch from 9a94c8a to 1d1926b Compare March 7, 2025 13:13
@Kerollmops Kerollmops force-pushed the allow-nested-rtxn-from-wtxn branch 6 times, most recently from a54defd to 05d9f22 Compare April 1, 2025 15:11
@Kerollmops Kerollmops mentioned this pull request Sep 30, 2025
3 tasks
@Kerollmops Kerollmops force-pushed the allow-nested-rtxn-from-wtxn branch 3 times, most recently from 484b166 to 95a3907 Compare October 1, 2025 11:56
@Kerollmops Kerollmops force-pushed the allow-nested-rtxn-from-wtxn branch 3 times, most recently from 9dfc1cd to 6eb0384 Compare October 13, 2025 17:17
@Kerollmops Kerollmops force-pushed the allow-nested-rtxn-from-wtxn branch 3 times, most recently from f70c41b to 4a88b58 Compare October 24, 2025 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

experimental Is unstable and must be tested more thoroughly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants