Avoid sqlite3 lock contention by serializing access through the cache thread.#927
Merged
Conversation
Enable WAL journal mode and a 30s busy timeout on repodata_shards.db so the cache reader thread no longer races with the network writer thread. Falls back gracefully on filesystems where WAL is unsupported.
Co-authored-by: Daniel Holth <dholth@anaconda.com>
3 tasks
conda-bot
added
the
cla-signed
Member
jezdez
reviewed
Apr 30, 2026
Member
jezdez
left a comment
jezdez
left a comment
There was a problem hiding this comment.
Nice idea to serialize all SQLite access through the cache thread. One suggestion to avoid the isinstance dance: use a separate insert queue instead of mixing types on the lookup queue.
dholth
force-pushed
the
cache-insert-queue
branch
from
fbc8fee to
207d7d6
Compare
dholth
changed the title
Contributor
Author
|
Will continue this in the conda repository. |
Contributor
Author
|
@jezdez if we really get frustrated, we could fall back to no-cache. Or the memory:// sqlite database! Are we closing the transaction correctly in our current write-to-cache? |
danyeaw
approved these changes
May 12, 2026
Co-authored-by: Dan Yeaw <dan@yeaw.me>
danyeaw
approved these changes
May 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Instead of giving the network thread a sqlite3 connection, add "store record" requests to the cache thread's input queue. The cache thread processes either "fetch record" or "store record" requests as they come in, and there is no database contention within a single conda process.
Checklist - did you ...
newsdirectory (using the template) for the next release's release notes?