Conversation
nix-locate decompressed and scanned the whole database in one thread.
Because it was a single zstd frame, the work could not be split across
cores.
Store the database as independently-compressed zstd frames, cut only at
package boundaries so each is a self-contained frcode stream. Frame
offsets go in a trailing skippable frame that plain `zstd -d` ignores.
nix-locate then searches all frames in parallel, and nix-index compresses
them in parallel.
Version 1 databases are still read, and `nix-index --format-version 1`
still writes them for consumers that need the legacy format (e.g.
nix-index-database).
Benchmark (`nix-locate bin/ls`, full nixpkgs index, 16 cores, hyperfine):
before: 2.225 s ± 0.041 s
after: 0.690 s ± 0.027 s
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nix-locate decompressed and scanned the whole database in one thread. Because it was a single zstd frame, the work could not be split across cores.
Store the database as independently-compressed zstd frames, cut only at package boundaries so each is a self-contained frcode stream. Frame offsets go in a trailing skippable frame that plain
zstd -dignores. nix-locate then searches all frames in parallel, and nix-index compresses them in parallel.Version 1 databases are still read, and
nix-index --format-version 1still writes them for consumers that need the legacy format (e.g. nix-index-database).Benchmark (
nix-locate bin/ls, full nixpkgs index, 16 cores, hyperfine):