Add custom optimize command
#210
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vlasky
added a commit
to vlasky/sqlite-vec
that referenced
this pull request
Nov 28, 2025
Implements a custom 'optimize' command (similar to SQLite FTS5) that allows
reclaiming disk space after DELETE operations:
INSERT INTO vec_table(vec_table) VALUES ('optimize');
VACUUM;
How it works:
- Identifies fragmented chunks from deletions
- Migrates all vectors to new, contiguous chunks
- Preserves partition keys and metadata during migration
- Deletes old fragmented chunks
- Allows VACUUM to reclaim freed disk space
Implementation details:
- Adds hidden 'table_name' column to trigger special insert commands
- vec0Update_SpecialInsert_Optimize(): Main optimization logic
- Iterates all rows and copies to new chunks
- Copies metadata values to new chunk positions
- Cleans up old chunks and vector data
- vec0Update_SpecialInsert_OptimizeCopyMetadata(): Handles metadata migration
Schema improvements:
- Change PRIMARY KEY → INTEGER PRIMARY KEY in shadow tables
- Makes rowid an alias instead of separate index
- Reduces storage overhead and improves performance
Use cases:
- After bulk deletions to reclaim disk space
- Periodic maintenance to defragment vector storage
- Before backups to minimize database file size
Caveats:
- Can be slow on large tables (rebuilds all chunks)
- Should be run during maintenance windows
- Not transaction-safe for concurrent reads
- Requires VACUUM afterward to actually free space
Merged from upstream PR asg017#210 by wilbertharriman.
Fixes issue asg017#185.
Co-Authored-By: Wilbert Harriman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#185