Skip to content

Commit 6815d95

Browse files
vlaskyWilbert Harriman
andcommitted
Add optimize command for space reclamation after deletions
Implements a custom 'optimize' command (similar to SQLite FTS5) that allows reclaiming disk space after DELETE operations: INSERT INTO vec_table(vec_table) VALUES ('optimize'); VACUUM; How it works: - Identifies fragmented chunks from deletions - Migrates all vectors to new, contiguous chunks - Preserves partition keys and metadata during migration - Deletes old fragmented chunks - Allows VACUUM to reclaim freed disk space Implementation details: - Adds hidden 'table_name' column to trigger special insert commands - vec0Update_SpecialInsert_Optimize(): Main optimization logic - Iterates all rows and copies to new chunks - Copies metadata values to new chunk positions - Cleans up old chunks and vector data - vec0Update_SpecialInsert_OptimizeCopyMetadata(): Handles metadata migration Schema improvements: - Change PRIMARY KEY → INTEGER PRIMARY KEY in shadow tables - Makes rowid an alias instead of separate index - Reduces storage overhead and improves performance Use cases: - After bulk deletions to reclaim disk space - Periodic maintenance to defragment vector storage - Before backups to minimize database file size Caveats: - Can be slow on large tables (rebuilds all chunks) - Should be run during maintenance windows - Not transaction-safe for concurrent reads - Requires VACUUM afterward to actually free space Merged from upstream PR asg017#210 by wilbertharriman. Fixes issue asg017#185. Co-Authored-By: Wilbert Harriman <[email protected]>
1 parent f10e191 commit 6815d95

File tree

7 files changed

+437
-49
lines changed

7 files changed

+437
-49
lines changed

0 commit comments

Comments
 (0)