Fix VectorSet index keys not deleted after MIGRATE KEYS#1758
Draft
tiagonapoli wants to merge 2 commits intomainfrom
Draft
Fix VectorSet index keys not deleted after MIGRATE KEYS#1758tiagonapoli wants to merge 2 commits intomainfrom
tiagonapoli wants to merge 2 commits intomainfrom
Conversation
Documents a known limitation: when a VectorSet key is evicted below HeadAddress and then deleted via DEL, the InPlaceDeleter gate (the only path that redirects to TryDeleteVectorSet) never fires because the record is not in memory. Tsavorite writes a blind tombstone, leaving the DiskANN native index unreleased and the context permanently marked as in-use. The test creates a VectorSet, forces eviction via ShiftHeadAddress, then DELs the key and asserts that InUseContextCount remains unchanged (proving DropIndex was never called). Also adds VectorManager.InUseContextCount internal accessor for test introspection of the context metadata bitmap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After transmitting VectorSet indexes in the KEYS migration path, mark the VectorSet keys in the sketch as transmitted (Item2 = true) so that the existing DeleteKeys() flow deletes them during the DELETING phase. Previously, TransmitKeys() skipped VectorSet keys (they are in the ignore set for out-of-band transmission), leaving Item2 = false, which caused DeleteKeys() to skip them as well. This resulted in VectorSet keys remaining on the source node after migration. Also adds a DEBUG EXISTS subcommand for raw store existence checks bypassing cluster routing/EPSM, and uses it to verify key deletion in the VectorSetMigrateByKeys test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After transmitting VectorSet indexes in the KEYS migration path, mark the VectorSet keys in the sketch as transmitted (\Item2 = true) so that the existing \DeleteKeys()\ flow deletes them during the DELETING phase.
Previously, \TransmitKeys()\ skipped VectorSet keys (they are in the ignore set for out-of-band transmission), leaving \Item2 = false, which caused \DeleteKeys()\ to skip them as well. This resulted in VectorSet keys remaining on the source node after migration.
Changes