Add support for compressed vector search to IVF #184
Conversation
fix formatting
ibhati
changed the title
ibhati
requested review from
mihaic,
homksei,
yuejiaointel,
ethanglaser,
ahuber21,
dian-lun-lin and
Alexsandruss
as code owners
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive support for compressed vector search to the IVF (Inverted File Index) system. The main purpose is to enable IVF search with scalar quantized data (SQData) and establish a complete testing infrastructure for IVF functionality.
Key changes:
- Implemented compressed vector search support for IVF with scalar quantization
- Added comprehensive Python test suite for IVF functionality
- Enabled IVF tests in CI pipeline with matrix builds
Reviewed Changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/integration/ivf/scalar_search.cpp | New C++ integration test for IVF scalar quantization search |
| tests/integration/ivf/index_build.cpp | Temporarily disabled problematic thread tests |
| tests/CMakeLists.txt | Added scalar search test to build |
| include/svs/quantization/scalar/scalar.h | Added setter methods and type alias for SQDataset |
| include/svs/extensions/ivf/scalar.h | New extension implementing IVF support for scalar quantized data |
| data/test_dataset/reference/ivf_reference.toml | Updated reference timestamps |
| bindings/python/tests/test_vamana.py | Fixed semicolon style issues |
| bindings/python/tests/test_ivf.py | New comprehensive Python test suite for IVF |
| bindings/python/tests/common.py | Added IVF test configuration constants |
| bindings/python/src/ivf.cpp | Changed default hierarchical parameter to true |
| bindings/python/setup.py | Enabled IVF experimental features in build |
| bindings/python/include/svs/python/ivf.h | Simplified specialization patterns |
| bindings/python/CMakeLists.txt | Removed extra whitespace |
| benchmark/include/svs-benchmark/ivf/test.h | Removed unused LeanVec-specific members |
| .github/workflows/build-linux.yml | Added IVF matrix testing to CI |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| # Build from file loader | ||
| loader = svs.VectorDataLoader(test_data_svs, svs.DataType.float32) | ||
| matcher = UncompressedMatcher("bfloat16") | ||
| self._test_build(loader, svs.DistanceType.L2, matcher) |
There was a problem hiding this comment.
I would have a dedicated test for each distance.
There was a problem hiding this comment.
Yes, only L2 and MIP is supported in IVF right now. Check here the corresponding Vamana tests: https://github.com/intel/ScalableVectorSearch/blob/main/bindings/python/tests/test_vamana.py#L377
| ) | ||
|
|
||
| for expected in expected_results: | ||
| n_probes, k_reorder, k, nq, expected_recall = \ |
There was a problem hiding this comment.
What is this auto-formatting with \? Is this configured in our formatter? That's ancient, we should update those rules.
There was a problem hiding this comment.
We don't have formatting for Python code in the library
| size_t new_size, | ||
| const Alloc& SVS_UNUSED(allocator) | ||
| ) { | ||
| auto new_sqdata = SQDataset<typename Data::element_type, Data::extent>( |
There was a problem hiding this comment.
For my understanding: We create an empty SQDataset of new_size x "old" dimensions with same scale and bias?
| using value_type = const_value_type; | ||
|
|
||
| // Data wrapped in the library allocator. | ||
| using lib_alloc_data_type = SQDataset<T, Extent, lib::Allocator<T>>; |
There was a problem hiding this comment.
| // With 4 inner threads | ||
| test_build<svs::BFloat16>(svs::DistanceL2(), 4); | ||
| test_build<svs::BFloat16>(svs::DistanceIP(), 4); | ||
| // TBD: CI is not happy with this, investigate |
There was a problem hiding this comment.
Needs to be fixed before merge? Or follow-up?
There was a problem hiding this comment.
I will keep the comment for now, it passes on my local machine but not on the CI machines.
Uh oh!
There was an error while loading. Please reload this page.