FEATURE: improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU

[cpegeric]

User description

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #23254

What this PR does / why we need it:

1. improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU
2. Brute force index with GPU, Usearch and Go implementations
3. distance functions with loop unrolling and BCE hint
4. remove blas library (super slow)
5. chunk based Thread executor to replace the modulo based one.
6. change kmean clustering multiple thread to chunk based Thread executor

Current status:

1. Need to wait for usearch to merge the bug fix with their exact search golang api

• Fix: Exact search to return maxResults per query in golang unum-cloud/USearch#685

2. Need to wait for cuvs bug fix and merge fix with their golang API

• Ivf_flat extends golang APIs rapidsai/cuvs#1600
• [BUG] brute force index crash after ivf index build with golang rapidsai/cuvs#1622

---

PR Type

Enhancement, Bug fix

---

Description

• Implement brute force vector index for efficient centroid search in ProductL2

• Replace manual distance computation with vectorized search using USearch and GPU support

• Refactor IVFFlat centroid lookup to use brute force index instead of heap-based search

• Add CPU and GPU implementations with metric type conversions and proper resource management

---

Diagram Walkthrough

flowchart LR
  A["ProductL2 Operator"] -->|build phase| B["Create BruteForceIndex<br/>from centroids"]
  B -->|CPU/GPU| C["UsearchBruteForceIndex<br/>or GoBruteForceIndex"]
  A -->|probe phase| D["Search nearest<br/>centroid"]
  D -->|exact search| C
  C -->|return keys| E["Assign vectors<br/>to clusters"]
  F["IVFFlat Search"] -->|load centroids| G["BruteForceIndex<br/>for centroid lookup"]
  G -->|findCentroids| H["Return probe<br/>centroid IDs"]

Loading

File Walkthrough

	Relevant files
Enhancement	10 files types.go Add GetArrayElementSize method for vector types                    +10/-0    product_l2.go Refactor to use brute force index for centroid search        +107/-108 types.go Add brute force index field to container struct                    +10/-6    brute_force.go Implement brute force index with USearch and Go backends  +295/-0  cpu.go CPU-only brute force index factory function                            +31/-0    gpu.go GPU brute force index using CUVS library with resource management +195/-0  search.go Replace centroid heap search with brute force index lookup +36/-41  gpu.go Add metric type to CUVS distance mapping                                  +32/-0    types.go Add metric type to USearch metric mapping                                +7/-0      types.go Add NThreads field to RuntimeConfig for brute force index +1/-0
Tests	4 files brute_force_test.go Add unit tests for brute force index implementations          +66/-0    gpu_test.go GPU brute force index concurrent stress tests                        +72/-0    gpu_test.go Add integration test for IVF and brute force index              +76/-1    issue_test.go Add GPU stress test for centroid search with concurrent queries +264/-0
Bug fix	1 files gpu.go Refactor GPU K-means to defer resource allocation until clustering +31/-43
Dependencies	2 files go.mod Add viterin/partial dependency and use custom USearch fork +2/-0      go.sum Update checksums for new dependencies                                        +4/-2
Configuration changes	1 files Makefile Update USearch to hotfix version with SimSIMD support        +4/-4

---

[mergify]

Merge Queue Status

✅ The pull request has been merged at 69ed5d2

This pull request spent 55 minutes 22 seconds in the queue, including 55 minutes 11 seconds running CI.
The checks were run in-place.

Required conditions to merge

• #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • FEATURE: improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU #23255
• #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
  • FEATURE: improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU #23255
• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • FEATURE: improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU #23255
• branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
  • FEATURE: improve IVFFLAT assign center in ProductL2 by usearch exact search and GPU #23255
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Ubuntu/x86
  • check-neutral = Matrixone CI / SCA Test on Ubuntu/x86
  • check-skipped = Matrixone CI / SCA Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
  • check-neutral = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
  • check-skipped = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage