[Bug]: <Title>Parallel_Build_Assertion_Failure

[chenghuaijun]

What happened?

Bug: Parallel Index Build Assertion Failure in DiskANN

Summary

When building DiskANN index with diskann.force_parallel_workers >= 32, an assertion failure occurs in parallel workers after the single-threaded training quantizer phase completes. The error manifests as:

ERROR:  assertion `left == right` failed
  left: ItemPointer { block_number: 0, offset: 2 }
 right: ItemPointer { block_number: 0, offset: 1 }
CONTEXT:  parallel worker

Environment

Component	Version
pgvectorscale	0.9.0(main branch this commit-id 4c04103 )
PostgreSQL	16
pgvector	0.8.2
Platform	Linux x86_64
Container	32 vCPU / 128 GB RAM

Reproduction Steps

Step 1: Create Table with 1M Vectors

CREATE TABLE cohere_base (
    id SERIAL PRIMARY KEY,
    embedding vector(768)
);

-- Insert 1,000,000 vectors (768 dimensions)
-- Use COPY or batch INSERT to load data

Step 2: Set Parallel Build Parameters

-- PostgreSQL parallel parameters
SET max_parallel_maintenance_workers = 32;
SET max_parallel_workers = 32;
SET maintenance_work_mem = '32GB';
SET maintenance_io_concurrency = 1024;

-- DiskANN parallel parameter (TRIGGER CONDITION)
SET diskann.force_parallel_workers = 32;

Step 3: Create DiskANN Index

\timing on
CREATE INDEX cohere_diskann_idx ON public.cohere_base
USING diskann (embedding vector_l2_ops)
WITH (
  storage_layout = 'memory_optimized',
  num_neighbors = 64,
  search_list_size = 200
);

Step 4: Observe Error

After approximately 60 minutes (during the single-threaded training quantizer phase), the parallel build phase starts and immediately fails:

NOTICE:  Starting index build with num_neighbors=64, search_list_size=200, max_alpha=1.2, storage_layout=SbqCompression.
NOTICE:  Parallel build with 32 workers
NOTICE:  Indexed 35953 tuples
ERROR:  assertion `left == right` failed
  left: ItemPointer { block_number: 0, offset: 2 }
 right: ItemPointer { block_number: 0, offset: 1 }
CONTEXT:  parallel worker
Time: 3609879.901 ms (01:00:09.880)

Root Cause Analysis

Assertion Failure Location

File: pgvectorscale/src/access_method/meta_page.rs
Lines: 378, 383

// meta_page.rs:375-383
let bytes = header.serialize_to_vec();
let off = tape.write(&bytes);
assert_eq!(off, ItemPointer::new(META_BLOCK_NUMBER, META_HEADER_OFFSET));  // Expects offset=1

let bytes = self.serialize_to_vec();
let off = tape.write(&bytes);
assert_eq!(off, ItemPointer::new(META_BLOCK_NUMBER, META_OFFSET));  // Expects offset=2

Call Stack

Main Process:
ambbuild() [build.rs:296]
  └─> meta_page.store(&index_relation, false) [build.rs:319]
      └─> ChainTapeWriter::reinit(index, PageType::Meta, stats, META_BLOCK_NUMBER) [meta_page.rs:372]
          └─> WritablePage::modify(index, block_number) [chain.rs:63]
          └─> page.reinit(page_type) [chain.rs:64]
              └─> pg_sys::PageInit(...) [page.rs:129]  // Clears page
          └─> tape.write(&header_bytes) [meta_page.rs:377]  // offset=1
          └─> tape.write(&meta_bytes) [meta_page.rs:382]   // offset=2
  └─> LaunchParallelWorkers(pcxt) [build.rs:415]

Parallel Worker (each of 32 workers):
_vectorscale_build_main() [build.rs:620]
  └─> MetaPage::fetch(&index_relation) [build.rs:696]
      └─> match page_type {
          PageType::MetaV1 => {
              └─> new_meta.store(index, false) [meta_page.rs:411]  // ⚠️ WRITING IN FETCH!
                  └─> ChainTapeWriter::reinit(...) [meta_page.rs:372]
                      └─> page.reinit(page_type) [chain.rs:64]
                          └─> pg_sys::PageInit(...) [page.rs:129]  // ⚠️ CLEARS PAGE AGAIN!
                      └─> tape.write(&header_bytes)  // offset=1
                      └─> tape.write(&meta_bytes)   // offset=2
          }
      └─> assert_eq!(off, ItemPointer::new(0, 1))  // FAILS: sees offset=2

Timeline

T0: Main process calls store() -> reinit() clears block 0 -> writes header(1), meta(2)
T1: LaunchParallelWorkers() starts 32 workers
T2: Worker 1 calls fetch() -> MetaV1 branch -> store() -> reinit() clears block 0
T3: Worker 2 calls fetch() -> MetaV1 branch -> store() -> reinit() clears block 0
...
Tn: Multiple workers overwrite the same meta page
Tn+1: Assertion check reads overwritten data -> expects offset=1, sees offset=2 -> FAIL

The Bug

MetaPage::fetch() is a READ operation but contains a WRITE operation in the PageType::MetaV1 branch.

// meta_page.rs:399-419
pub fn fetch(index: &PgRelation) -> MetaPage {
    match page_type {
        PageType::MetaV1 => {
            // This is a READ function, but it WRITES to the page!
            new_meta.store(index, false);  // ⚠️ Should NOT be here
            new_meta
        }
        PageType::MetaV2 => MetaPageV2::from_page(page).into(),  // READ only ✅
        PageType::Meta => Self::load(index),  // READ only ✅
        ...
    }
}

Each parallel worker calls fetch(), which triggers store() -> reinit() -> PageInit(), clearing the meta page. Multiple workers overwrite each other's data, causing the assertion to fail.

Why Buffer Locks Don't Prevent This

1. WritablePage::modify() acquires BUFFER_LOCK_EXCLUSIVE
2. Lock is released after commit() (via Drop trait)
3. Worker 1 writes -> commits -> releases lock
4. Worker 2 acquires lock -> reinitializes page -> commits -> releases lock
5. Worker 1's assertion reads Worker 2's data

Minimal Reproducible Example

-- Setup
CREATE TABLE test_vectors (id SERIAL, embedding vector(768));
INSERT INTO test_vectors (embedding)
SELECT '[' || array_to_string(array_agg(random()), ',', '0') || ']'::vector
FROM generate_series(1, 768 * 100000) i
GROUP BY i % 100000;

-- Trigger the bug
SET diskann.force_parallel_workers = 32;
SET max_parallel_maintenance_workers = 32;
CREATE INDEX ON test_vectors USING diskann (embedding vector_l2_ops)
WITH (storage_layout = 'memory_optimized');

-- Result: assertion failure in parallel worker

Workarounds

Workaround 1: Disable Forced Parallel Workers

-- Don't set diskann.force_parallel_workers, or set to -1 (automatic)
SET diskann.force_parallel_workers = -1;

Workaround 2: Reduce Parallel Workers

-- Use fewer parallel workers to reduce race condition probability
SET diskann.force_parallel_workers = 16;
-- Or even lower
SET diskann.force_parallel_workers = 8;

Workaround 3: Use Serial Build

-- Disable parallel build entirely
SET max_parallel_maintenance_workers = 0;

Suggested Fix

Option 1: Remove Write Operation from fetch()

pub fn fetch(index: &PgRelation) -> MetaPage {
    match page_type {
        PageType::MetaV1 => {
            // Only convert, don't write
            let old_meta = MetaPageV1::page_get_meta(...);
            let new_meta: MetaPage = (&*old_meta).into();
            new_meta  // Return converted struct without calling store()
        }
        PageType::MetaV2 => MetaPageV2::from_page(page).into(),
        PageType::Meta => Self::load(index),
        ...
    }
}

Option 2: Add Atomic Flag for Conversion

use std::sync::atomic::{AtomicBool, Ordering};

static META_CONVERSION_DONE: AtomicBool = AtomicBool::new(false);

pub fn fetch(index: &PgRelation) -> MetaPage {
    match page_type {
        PageType::MetaV1 => {
            if META_CONVERSION_DONE.compare_exchange(
                false, true, Ordering::SeqCst, Ordering::SeqCst
            ).is_ok() {
                new_meta.store(index, false);
            }
            // Wait and reload
            std::thread::yield_now();
            Self::load(index)
        }
        ...
    }
}

Option 3: Complete Conversion Before Launching Workers

In ambuild(), after meta_page.store(), ensure the meta page is in Meta or MetaV2 format before calling LaunchParallelWorkers().

Related Files

File	Lines	Description
pgvectorscale/src/access_method/build.rs	296-460	ambuild function
pgvectorscale/src/access_method/build.rs	620-716	_vectorscale_build_main worker entry
pgvectorscale/src/access_method/meta_page.rs	359-384	store() function
pgvectorscale/src/access_method/meta_page.rs	399-419	fetch() function (bug location)
pgvectorscale/src/util/chain.rs	56-73	reinit() clears page
pgvectorscale/src/util/page.rs	127-135	reinit() calls PageInit

Additional Notes

• The bug is triggered with diskann.force_parallel_workers >= 32 but may occur with lower values
• The training quantizer phase is single-threaded and takes ~60 minutes for 1M vectors
• The failure occurs immediately when parallel build phase starts
• Data integrity is preserved (transaction rolled back)
• Tested on Neon storage engine; may behave differently on standard PostgreSQL

pgvectorscale extension affected

0.9.0

PostgreSQL version used

16

What operating system did you use?

Linux 19ebb689cdc0 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

What installation method did you use?

Docker

What platform did you run on?

Other

Relevant log output and stack trace

How can we reproduce the bug?

see the markdown

Are you going to work on the bugfix?

None