Writer fencing deviates from the FizzBee spec in the Go implementation

[jayaprabhakar]

I am setting up model based testing for the WIP WriterFencing spec in the SlateDB Go implementation.

I see that the spec and the implementation significantly deviates from each other.

With a single writer slatedb.Open is called, I see 3 manifest files created in the S3 bucket. there are no WAL files created.

manifest/00000000000000000001.manifest
manifest/00000000000000000002.manifest
manifest/00000000000000000003.manifest

But as per the spec, when a Writer is opened, there should be a single WAL file and a single manifest file.

Please let me know if the writer fencing has been implemented. Also, are these 3 manifest files expected? If so, is there a way to filter/map these 3 objects to the 1 manifest sst object as expected in the design.

[naveen246]

Some observations:

Rust implementation, test code in the following repository
https://github.com/naveen246/slatedb_rs

// Just open and close the DB without writing any key

~/projects/slatedb_rs $ ls /tmp/test_kv_store/manifest 
00000000000000000001.manifest   00000000000000000002.manifest   00000000000000000003.manifest   00000000000000000004.manifest

~/projects/slatedb_rs $ ls /tmp/test_kv_store/wal 
00000000000000000001.sst

// Open, Put, Get and Delete a single key, and then close the DB

~/projects/slatedb_rs $ ls /tmp/test_kv_store/manifest 
00000000000000000001.manifest   00000000000000000002.manifest   00000000000000000003.manifest   00000000000000000004.manifest

~/projects/slatedb_rs $ ls /tmp/test_kv_store/wal     
00000000000000000001.sst        00000000000000000002.sst        00000000000000000003.sst

[naveen246]

Go implementation, test code in the following repository
https://github.com/naveen246/slatedb_go

// Just open and close the DB without writing any key

~/projects/slatedb_go $ ls /tmp/testDB/manifest 
00000000000000000001.manifest   00000000000000000002.manifest   00000000000000000003.manifest   00000000000000000004.manifest

No `wal` created

// Open, Put, Get and Delete a single key, and then close the DB

~/projects/slatedb_go $ ls /tmp/testDB/manifest 
00000000000000000001.manifest   00000000000000000002.manifest   00000000000000000003.manifest   00000000000000000004.manifest

~/projects/slatedb_go $ ls /tmp/testDB/wal     
00000000000000000001.sst        00000000000000000002.sst

[jayaprabhakar]

Thanks for checking. In the FizzBee model for writer fencing , there if only one manifest file, where as in the implementation they create 3 manifest files.

Do you know what are the contents of the other 2 files?

[naveen246]

The final struct stored in manifest is this

type ManifestV1T struct {
    ManifestId         uint64
    WriterEpoch        uint64
    CompactorEpoch     uint64
    WalIdLastCompacted uint64   
    WalIdLastSeen      uint64
    L0LastCompacted    *CompactedSstIdT
    L0                 []*CompactedSsTableT
    Compacted          []*SortedRunT
    Snapshots          []*SnapshotT
}

in this document:
https://github.com/slatedb/slatedb/blob/main/rfcs/0001-manifest.md

its mentioned:

Writers: Writers must update writer_epoch once on startup.
Compactors: Compactors must update compactor_epoch once on startup

on DB startup DB thread creates - manifest01
MemtableFlush thread increments writer_epoch and creates new manifest with updated writer_epoch - manifest02
Compaction thread increments compactor_epoch and creates new manifest with updated compactor_epoch - manifest03
last manifest is when MemtableFlush thread writes latest manifest before closing the DB - manifest04