feat: mmap-backed subtrees and disk-backed TxMap for block RAM reduction

[icellan]

Summary

Both blockvalidation and blockassembly/subtreeprocessor hold subtree Node arrays and transaction tracking maps entirely in Go heap memory. At scale (1000 subtrees × 1M transactions = 1 billion tx per block), this consumes ~376 GB of heap and creates severe GC pressure.

This PR moves the two largest memory consumers off-heap:

• Node arrays (48 GB) → file-backed mmap: The Node struct has zero pointer fields ([32]byte + uint64 + uint64 = 48 bytes), making it safe for mmap. []Node slices backed by mmap are indistinguishable from heap-backed — all existing code works unchanged. The OS manages paging between RAM and disk transparently.

• currentTxMap (200 GB) → sharded cuckoo filters + multi-disk BadgerDB: The hot path (SetIfNotExists) only needs an existence check. 4096 independent cuckoo filter shards provide 35M ops/sec for dedup with ~1 GB memory. TxInpoints data is persisted to BadgerDB across multiple physical disks via dedicated writer goroutines — one per disk for linear I/O scaling.

All changes are fully opt-in via new settings. When unconfigured (empty strings), behavior is identical to before.

Architecture

mmap-backed Node arrays (go-subtree v1.1.9)

Subtree.Nodes []Node  →  backed by file-backed mmap instead of Go heap
                          ├── GC never scans mmap'd memory
                          ├── OS pages cold subtrees to disk automatically
                          ├── fd closed after mmap (zero persistent file descriptors)
                          └── All existing code works unchanged (same []Node type)

DiskTxMap (replaces SplitTxInpointsMap when configured)

Hot path (35M ops/sec):
  hash → shardOf(hash) → shard.mu.Lock()
       → cuckoo filter Lookup/Insert
       → recent map check
       → shard.mu.Unlock()
       → async channel send to writer goroutine

Writer goroutines (one per disk, linear I/O scaling):
  channel recv → serialize TxInpoints → Badger batch.Set()
              → auto-flush every 50K entries

Benchmarks

Metric	Result
Existence check throughput (dedup layer)	35M ops/sec @ 28 ns/op
Full SetIfNotExists, 4 disks	1.56M ops/sec @ 641 ns/op
mmap Node AddNode overhead vs heap	+0.5 ns (2.47 → 2.94 ns/op)
Heap reduction (128 subtrees × 64K nodes)	383 MB → 0 MB (100%)

New Settings

Setting	Default	Description
blockassembly_subtreeMmapDir	""	Directory for mmap-backed subtree Node files
blockassembly_txMapDirs	""	Pipe-separated paths for multi-disk BadgerDB TxInpoints storage
blockvalidation_subtreeMmapDir	""	Directory for mmap-backed subtree loading during validation

Example production configuration:

blockassembly_subtreeMmapDir=/mnt/nvme0/subtree-mmap
blockassembly_txMapDirs=/mnt/nvme0/txmap|/mnt/nvme1/txmap|/mnt/nvme2/txmap|/mnt/nvme3/txmap
blockvalidation_subtreeMmapDir=/mnt/nvme0/bv-subtree-mmap

Files Changed

File	Change
go.mod	go-subtree v1.1.9, cuckoo filter dependency
stores/tempstore/badger.go	Added Delete() method
subtreeprocessor/disk_tx_map.go	DiskTxMap: sharded cuckoo filters + multi-disk Badger
subtreeprocessor/disk_tx_map_test.go	8 functional tests + concurrent stress test
subtreeprocessor/disk_tx_map_benchmark_test.go	Throughput + multi-disk benchmarks
subtreeprocessor/SubtreeProcessor.go	mmap subtree creation, DiskTxMap init, cleanup paths
subtreeprocessor/options.go	WithMmapDir, WithTxMapDirs options
blockassembly/BlockAssembler.go	Pass settings to subtreeprocessor
blockvalidation/BlockValidation.go	mmap subtree loading, cleanup
blockvalidation/quick_validate.go	mmap subtree deserialization
blockvalidation/get_blocks.go	mmap subtree loading in catchup
settings/blockassembly_settings.go	SubtreeMmapDir, TxMapDirs
settings/blockvalidation_settings.go	SubtreeMmapDir

Test plan

• All existing blockassembly tests pass
• All existing blockvalidation tests pass (263s suite)
• DiskTxMap: 8 functional tests + 16-goroutine concurrent test with race detector
• Memory profiling: 128 mmap subtrees = 0 MB heap (vs 383 MB heap-backed)
• Throughput: existence layer 35M ops/sec, multi-disk 1.56M ops/sec
• Integration: run in dev mode with mmap dirs configured

[github-actions]

🤖 Claude Code Review

Status: Complete

---

Summary

This PR successfully implements off-heap memory optimization for block processing through mmap-backed subtrees and disk-backed transaction maps. The implementation is well-architected with appropriate opt-in configuration and maintains backward compatibility.

Key Findings

Architecture (Strong):

• ✅ Sharded cuckoo filters (4096 shards) provide 35M ops/sec existence checks
• ✅ Multi-disk BadgerDB with dedicated writer goroutines enables linear I/O scaling
• ✅ Proper separation of hot path (filter+channel) vs cold path (disk flush+check)
• ✅ The slowMu lock correctly prevents race conditions in the slow path

Concurrency Safety (Verified):

• ✅ Previous race condition concerns have been addressed via slowMu lock
• ✅ Lock is held across channel send, ensuring FIFO ordering prevents duplicates
• ✅ clearRecentMapsForDisk only runs in writer goroutine after flush

Configuration (Well Designed):

• ✅ Fully opt-in via settings (empty = original behavior)
• ✅ Clear documentation with memory/performance tradeoffs
• ✅ Multi-disk paths support optimal I/O parallelism

Testing:

• ✅ 8 functional tests + concurrent stress test with race detector
• ✅ Benchmarks demonstrate 1.56M ops/sec with 4 disks
• ✅ Memory profiling shows 100% heap reduction for mmap nodes

Code Quality:

• ✅ Follows project conventions (snake_case files, proper error handling)
• ✅ gosec linter added to CI for security scanning
• ✅ Appropriate use of atomic operations and channels

Previous Review Items

The race condition issues flagged in earlier reviews have been resolved through the addition of the slowMu lock pattern. The current implementation correctly serializes the slow path to prevent the "recent map cleared during disk check" scenario.

---

No blocking issues found. The implementation is production-ready with appropriate safeguards and monitoring via Stats() methods.

[github-actions]

🤖 Claude Code Review

Status: Complete

---

Current Review:

After thorough analysis of the mmap-backed subtrees and disk-backed TxMap implementation, the code appears well-designed and functionally correct. The complex concurrency patterns using slowMu locks effectively prevent race conditions.

Note on existing inline comments: Three previous inline comments claim race conditions exist in clearRecentMapsForDisk. After detailed analysis, these appear to be incorrect. The slowMu lock is held across the entire slow path (including the channel send), which ensures:

• Thread A writes to channel before releasing slowMu
• Thread B acquires slowMu only after Thread A releases it
• Thread B flushes Thread A write before checking disk
• Recent map clearing occurs after flushing to disk, maintaining correctness

The implementation correctly handles cross-shard clearing through proper lock ordering and FIFO channel semantics.

Positive observations:

• Comprehensive benchmarks showing 35M ops/sec dedup layer performance
• Proper resource cleanup with defer patterns
• Good separation of concerns (filter layer, disk layer, metrics)
• Settings are opt-in (empty strings = no change)
• Multi-disk I/O parallelism is well-architected

Minor notes:

• gosec linter addition is good security practice
• Code complexity is high but necessary for performance requirements
• Prometheus metrics provide good observability

[github-actions]

🤖 Claude Code Review

Status: Complete

---

Current Review: No Critical Issues Found

This PR implements mmap-backed subtrees and disk-backed transaction maps to reduce block RAM usage at scale. The implementation is sound with all previously identified race conditions resolved.

Key Findings:

• ✅ Race condition fix confirmed - All three disk map implementations now use slowMu lock pattern that prevents clearRecentMapsForDisk from racing with slow-path operations
• ✅ Opt-in architecture - Empty settings strings preserve original heap-based behavior (backward compatible)
• ✅ Multi-disk sharding - Proper modulo distribution across disk shards for linear I/O scaling
• ✅ Error handling - Comprehensive cleanup in constructor failure paths

Architecture:

• Sharded cuckoo filters (~1GB) for fast in-memory deduplication at 35M ops/sec
• BadgerDB persistence across multiple disks with dedicated writer goroutines
• File-backed mmap for Node arrays (GC-free, OS-managed paging)

Settings:

• blockassembly_subtreeMmapDir: Off-heap Node storage (saves ~48GB at 1000 subtrees × 1M nodes)
• blockassembly_txMapDirs: Multi-disk TxInpoints map (pipe-separated paths)
• blockvalidation_subtreeMmapDir: mmap subtree loading during validation

No blocking issues identified. The slowMu pattern effectively serializes the slow path to prevent the previously flagged race condition.

---

History:

• ✅ Fixed: Race condition in clearRecentMapsForDisk resolved via slowMu serialization pattern
• ✅ Fixed: diskOf() modulo distribution corrected

[sonarqubecloud]

Quality Gate failed

Failed conditions
75.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud