Add tombstone purge tests with 1-hour interval validation

[pimpin]

Context

This PR addresses the tombstone purge issue discussed in Couchbase support ticket #70596.

Final Test Results ✅

After extensive testing, we have successfully validated the tombstone purge behavior with the following confirmed findings:

✅ Tombstone Purge Works Correctly

Test configuration:

• Metadata purge interval: 0.04 days (1 hour)
• Configured at bucket creation via REST API
• Wait time: 65 minutes + compaction

Result: Tombstone completely purged from CBS (xattr _sync absent)

• Document can be re-created with same ID
• Re-created document has flags=0 (treated as new)
• Test report: test_results/test_run_2025-11-01_06-54-46_a427ac3/

⚠️ Critical Finding: Configuration Timing Matters

When Configured	Tombstone Purge Result
At bucket creation	✅ Purged after 1h + compaction
After tombstone exists	❌ Persists despite 1h + compaction

Conclusion: The metadata purge interval must be configured before tombstones are created. Retroactive configuration does not purge existing tombstones.

❌ Reset Checkpoint Limitation Discovered

Testing revealed that reset checkpoint alone does NOT re-push unmodified documents:

• Reset checkpoint clears sync history
• But CBLite only pushes documents that changed since last successful sync
• Unmodified documents are not re-pushed even with reset

SGW logs evidence: Type:proposeChanges #Changes: 0 after reset

Implication for BC-994: Documents must be modified locally before reset checkpoint to trigger re-push.

What's in this PR

1. Automated Test Infrastructure

Complete test automation infrastructure with:

• Git validation: Ensures clean state before running
• Docker management: Automatic rebuild for reproducible environment
• Timezone sync: Validates container/host timezone alignment
• Structured reporting: Generates comprehensive reports in test_results/

Each test run produces:

• Executive summary (README.md)
• Full _sync xattr states at each checkpoint
• Complete console output
• CBS and SGW container logs
• GitHub commit links

2. CBS/SGW Helper Functions

CBS operations (utils/cbs_admin.rs):

• get_metadata_purge_interval(): Query current purge interval configuration
• set_metadata_purge_interval(): Configure via REST API with validation
• get_sync_xattr(): Extract _sync xattr for tombstone detection
• check_doc_in_cbs(): Verify document state (live/tombstone/purged)
• compact_cbs_bucket(): Trigger manual compaction

SGW operations (utils/sgw_admin.rs):

• add_or_update_user(), get_session(): User management
• get_doc_rev(): Get document revision (including deleted)
• delete_doc_from_central(): Delete document from central only
• check_doc_exists_in_central(): Verify document existence in SGW
• purge_doc_from_sgw(): Purge tombstone from SGW
• compact_sgw_database(): Trigger SGW compaction

Infrastructure (utils/):

• git_checker.rs: Git status validation and commit info extraction
• docker_manager.rs: Docker lifecycle management
• test_reporter.rs: Structured report generation

3. Test Examples

tombstone_purge_test.rs (~70 minutes)

Complete automated test validating tombstone purge after 1-hour interval.

Features:

• Automatic Docker environment rebuild
• Git validation
• Structured reporting with full state capture
• Validates Thomas's recommendations

Run: cargo run --features=enterprise --example tombstone_purge_test

tombstone_quick_check.rs (~30 seconds)

Rapid validation of tombstone detection via _sync xattr queries.

Run: cargo run --features=enterprise --example tombstone_quick_check

check_cbs_config.rs (instant)

Utility to verify CBS bucket metadata purge interval configuration.

Run: cargo run --features=enterprise --example check_cbs_config

ticket_70596.rs

Antoine's original test for auto-purge when documents move to inaccessible channels.

Run: cargo run --features=enterprise --example ticket_70596

4. Docker Configuration Improvements

Metadata purge interval (configure-server.sh):

• Uses REST API instead of CLI for per-bucket configuration
• Configured to 0.04 days (1 hour) at bucket creation
• Required parameters: autoCompactionDefined=true, purgeInterval, parallelDBAndViewCompaction

Sync function (sync-function.js):

• Includes BC-994 soft_delete logic for testing
• Detects resurrection: !oldDoc && updatedAt > 1 hour
• Routes to soft_deleted channel + sets TTL

Timezone sync (docker-compose.yml):

• Passes TZ environment variable to containers
• Ensures time-based logic consistency

Critical Discoveries

Discovery #1: _sync xattr Structure (flags field)

Tombstone detection requires checking _sync.flags:

Live document:

{
  "cas": "...",
  "channels": { "channel1": null },
  "sequence": 7,
  "value_crc32c": "0x548d82c8"
}

Tombstone:

{
  "flags": 1,                    // PRIMARY INDICATOR
  "tombstoned_at": 1761980103,   
  "channels": { "channel1": { "del": true } },
  "value_crc32c": "0x00000000"
}

Purged:

• _sync xattr completely absent
• Only document ID in query results

NB : to get sync meta data, you must query them explicitly (otherwise they are filtered out from meta() :

SELECT META().id, META().xattrs._sync as sync_metadata 
FROM `billeo-cb-bucket`
USE KEYS ["b596f899-f2d0-49e7-a8a8-a67955329802","880e8265-9504-4c16-b57f-e821b3f4f1e7"
]

Discovery #2: Metadata Purge Interval Configuration

CLI method (cluster-wide, insufficient):

couchbase-cli setting-compaction --bucket my-bucket --metadata-purge-interval 0.04

Result: autoCompactionSettings: false at bucket level

REST API method (per-bucket, correct):

curl -X POST http://localhost:8091/pools/default/buckets/my-bucket \
  -d "autoCompactionDefined=true" \
  -d "purgeInterval=0.04" \
  -d "parallelDBAndViewCompaction=false"

Result: purgeInterval: 0.04 at bucket root level

Discovery #3: CBS API Validation Bug

The REST API accepts purge interval values below 0.04 days with 200 OK instead of rejecting them:

• Documentation minimum: 0.04 days (1 hour)
• API accepts: 0.0007 days (1 minute) without error
• Configuration is silently ignored

Recommendation: API should validate like Web UI does.

Discovery #4: Tombstone Purge Timing

Purge interval must be configured before tombstones are created:

• Purge eligibility based on tombstoned_at + purge_interval
• Retroactive configuration does not apply to existing tombstones
• Re-created documents always get flags=0 regardless

Discovery #5: Reset Checkpoint Does Not Re-Push Unmodified Documents

Testing BC-994 scenario revealed:

• Reset checkpoint clears replication history
• But CBLite only pushes modified documents
• Unmodified documents are not re-evaluated/re-pushed

SGW logs evidence:

Type:proposeChanges #Changes: 0

Implication: For BC-994, documents must be modified locally before reset to trigger push.

Testing

Prerequisites

Set timezone for time-based logic:

export TZ="Europe/Paris"  # Or your local timezone

Quick Validation

Verify tombstone detection (~30 seconds):

cargo run --features=enterprise --example tombstone_quick_check

Verify CBS configuration:

cargo run --features=enterprise --example check_cbs_config

Full Test

Complete tombstone purge test with automation (~70 minutes):

cargo run --features=enterprise --example tombstone_purge_test

What it does automatically:

• ✅ Validates git status
• ✅ Rebuilds Docker environment (down -v && up)
• ✅ Verifies timezone synchronization
• ✅ Verifies CBS purge interval configuration
• ✅ Runs test with state capture at each checkpoint
• ✅ Generates structured report in test_results/
• ✅ Captures CBS and SGW logs

Test Report

Report location: test_results/test_run_<timestamp>_<commit_sha>/

Contents:

• README.md: Executive summary
• metadata.json: Test metadata and GitHub link
• tombstone_states.json: _sync xattr at each checkpoint
• test_output.log: Complete console output
• cbs_logs.log, sgw_logs.log: Container logs

Related

• PR Add Docker-based replication testing environment with Sync Gateway & Couchbase Server #25 by @Nao-ris - Docker-based replication testing environment
• Couchbase support ticket #70596
• billeo-engine PR #7672 - BC-994 soft_delete implementation

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]