Sync: delete propagation is fundamentally broken — tombstones unfinished, shared deletes never broadcast

[jamiepine]

Summary

Delete propagation across synced devices is broken for virtually every model type. The tombstone system for device-owned models is scaffolded but not wired up end-to-end, and shared model deletes (tags, spaces, space items, collections, metadata) are never broadcast at all. The net result: deleting anything on one device has no effect on peers, and deleted records resurrect on next sync.

This is a critical data integrity issue — users cannot remove data from their library in a way that persists across devices.

Architecture Context

Spacedrive has two sync models:

• Shared (HLC log-based): space, space_group, space_item, tag, collection, user_metadata, etc. Changes are written to shared_changes peer log with HLC timestamps, broadcast to peers, and applied via apply_shared_change.
• Device-owned (state-based): entry, location, volume. Owner device broadcasts current state; peers apply via apply_state_change. Deletions are tracked via device_state_tombstone table.

Both systems have working infrastructure for deletes. The problem is that almost nothing in the application layer actually calls into that infrastructure.

---

Bug 1: Shared model deletes — never broadcast

The shared sync pipe handles ChangeType::Delete correctly end-to-end:

1. sync_model(model, ChangeType::Delete) → writes to peer_log ✓
2. Peer receives → apply_shared_change → DELETE FROM table WHERE uuid = ? ✓
3. Backfill includes Delete entries from peer_log in HLC order ✓
4. Every shared model has a working ChangeType::Delete handler ✓

The problem: no user-facing action ever fires that pipe. Only the indexing system does (for entries).

Audit of all delete code paths

Action / Function	File	Calls sync_model(_, Delete)?
spaces.delete_item	core/src/ops/spaces/delete_item/action.rs:43	No
spaces.delete_group	core/src/ops/spaces/delete_group/action.rs:44	No
spaces.delete	core/src/ops/spaces/delete/action.rs:44	No
spaces.reorder_items	core/src/ops/spaces/reorder/action.rs:98	No (Update also missing)
spaces.reorder_groups	core/src/ops/spaces/reorder/action.rs:47	No (Update also missing)
TagManager::delete_tag	core/src/ops/tags/manager.rs:360	No
TagManager::remove_relationship	core/src/ops/tags/manager.rs:1274	No
MetadataManager::remove_semantic_tags	core/src/ops/metadata/manager.rs:332	No
network.revoke (device)	core/src/ops/network/revoke/action.rs	No
Indexing entry deletion	core/src/ops/indexing/change_detection/persistent.rs:370	Yes ✓
Indexing processing deletion	core/src/ops/indexing/phases/processing.rs:706	Yes ✓

Every shared model delete in the app (except entries via indexing) does model.delete(db) + emit_deleted() for local UI — but never writes to the peer log.

Consequence

Alice deletes a tag → Alice's DB removes it → peer log has no Delete entry → Bob's DB still has it → on next get_full_shared_state() backfill (which scans Bob's DB directly via Syncable::query_for_sync), Bob ships the tag back → Alice's apply_shared_change upserts it → tag reappears.

---

Bug 2: Shared delete pruning — no durable record

Even for entries where ChangeType::Delete IS written to the peer log, the peer log is ephemeral:

• prune_acked() in peer_log.rs deletes entries once all peers ACK
• Once a Delete entry is pruned, it's gone forever
• No tombstone equivalent exists for shared models
• If a new device joins after pruning, or a device was offline during the Delete window, it will never learn about the deletion
• get_full_shared_state() scans live DB records — a record deleted on the sender but still present on the receiver gets resurrected

This is an architectural gap: device-owned sync has persistent tombstones; shared sync has only the ephemeral peer log.

---

Bug 3: Device-owned tombstones — scaffolded but not wired up

What's built

• device_state_tombstone table (model_type, record_uuid, device_id, deleted_at)
• is_tombstoned() guard in apply_state_change for entry, location, volume
• get_deletion_tombstones() query function
• apply_deletion() implementations for entry, location, volume
• Protocol sends deleted_uuids in StateResponse
• Backfill applies deleted_uuids via apply_deletion

Where it breaks

Only 2 callers ever write tombstones

Caller	File	Model
Location delete	core/src/location/manager.rs:609	location
Volume untrack	core/src/volume/manager.rs:1595	volume

Entry — the most important model — never writes tombstones. Entry deletion during indexing calls sync_models_batch(entries, ChangeType::Delete) which goes through the shared peer_log path. The delete_subtree docstring at database_storage.rs:1162 explicitly states it does NOT create tombstones.

Tombstones are never sent on initial backfill

// core/src/service/network/protocol/sync/handler.rs:249
} else {
    vec![] // Full sync doesn't need tombstones
};

start_backfill() always passes since: None → StateRequest has no watermark → handler returns empty deleted_uuids. The theory is "full sync sends all current records, so missing = implicitly deleted." But this fails when the receiving side already has stale records from a previous partial sync.

Per-resource watermarks may never initialize

// core/src/service/sync/backfill.rs:425-428
if let Some(max_ts) = max_received_timestamp {
    self.peer_sync
        .update_resource_watermark(primary_peer, &model_type, max_ts)
        .await?;
}

If a model type has zero records during initial backfill, max_received_timestamp is None → watermark never set. Subsequent catch_up_from_peer calls get_resource_watermark → None → passes since: None → full sync → tombstones skipped. Permanent loop: watermarks never initialize, tombstones never queried.

The _since_watermark parameter is deprecated and ignored

// core/src/service/sync/backfill.rs:368
_since_watermark: Option<chrono::DateTime<chrono::Utc>>, // Deprecated: use per-resource watermarks

catch_up_from_peer passes the global watermark but it's silently dropped. Per-resource watermarks are used instead — which may not be initialized (above).

---

Complete status matrix

Model	Type	Tombstone written?	Delete broadcast?	Resurrection protected?	Status
Entry	Device-owned	No	Via shared peer_log	is_tombstoned() check exists but no tombstone written	Partially works (until peer_log pruned)
Location	Device-owned	Yes	Via tombstone in StateResponse	is_tombstoned() ✓	Works IF watermarks initialize
Volume	Device-owned	Yes	Via tombstone in StateResponse	is_tombstoned() ✓	Works IF watermarks initialize
Tag	Shared	N/A	Never — delete_tag() skips sync	N/A	Broken
Space	Shared	N/A	Never — action skips sync	N/A	Broken
SpaceItem	Shared	N/A	Never — action skips sync	N/A	Broken
SpaceGroup	Shared	N/A	Never — action skips sync	N/A	Broken
Collection	Shared	N/A	Never	N/A	Broken
UserMetadata	Shared	N/A	Never	N/A	Broken
Device (revoke)	Device-owned	No	Never	N/A	Broken

---

Proposed Fix

Phase 1: Wire up shared model deletes (immediate)

Add library.sync_model(&model, ChangeType::Delete) before every model.delete(db) call:

• delete_item/action.rs
• delete_group/action.rs
• delete/action.rs (space)
• tags/manager.rs (delete_tag, remove_relationship)
• metadata/manager.rs (remove_semantic_tags)
• network/revoke/action.rs

Also add sync_model(&model, ChangeType::Update) to both reorder actions.

Phase 2: Persistent shared deletion records

The peer_log prune-after-ACK design means shared deletes are ephemeral. Options:

• Option A: Add a shared_tombstone table parallel to device_state_tombstone — write on Delete, check before Insert in apply_shared_change, send during full-state backfill
• Option B: Never prune Delete entries from peer_log (only prune Insert/Update)
• Option C: During get_full_shared_state, cross-reference what the requesting peer previously had (requires tracking) — complex

Phase 3: Fix device-owned tombstone chain

• Write entry tombstones during indexing deletion (not just peer_log)
• Initialize watermarks to epoch (not None) after initial backfill completes, even for model types with zero records
• Send tombstones during initial backfill (remove the vec![] shortcut) OR reconcile by diffing received state against local state

Phase 4: Anti-entropy reconciliation

Add a periodic reconciliation mechanism that detects records present on one device but not another, and resolves the divergence. This is the safety net for when all other mechanisms fail (pruned logs, missed tombstones, network partitions).

---

Affected Files

Shared delete callers (missing sync_model):

• core/src/ops/spaces/delete_item/action.rs
• core/src/ops/spaces/delete_group/action.rs
• core/src/ops/spaces/delete/action.rs
• core/src/ops/spaces/reorder/action.rs
• core/src/ops/tags/manager.rs
• core/src/ops/metadata/manager.rs
• core/src/ops/network/revoke/action.rs

Tombstone infrastructure:

• core/src/infra/db/entities/device_state_tombstone.rs
• core/src/infra/sync/syncable.rs (is_tombstoned)
• core/src/service/sync/peer.rs (get_deletion_tombstones)
• core/src/service/network/protocol/sync/handler.rs:249 (tombstone skip on full sync)

Watermark initialization:

• core/src/service/sync/backfill.rs:425 (conditional watermark update)
• core/src/service/sync/backfill.rs:368 (deprecated _since_watermark)

Entry tombstone gap:

• core/src/ops/indexing/change_detection/persistent.rs
• core/src/ops/indexing/phases/processing.rs
• core/src/ops/indexing/database_storage.rs