[ENH] Hard delete wal3 collections.

[rescrv]

Description of changes

This PR introduces code to destroy a wal3 log by deleting everything it reasonably could be known to
contain.

Test plan

Two new integration tests passed.

Documentation Changes

N/A

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Implement Hard Delete for wal3 Collections & Enhance Log Contention Handling

This PR introduces a comprehensive 'destroy' mechanism for wal3 logs, enabling hard deletion of all WAL3-managed files in object storage tied to a collection. It modifies, extends, and thoroughly tests the delete path: recursively traversing manifest, snapshots, fragments, cursors, and GC records, ensuring all are removed and reporting failure if any remain. In addition, major changes rework the log writer's contention/retry logic to robustly handle concurrent writers and garbage collection, notably making contention failovers more predictable and recoverable. S3 delete error handling and several test suites are also updated to reflect these new semantics and to harden durability against edge cases.

Key Changes:
• Add destroy.rs with a recursive destroy log routine (collection hard delete for WAL3 objects)
• Export destroy function in wal3 public interface and wire up in GC orchestrator (orchestrator_v2)
• Refactor log writer concurrency logic: add better LogContention error granularity and robust contention recovery/retry (LogWriter, writer.rs)
• Update all error handling, especially for S3 deletion (return NotFound as expected in GC/deletes)
• Add comprehensive integration tests (test_k8s_integration_97_destroy, destroy_wedge, contention) covering destruction, edge cases, and log contentions
• Update manifest and garbage collection file/layout semantics to ensure all generated garbage is cleaned and checked
• Adapt log-contention error handling logic across all callsites, including in log-service and GC orchestrator
• Add path utility helpers and update path conventions to ensure correct file prefixing and deletion semantics

Affected Areas:
• wal3 (destroy.rs, lib.rs, writer.rs, gc.rs, manifest.rs)
• garbage_collector_orchestrator_v2.rs
• log-service (push/read/GC apis and error handling)
• storage/s3.rs (delete error propagation and path prep)
• Integration/CI tests for destroy/contention/gc
• README and utility path generation

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[BestPractice]

Path concatenation is missing a separator - resulting paths will look like "<prefix>log/..." instead of "<prefix>/log/...".

format!(
-        "{}{}/Bucket={:016x}/FragmentSeqNo={:016x}.parquet",
-        prefix,
-        fragment_prefix(),
+        "{}/{}/Bucket={:016x}/FragmentSeqNo={:016x}.parquet",
+        prefix,
+        fragment_prefix(),
         fragment_seq_no.bucket(),
         fragment_seq_no.0,
     )

[propel-code-bot]

[PerformanceOptimization]

[performance] Arc::new(self.storage.clone()) is rebuilt for every iteration even though the Storage instance is immutable. Construct the Arc once before the loop to avoid repeated allocation and reference-count churn.

Suggested change

	if let Err(err) = wal3::destroy(
	let storage = Arc::new(self.storage.clone());
	for collection_id in ordered_soft_deleted_to_hard_delete_collections {
	if let Err(err) = wal3::destroy(
	Arc::clone(&storage),
	collection_id.storage_prefix_for_log(),
	)
	.await
	{
	tracing::error!("could not destroy/hard delete wal3 instance: {err:?}");
	// NOTE(rescrv): The alternative is to have a dead letter queue for this
	// collection. Once hard deleted from sysdb it'll be "impossible" to find the
	// garbage later, so leave the soft-deleted state to drive this state machine
	// to resolution, probably with operator involvement as this should never
	// happen and is tested.
	continue;
	}
	self.sysdb_client
	.finish_collection_deletion(
	self.tenant.clone().ok_or(GarbageCollectorError::InvariantViolation(
	"Expected tenant to be set".to_string(),
	))?,
	self.database_name.clone().ok_or(GarbageCollectorError::InvariantViolation(
	"Expected database to be set".to_string(),
	))?,
	collection_id,
	)
	.await?;
	}

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

[Sicheng-Pan]

q: if log delete fails should we continue instead of return Err(...)? I suppose if we let it pass here then this collection entry got removed from sysdb, and the log dangles in object store

[rescrv]

I chose to let it hard delete the next collection in the list. Returning error stops that possibility. What would you do instead?

[Sicheng-Pan]

q: do we need to think more carefully about what delete options to use here (and in other places as well)