run Replication::FailureRemediator.prune_replication_failures method on all druids for ZippedMoabVersion rows created/updated since 2025-12-18

[jmartin-sul]

storytime topic: zipped_moab_versions being created before zip creation to indicate that replication has started for the druid/version/endpoint; and the related inscrutability/inconvenience of having to figure out what needs cleanup in the DB, as happens when the ZMV rows get created but then related jobs get dropped from the replication queue. Would collapsing zip_parts into zipped_moab_versions help with this? that is ticketed as #2505

blocked by storytime -- Before remediating and losing possible test data, we want to discuss data model and/or code changes that might make this corner of replication tracking more legible. Though it would not be terribly hard to recreate this in stage (e.g. pause zipmaker, version some stuff, let the zipmaker jobs get queued, delete the queue contents to simulate replication jobs getting inadvertently evicted and leaving permanently childless ZMVs).

For the query, this will require joining back to PreservedObject to get the druid, since that's what the method wants. But we should use the prune_replication_failures method because it's tested and has some safeguards in place to prevent pruning too aggressively, and to make sure that deletes are done transactionally, and in the right order.

This could take a while (possibly days) to run, and so should be done in a screen session.

If it's looking like this will take weeks to run, we can discuss a more efficient approach.

[jmartin-sul]

note from planning this morning: this is one of our more frequent inscrutable error situations. would be good to have a story time on data model rethinking, to see if we can make this easier to undrestand, less error prone, easier to remediate when needed, etc

[jmartin-sul]

idea from storytime: can we at least take the status and parts count off of zip parts, move that to parent ZMV table, to make the status of the ZMV more easily queryable, w/o having to join to zip parts so much?

[jmartin-sul]

idea from storytime: can replication audit run prune_replication_failures?

[jmartin-sul]

possible states for ZMV if status and parts count were moved up to ZMV:

• created (no zip parts yet, no zip file created -- should exist on an endpoint, but no zip file made yet)
• unreplicated (zip files created, but not yet shipped)
• partially_replicated (zips created, delivery started but unfinished, rest could be replicated)
  • if auditor detects partially_replicated, kick off replication job to finish. will get de-duped by usual job locking mechanism. assumes zip file is still present?
• replication_error or replication_failed -- partially replicated, but can't do the rest automatically, need to intervene manually and delete from cloud. would happen when e.g. a multi-part zip hits an error in the replication pipeline. this status would probably be set by audit method, whether manually or automatically triggered.
• ok -- all parts replicated
• ???

bonus: this would make it easier to make the dashboard more honest, because we could query for ZMVs in an ok state, instead of just ZMV count, w/o needing to join to zip parts to get a real picture of the health of a ZMV.

more storytime loose ends:

• comment DruidVersionZip.create_zip! to explain how writing md5 sidecar files works and why it's useful
• rewrite replication -- write an ADR or other doc to outline our implementation and testing proposal. @justinlittman will try to write up a doc for our consideration, then maybe we can start ticketing that work, and coding against those tickets.

[jmartin-sul]

blocked by #2518

[aaron-collier]

We believe we no longer need this tooling, once verified, this is closable.