bug: Data retention job fails due to MissingContentMD5 error on MinIO DeleteObjects

[LukasKannengiesser]

Describe the bug

When langfuse is provided with an enterprise key and the data retention is set to a certain number of days, the data retention job fails silently when using the internal MinIO (deployed via the Langfuse Helm chart) as S3-compatible storage. Traces and observations older than the configured retention period are never deleted.

PR fix(storage): disable default S3 checksums for GCS multipart uploads
#13280 (langfuse/langfuse#13280) does not seam to fix this.

This is the error observed in worker logs
{
"msg": "Failed to delete files from S3",
"errorName": "MissingContentMD5",
"errorMessage": "Missing required header for this request: Content-Md5.",
"httpStatus": 400,
"errorCode": "MissingContentMD5"
}

To reproduce

1. Deploy Langfuse via the Helm chart with internal MinIO enabled and provide langfuse enterprise key
2. Configure a data retention policy (e.g. 90 days) in the Langfuse project settings
3. Ingest traces and make sure Postgres contains references to media files such that data retention job actually triggers S3 clean-up
4. Wait for the nightly data retention job to run
5. Observe the worker logs. The job fails with MissingContentMD5 and exits without deleting any traces

Chart version and values.yaml configuration

The error occurred using chart version 1.5.31

Kubernetes Status

No response

Logs

No response

Additional information

No response

Are you interested to contribute a fix for this bug?

No

[Steffen911]

@LukasKannengiesser Are there any logs on the MinIO side that you observe and can share?

[LukasKannengiesser]

Thank you for the fast reply Steffen. The last MinIO log is over a week old, whereas the last failed data retention job failed this night with a log like described in my bug report. We are currently checking our MinIO setup. As soon as I gained any insights I will share them here.

[LukasKannengiesser]

A brief update:
MinIO appears to be properly configured, as media files are being uploaded reliably. However, there are no relevant logs I can share at the moment. The latest available log on the MinIO side is more than a week old, while the most recent worker error related to the failed data retention job only appeared last night.
Please refer to my initial message for the worker error log.

[r4881t]

Additional datapoint from a self-hosted Helm deployment (internal MinIO)

We hit a related failure mode while deleting projects from the Langfuse UI on a Langfuse 3.179.1 self-hosted install using the bundled in-cluster MinIO (chart default S3-compatible storage on a PVC — not external AWS S3).

Symptoms

• Project deletion appeared to run for many hours; org deletion was blocked with "Deletion of your projects is still being processed".
• langfuse-worker logs showed repeated failures on the project-delete BullMQ queue:

  Failed to delete files from S3
  Queue job project-delete ... failed Failed to delete files from S3
  

• Jobs retried on a ~10 minute cadence for several hours, then stopped with jobs sitting in the failed queue (no longer actively processing).
• In Postgres, projects remained soft-deleted (deleted_at set) but were never hard-deleted.

Environment / scale (sanitized)

• Helm chart deployment with internal MinIO, ClickHouse, Postgres, Redis/Valkey.
• Four projects stuck in deleting state.
• MinIO held on the order of tens of thousands of event JSON objects across those projects (largest project ~18k objects; another ~42k).
• ClickHouse blob_storage_file_log still had ~61k rows referencing blobs, while traces / observations / scores were already at 0 for those project IDs (metadata/blob index out of sync with analytics tables).
• Some worker attempts also logged ClickHouse client aborts (socket was closed... Error: aborted) during the same deletion pass.

What worked vs what failed

• Worked: MinIO health checks, listing objects, and single-object deletes via mc using the same credentials the worker uses.
• Failed: the worker's bulk project-delete path that calls S3 DeleteObjects while cleaning ingestion blobs (driven by blob_storage_file_log).

We did not capture the underlying AWS SDK error code in logs (only the wrapped Failed to delete files from S3 message), but the setup matches this issue closely: Helm-bundled MinIO + DeleteObjects during retention/deletion jobs.

Workaround that unblocked us

Manual cleanup:

1. Remove project prefixes from the MinIO langfuse bucket
2. ALTER TABLE ... DELETE in ClickHouse for blob_storage_file_log and related tables
3. Hard-delete soft-deleted project rows in Postgres
4. Clear failed project-delete jobs from Redis/BullMQ

Suggestions / questions for maintainers

1. Log the underlying S3 error (errorCode, httpStatus, errorMessage) on Failed to delete files from S3 — the generic wrapper made root-cause confirmation harder.
2. LANGFUSE_BATCH_PROJECT_CLEANER_ENABLED defaults to false — we enabled it after the incident as a safety net for stuck soft-deletes; consider enabling by default for self-hosted Helm installs using bundled MinIO, or documenting it prominently.
3. For bundled MinIO, is there a recommended MinIO version / server config to avoid DeleteObjects + Content-MD5 issues with the AWS SDK client Langfuse uses?

Happy to provide more detail if useful. Thanks for tracking this.