WAL replay backpressure failures are reported as WAL corruption

[jnewbigin]

Describe the bug

A WAL replay failure caused by memory backpressure is reported as WAL corruption. It increments loki_ingester_wal_corruptions_total and logs a message that explicitly tells the operator no action is needed when in fact the ingester has come up with an incomplete WAL and does need attention.

To Reproduce

Any condition where a replay-triggered flush cannot bring in-memory bytes below 90% of replay_memory_ceiling: slow or unavailable object storage, chunk_retain_period holding freshly flushed chunks, or immediate flush ops failing and being re-enqueued.

1. Start an ingester with WAL replay and a replay_memory_ceiling that the workload cannot get under.

2. replayController.WithBackPressure gives up and returns an error (pkg/ingester/replay_controller.go:141):

   WAL replay flush made no progress: <N> in use, ceiling <M>; cannot recover
   

3. That error propagates out through ingesterRecoverer.Series/Store (pkg/ingester/recovery.go:112, :191) and becomes the return value of RecoverCheckpoint/RecoverWAL.

4. Ingester.starting() handles it as corruption (pkg/ingester/ingester.go:582 and :604):

   i.metrics.walCorruptionsTotal.WithLabelValues(walTypeSegment).Inc()
   level.Error(i.logger).Log(
       "msg",
       "Recovered from WAL segments with errors. Some streams and/or entries were likely not recovered due to WAL segment file corruptions (or WAL file deletions while Loki is running). No administrator action is needed and data loss is only a possibility if more than (replication factor / 2 + 1) ingesters suffer from this.",
       ...
   )

5. Startup then continues — endReplay(), i.wal.Start() — and the ingester goes ACTIVE having skipped WAL entries.

Expected behavior

A backpressure failure and a corrupt WAL segment are different problems with different remedies, so they should not share a metric or a message.

• loki_ingester_wal_corruptions_total ("Total number of WAL corruptions encountered.") should count corruption only. As it stands, any alert on it fires for a memory/storage condition, and the type label offers no way to tell the two apart.
• The log text is actively misleading here. "No administrator action is needed" is true for a torn WAL segment; for this failure an administrator very much does need to act — raise replay_memory_ceiling, or fix whatever is stopping flushes from draining.

A distinct metric (or an extra type value such as backpressure) plus its own log message would make the two cases separable.

Environment:

• Not environment specific — the code path is the same on main, release-3.7.x and release-3.6.x.

Additional context

Worth noting the practical impact for anyone upgrading onto the fix: the old behaviour was a loud crash loop (the ingester never became ready and the startup probe killed it), so an operator could hardly miss it. The new behaviour is quiet, and the one signal that would surface it points at the wrong cause. We hit the original hang with a customer on GEL 3.6.10, which is what surfaced this.