Loki ingester: replay_memory_ceiling unset (4GB default vs 1Gi limit) makes WAL-replay OOM a self-sustaining crash-loop

[aweingarten]

Summary

The shipped Loki ingester defaults make WAL replay OOM-prone, and because the WAL survives container restarts the resulting crash-loop is self-sustaining — an ingester that OOMs once never recovers without manual intervention.

Two of three ingesters on an LKE-E cluster sat in CrashLoopBackOff for 16 days (3120 and 4498 restarts) before anyone noticed. With replication_factor: 1 there is no quorum fallback, so the distributor returned 500 on every push routed to a dead ingester — log ingestion was silently dropped for the whole period.

Environment

apl-core	v6.0.0
Kubernetes	v1.34.2 (LKE-E)
Loki chart	loki-6.55.0, Loki 3.6.7
Mode	deploymentMode: Distributed, object storage enabled
Resources	shipped defaults, unmodified

Root cause

ingester.wal.replay_memory_ceiling is never set. The rendered loki ConfigMap contains no wal block at all:

ingester:
  chunk_encoding: snappy

So it falls back to Loki's default ceiling of 4GB, while helmfile.d/snippets/defaults.yaml gives the container:

ingester:
  requests: { cpu: 250m, memory: 512Mi }
  limits:   { cpu: 500m, memory: 1Gi }

Loki is therefore permitted to consume 4× the container limit during WAL replay before it begins flushing to relieve memory pressure. Any ingester with a non-trivial WAL is guaranteed to be OOMKilled before replay completes.

Why it never recovers

The ingester data volume is an emptyDir (no volumeClaimTemplates on the StatefulSet). An emptyDir survives container restarts within the same pod, so:

OOM during WAL replay → container restarts in place → same WAL is still there
  → replay OOMs again → never completes → never flushes → WAL never truncates
  → repeat forever

Restarting the container cannot break the loop; only deleting the pod (discarding the emptyDir) does. That is not a discoverable remedy — it looks like data destruction, and with replication_factor: 1 it genuinely is for anything unflushed.

Evidence that this is a limit/ceiling problem, not workload sizing

From the same cluster, the one surviving ingester while carrying all three shares of traffic:

• steady state: ~790Mi — comfortably under the 1Gi limit
• OOMKilled at ~892Mi during WAL replay (exit 137, ~10s in, immediately after msg="recovering from WAL")

Steady-state ingestion fits the limit. Replay does not. That is exactly the gap replay_memory_ceiling exists to close.

The two dead ingesters had accumulated large WALs, so every replay attempt blew the limit; the survivor lived only because it had flushed recently. After deleting the two crash-looping pods, all three came back at 55–137Mi and ingestion recovered immediately — confirming the workload itself fits comfortably.

Impact

• Silent, indefinite loss of log ingestion (POST /loki/api/v1/push → 500)
• replication_factor: 1 means a single dead ingester drops every stream hashed to it, with no fallback
• Self-sustaining: the failure cannot self-heal, and the fix is non-obvious
• Cascade risk: when one ingester dies its streams rehash onto the others, raising their memory and their replay cost — which is a plausible path from one restart to a whole-cluster outage

Suggested fixes

1. Set ingester.wal.replay_memory_ceiling relative to the container limit (Loki's own guidance is a fraction of available memory — commonly ~50%). This is the targeted fix: it makes replay flush early instead of OOMing, so the loop cannot start. With the current defaults, ~512Mi against the 1Gi limit.

2. Raise the ingester memory limit. 1Gi leaves ~10% headroom over observed steady state before replay is even considered. Worth revisiting regardless of (1).

3. Consider persistence for the ingester. With an emptyDir the WAL is neither durable nor inspectable, and replication_factor: 1 means unflushed data has no second copy. If the emptyDir is intentional, that is a reasonable trade — but it deserves to be explicit, because it interacts badly with (1) being unset.

(1) alone would have prevented this outage.

Workaround for anyone hitting this

kubectl -n monitoring delete pod loki-ingester-<n>   # NOT a container restart

Deleting the pod discards the emptyDir and its poisoned WAL, so the ingester starts clean with no replay. Safe while at least one other ingester is healthy. Unflushed data in that WAL is lost — but with replication_factor: 1 and a replay that has already failed thousands of times, it was unrecoverable anyway.