Memory leak via kernel slab growth causes OOMKill under high-target load

[aahmadzada-png]

What did you do?

Ran ipmi_exporter v1.10.1 in Kubernetes with 500 targets, 4 collectors
(ipmi, dcmi, chassis, bmc), scrape_interval=15s, container memory limit 500Mi.

What did you expect to see?

Stable memory usage, or at least a plateau well below the limit.

What did you see instead?

Linear memory growth over ~15 hours until OOMKill. Reproducible — both
replicas started simultaneously followed identical trajectories and
OOMKilled within the same second.

• Pod A: 37 MiB → 461 MiB → OOMKill → restart
• Pod B: 22 MiB → 452 MiB → OOMKill → restart
• Growth rate: ~30 MiB/hour

Diagnosis

The growth is not in Go memory. Go metrics are flat:

• go_memstats_heap_inuse_bytes: 12 MB
• go_memstats_sys_bytes: 49 MB

The growth is in kernel slab (dentry/inode cache). From
/sys/fs/cgroup/memory.stat at ~223 MiB working set:

anon:               19 MB   (process memory, flat)
file:                0.9 MB
kernel_stack:        2 MB
slab_reclaimable:  192 MB   ← the growth
slab_unreclaimable:  1.5 MB

Other observations:

• Only 5–6 live freeipmi subprocesses at any moment (no subprocess leak)
• /tmp contains 1 file (no temp-file leak)
• GOMEMLIMIT=156MiB and GOGC=80 had no effect (confirms it isn't Go heap)

Hypothesized cause

Each scrape creates a unique /tmp/ipmi_exporter-<hash> config file and
forks freeipmi binaries (ipmi-sensors, ipmi-dcmi, ipmi-chassis,
bmc-info), each loading ~15 shared libraries. At 500 targets × 4
collectors × 15s interval, that's ~130 fork+exec per second, generating
thousands of dentry/inode allocations per second. Kernel caches them
as slab_reclaimable. Under sustained churn near the memory limit,
reclaim can't keep pace → OOMKill.

Suggested fix (AI)

Avoid per-scrape temp file churn in
freeipmi.go:

1. Use memfd_create(2) for the config — no dentry/inode, lives in memory.
2. Or generate the config file once per target at startup and reuse it.

Environment

• ipmi_exporter: v1.10.1
• freeipmi:
• Kubernetes:
• Container runtime: containerd
• cgroup: v2
• Kernel: 6.8.0-100-generic (Alpine Linux v3.22)
• Target count: 500
• Scrape interval: 15s
• Collectors: ipmi, dcmi, chassis, bmc

Note: we also see this reported which possibly related

[rtreffer]

Wait, you budget 1MB/target and this runs out? I would recommend to reconsider this.
At 130 expected forks/s and a concurrency of 6 you would expect each IPMI fork+exec call to take <50ms (little's law). A simple network issue should balloon the process count and memory usage.

I find that way more likely than a memory leak as described because reclaimable memory should be reclaimable.
This means should not cause this.
See kernel cgroup v2 docs

slab_reclaimable
Part of “slab” that might be reclaimed, such as dentries and inodes.

Memory should be used, so there is no issue with reclaimable growing.
This is equivalent to seeing cached memory grow on a linux box.

OOM can happen independent of your memory limit. A memory limit is a guaranteed OOM, but an OOM does not prove that you ran over your memory limit.

• dmesg should have details on any OOM
• memory requests is relevant, too, to avoid oversubscription
• k8s does not set memory.min - this is locked behind memory QoS and memory QoS is unusable due to issues with memory.high which gets set by the same feature flag :-/

In particular I would like to see

• memory requests and allocation on the node
• oom-kill:constraint=CONSTRAINT_MEMCG for the cgroup
• actually killed processes
• memory.peak to compare micro-spikes vs. steady state
• memory.events (especially max and oom)

Other things that impact an OOM

• cgroup memory.oom.group
• vm.overcommit_memory
• vm.oom_kill_allocating_task

Overall missing:

• proof that the OOM cause is within the container
• proof that it is not the ipmi processes

[aahmadzada-png]

@rtreffer Thanks for the detailed feedback, that's fair.

You're right that slab_reclaimable growing isn't inherently a leak — I oversimplified. I'll try to gather the proof points you listed before making stronger claims:

One thing worth sharing now: we have another 500-target setup with a 4 GiB memory limit for ipmi-exporter. The graph shows the same steady growth pattern, and in this case the kernel does reclaim before hitting the limit — no OOM. But the growth itself is consistent.

Is that expected? And if so, how are operators meant to size the memory limit? At startup the container uses very little, but memory climbs steadily over time until it's close to whatever limit we set.

[aahmadzada-png]

@rtreffer I have to admit OOM depends on cgroup limits and isn't the core issue. Let me restate.

What we observe is unbounded slab_reclaimable growth while Go heap and anon memory stay flat. We suspect the per-scrape /tmp/ipmi_exporter- config file creation in freeipmi.go is the driver: at 500 targets × 4 collectors / 15s that's ~130 create/unlink per second, producing dentry and inode slab entries faster than the kernel reclaims them. The pattern fits — flat userspace, growing kernel slab.

Something like memfd_create(2) or reusing a per-target config would eliminate the /tmp round-trip entirely maybe and fix the memory grow issue. What you think?

[bitfehler]

Note: memfd is not an option, as the FreeIPMI tools expect an actual filename.

Right now, the config files are named pipes (so can be read only once) with random names. Yes, that might some pressure on the dentry cache, but we've run this against > 2000 hosts without issues. And it makes it certainly not impossible, but pretty hard, and notably detectable if a foreign process grabs the password out of these.

However, as an opt-in thing, I would not be opposed to pre-generating (and keeping) the config files. Maybe a command line switch that takes a directory that is then used for this.

[rtreffer]

@rtreffer I have to admit OOM depends on cgroup limits and isn't the core issue. Let me restate.

[...] producing dentry and inode slab entries faster than the kernel reclaims them. [...]

I know your AI told you this, but this is not how this should work. The optimal reclaim speed is 0.
The task of a kernel is to let your program run as fast as possible. Why should the kernel randomly interrupt you to reclaim memory unless needed? Reclaim should happen only if needed.

Check e.g. this LWN article about problems with reclaim (emphasis mine)

Slab reclaim is handled with shrinker callbacks; when the system needs more memory, the shrinkers are called and asked to free some objects.

Why would you expect that the system needs that memory?
What you see is intended behavior: memory grows until a cleanup is needed. The kernel does keep up with the cleanup.
A failure to keep up would be an OOM, not low free memory.

The chain for an OOM is:

1. system needs memory
2. system can't free memory
3. OOM

If you don't see (3) and you see a growth then it gets caught at step (2). Not seeing an OOM but an (initial) steady growth means (2) is working. memory.events should show you that, too.

I'll leave it at that. I don't see any issue after the OOM is resolved.