[Bug] First container on a node pays the page-cache cost for the image's files, causing unexpected memory overhead and OOM risk

[Fricounet]

Problem Description

This is not a big bug and more of an edge case. But it can be a bit of an unexpected behavior compared to regular OCI images so I wanted to bring it to your attention:

When a container using a newly pulled nydus image starts on a node, it is charged in its memory cgroup for the
kernel page-cache pages of every file it reads (notably its executable). The second and subsequent containers for the same image on the same node read the same pages for free.
This behavior happens because the kernel will reuse the pages that are already in the page cache for the 2nd container while the cgroup accounting will attribute the page memory cost to the first process which faulted this page (1st container).

This asymmetry does not exist for regular overlayfs images: there, containerd's layer-extraction (unpack) reads every file first, so the page-cache pages are charged to containerd's own cgroup, and no container pays.

Note: nydus's fs_prefetch / prefetch_all does not fix this. That prefetch populates nydusd's on-disk blob cache, not the kernel FUSE page cache. The container still faults (and is charged for) the FUSE page-cache pages on first read. (RAFS open() returnsOpenOptions::KEEP_CACHE, i.e. nydus deliberately uses the kernel page cache.)

Expected Behavior

Ideally, we shouldn't have a significant difference in memory usage between 2 containers doing the same thing on the same node.

Actual Behavior

The first pod per image per node will show a higher memory usage than identical later pods, and can be OOM-killed under a tight memory limit purely because it happened to be scheduled first. This is usually unnoticeable for apps that have large multi GB memory limits since loading their binaries in memory should amount to ~1% overhead.

However for applications that have a low memory limit like 50MiB, suddenly having to load a 40MiB go binary will be noticeable and risk the application being OOM killed. And raising the memory limit is not ideal because only the first pod pays this cost. So this potentially create a large increase in memory limits across multiple pods for a single one's sake.

How to reproduce

I've attached a simple reproducer script with the bare-minimal setup to illustrate the issue.
This is mainly just starting a nydus instance locally and attempts to read a file on the image from 2 different cgroups to illustrate the difference in memory accounting.
I can provide a more complete reproducer that uses kind and actual pods if needed as well.

Environment Details

• Nydus-snapshotter version: 0.15.15
• Nydus version: 2.3.0
• Container runtime: containerd
• Operating System: Linux
• Kernel version: 6.8

Additional Information

After digging into various solutions, I'm very conscious that this is not very easy to fix due to the very low-level nature of the issue, so if we feel like it's not worth it, I can accept this and we can close the issue. I mainly want to get other people's opinion on the issue.

For now my the 2 main avenues I've explored to fix this are:

• have a process running in an outside process from the container force the kernel to load the pages in memory. For instance using an annotation on the pod to specify which files to preload -> parse this annotation with a NRI plugin to the nydus-snapshotter -> when the snapshotter Prepare the nydus layer, have the snapshotter read the file before responding to containerd
  • the main concern with this is that we block the start of the container for longer while we are reading the files
• use the fuse notifications system to preemptively store things in the page cache from the nydusd process to load things in the page cache ahead of time
  • I initially thought that we could be reactive there, and have nydus notice when it's being asked to serve a file in the list; but this point is already too late as the application already page faulted at this time
  • we could attempt to preemptively load the rest of the file that hasn't been faulted yet by that point but it risks being racy
  • another idea being to consider those files as the usual prefetch files and store them as they are being prefetched. But it's subject to the same race if the app starts faulting those pages faster than nydusd can do it

So basically, the snapshotter approach makes us sure that the pages will be loaded by the time the container starts but at the cost of a slower startup. The nydusd approach doesn't impact the container startup significantly but at the cost of racing with the container startup causing it to still load some pages in memory.
I would really like to get your opinions on these and if you have other ideas as well!

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Zephyrcf]

@Fricounet I completely agree; attributing memory costs to the infrastructure rather than to the resource-constrained business applications is definitely the right approach.

However, while having the Snapshotter pre-read files correctly shifts the memory accounting, it faces a severe granularity mismatch. If an app only needs 100MB of a 1024MB file to boot, a naive file-level prefetch forces the Snapshotter to read the entire 1024MB before the container even starts, causing massive I/O amplification and unacceptable container startup blocking. Despite these challenges, the first option (Snapshotter pre-read) is still far more reasonable than racing with the app at runtime.

Doing this efficiently, though, would require passing precise chunk-level (offset/length) hints to the snapshotter during the prepare phase, rather than just raw file paths.

[Fricounet]

thanks for your answer @Zephyrcf

Doing this efficiently, though, would require passing precise chunk-level (offset/length) hints to the snapshotter during the prepare phase, rather than just raw file paths.

I agree that that'd be the ideal way to make sure we don't over fetch. But this seems like a poor UX no? Like, as a user, how can I easily know this without doing some heavy prior work to figure out this data? 😓 Especially since this will potentially change each time I recompile my binaries. Do you see a way where we could make this easy on users?

Despite these challenges, the first option (Snapshotter pre-read) is still far more reasonable than racing with the app at runtime.

To be fair, I haven't yet dug too deep in how much we'd race with the app if nydus were to try pushing things in the page cache while the app is starting (this idea "another idea being to consider those files as the usual prefetch files and store them as they are being prefetched"). Maybe the fact the nydus starts some time before the container actually start could help here.

[imeoer]

This is an excellent question! we’ve encountered it in production too, the page cache for FUSE mount point is charged to the first container. This issue has persisted for years, let’s explore how to fix it.

[Zephyrcf]

I agree that that'd be the ideal way to make sure we don't over fetch. But this seems like a poor UX no? Like, as a user, how can I easily know this without doing some heavy prior work to figure out this data? 😓 Especially since this will potentially change each time I recompile my binaries. Do you see a way where we could make this easy on users?

I agree with you; this does indeed degrade the user experience.
Currently, plugins like optimize_nydus_image capture file-level hot lists, which helps build better images. However, we strictly need chunk-level granularity here. Perhaps the upcoming nydus v3, could be leveraged to help us trace and export this precise data.