104GB RAM Usage during "stuck" Processing Vulkan Shaders at 100%

[hjpaul7]

Your system information

• Steam client version (build number or date): 1764106918

• Distribution (e.g. Ubuntu): Arch

• Opted into Steam client beta?: Yes

• Have you checked for system updates?: Yes

• Steam Logs: steam-logs.tar.gz

• GPU: Nvidia

Please describe your issue in as much detail as possible:

Steam is spawning multiple processes leading to 104GB RAM usage during "Processing Vulkan Shaders" while being stuck at 100%. I observed this for about 10 minutes and watched it reach 123GB out of 124GB multiple times and froze the system. Processing Vulkan shaders never completed, just sat there at 100% with this insanely high RAM usage.

Steps for reproducing this issue:

1. Launch Spider-Man 2
2. Let Vulkan Shaders process
3. Observe RAM usage and system freeze

[kakra]

I think the problem is that fossilize assigns the work into thread buckets statically and not dynamically, resulting in a progress of almost 100% near the end, waiting potentially on a lot of threads finishing just a few work buckets.

But I don't think that it uses 104 GB of RAM. You cannot just summarize the process RAM usage, that's not how fossilize works: Processes share RAM between each other. If possible, enable or look at the PSS column (proportional set size) which is a much better measure for processes sharing RAM. PSS is roughly calculated by adding the private set size plus and the proportional share of any memory it shares with other processes. Technically, it's not possible to calculate the real and exact memory usage of fossilize, it can only be estimated, and that's what PSS does.

[louzt]

Quick note on #288 from the RFC angle since I happened to be looking at this area.

PR #305 addresses the process orphaning family: workers that survive Steam/Proton parent death and keep compiling in the background. The 104GB RAM case you're hitting looks like a different pattern — compilation state / VM growth where the replayer keeps allocating without releasing, even though the process tree is clean.

A few things that would help triage independently:

• ps -eo pid,ppid,pcpu,pmem,vsz,rss,stat,cmd --sort=-rss | grep fossilize while the behavior is happening — if STAT ends in R for one worker and that worker's VSZ is climbing into the 100GB range, that's the leaking-worker pattern.
• nvidia-smi simultaneously — if VRAM is pinned at high %, the GPU side is also leaking.
• journalctl -k --since "5 min ago" | grep -i oom — OOM events around the same window would confirm the working-set expansion.

If the workers exit cleanly on Steam quit (no STAT / no <defunct> / no PID 1 parent), then PR #305 won't help here. The fix path is more likely the per-worker allocator bound or the cache rotation in ExternalReplayer::Options — a separate report with the ps output above would give the maintainer enough to bisect.

Happy to write up a minimal repro helper if it helps — but my read is that this is a state-explosion bug distinct from the orphan class, and conflating the two would muddy the PR scope.