Summary
The reconstruction download buffer is sized from constants, not from the machine. On a large host that leaves most of the link unused; in a small container it is the reported cause of OOM kills. Both ends would be served by deriving the default from available memory, and the client already has everything it needs to do so.
The measurement
192 cores, 1996 GiB RAM, ~20 Gbit/s link, hf-xet 1.5.2, huggingface_hub 0.36.2. A 96.8 GB GGUF repo downloaded in full per arm, each in its own HF_HOME so no run inherits another's dedup state, order interleaved with the baseline re-measured at both ends (drift 6%).
| configuration |
Mbit/s |
vs default |
peak RSS |
| defaults |
6709 |
1.00x |
2.0 G |
+ MIN/MAX_RECONSTRUCTION_FETCH_SIZE at their high-performance values |
6810 |
1.02x |
2.3 G |
+ MIN_PREFETCH_BUFFER raised |
6789 |
1.01x |
2.1 G |
| + the adaptive-concurrency band at its high-performance values |
6984 |
1.04x |
2.2 G |
+ the three DOWNLOAD_BUFFER_* knobs only |
18114 |
2.70x |
14.2 G |
| the whole high-performance preset |
19930 |
2.97x |
20.1 G |
The three DOWNLOAD_BUFFER_* knobs account for essentially the entire difference. Concurrency alone is inside the drift band, which is consistent with #709 and #711 both being abandoned for insufficient impact.
Why the defaults leave that on the table
#637 lowered download_buffer_size from 8gb to 2gb on the basis of benchmarks run on i4i.xlarge and m5d.xlarge, both capped at 10 Gbit/s. That is a sound result inside the range it was measured over. The host above is roughly twice that link speed, and there the 2 GB default costs 2.7x. #666 then made the buffer scale, but with the number of concurrent files rather than with the machine, so the ceiling is still a constant 8gb regardless of whether there are 8 GB or 2 TB of RAM.
The other end
The same constants are too large in a small container. huggingface_hub#3300 (open since 2025-08) reports an OOM kill in a Kubernetes init container with a 1 Gi limit, ~10 GB used against ~200 MB on an older huggingface_hub. A default that is simultaneously too small for a 2 TB host and too large for a 1 Gi container is the signature of a constant that wants to be a function of available memory.
Suggestion
Derive the buffer defaults from usable memory, keeping the current values as the clamps. Something as simple as a fraction of total RAM, floored at today's small-machine values and ceilinged at the existing high-performance numbers, reproduces the current behaviour in the middle of the range and fixes both ends. Two details that mattered when we tried it downstream:
- Read the cgroup limit, not the host total. Inside a container
sysinfo and psutil both report the host's memory, which is exactly how a 1 Gi pod ends up sizing a multi-GB buffer. The v2 limit is at the path named in /proc/self/cgroup rather than at /sys/fs/cgroup itself, and on v1 a nested path (a Slurm step, for instance) reads "unlimited" at the controller root.
- Keep
download_buffer_size + max_concurrent_file_downloads * download_buffer_perfile_size inside download_buffer_limit, so the three numbers describe one allocation rather than three that can overshoot together.
xet_runtime/src/logging/system_monitor.rs already pulls in sysinfo, so the memory figure is available without a new dependency; today it is only used for opt-in logging.
Happy to share the harness or run further arms on this host if that would help, and equally happy to be told the constant is deliberate and the answer is for callers to set the variables themselves. In that case the documentation issue is the one worth fixing, since the variables are currently discoverable only by reading the Rust.
Summary
The reconstruction download buffer is sized from constants, not from the machine. On a large host that leaves most of the link unused; in a small container it is the reported cause of OOM kills. Both ends would be served by deriving the default from available memory, and the client already has everything it needs to do so.
The measurement
192 cores, 1996 GiB RAM, ~20 Gbit/s link, hf-xet 1.5.2,
huggingface_hub0.36.2. A 96.8 GB GGUF repo downloaded in full per arm, each in its ownHF_HOMEso no run inherits another's dedup state, order interleaved with the baseline re-measured at both ends (drift 6%).MIN/MAX_RECONSTRUCTION_FETCH_SIZEat their high-performance valuesMIN_PREFETCH_BUFFERraisedDOWNLOAD_BUFFER_*knobs onlyThe three
DOWNLOAD_BUFFER_*knobs account for essentially the entire difference. Concurrency alone is inside the drift band, which is consistent with #709 and #711 both being abandoned for insufficient impact.Why the defaults leave that on the table
#637 lowered
download_buffer_sizefrom 8gb to 2gb on the basis of benchmarks run oni4i.xlargeandm5d.xlarge, both capped at 10 Gbit/s. That is a sound result inside the range it was measured over. The host above is roughly twice that link speed, and there the 2 GB default costs 2.7x. #666 then made the buffer scale, but with the number of concurrent files rather than with the machine, so the ceiling is still a constant 8gb regardless of whether there are 8 GB or 2 TB of RAM.The other end
The same constants are too large in a small container. huggingface_hub#3300 (open since 2025-08) reports an OOM kill in a Kubernetes init container with a 1 Gi limit, ~10 GB used against ~200 MB on an older
huggingface_hub. A default that is simultaneously too small for a 2 TB host and too large for a 1 Gi container is the signature of a constant that wants to be a function of available memory.Suggestion
Derive the buffer defaults from usable memory, keeping the current values as the clamps. Something as simple as a fraction of total RAM, floored at today's small-machine values and ceilinged at the existing high-performance numbers, reproduces the current behaviour in the middle of the range and fixes both ends. Two details that mattered when we tried it downstream:
sysinfoandpsutilboth report the host's memory, which is exactly how a 1 Gi pod ends up sizing a multi-GB buffer. The v2 limit is at the path named in/proc/self/cgrouprather than at/sys/fs/cgroupitself, and on v1 a nested path (a Slurm step, for instance) reads "unlimited" at the controller root.download_buffer_size + max_concurrent_file_downloads * download_buffer_perfile_sizeinsidedownload_buffer_limit, so the three numbers describe one allocation rather than three that can overshoot together.xet_runtime/src/logging/system_monitor.rsalready pulls insysinfo, so the memory figure is available without a new dependency; today it is only used for opt-in logging.Happy to share the harness or run further arms on this host if that would help, and equally happy to be told the constant is deliberate and the answer is for callers to set the variables themselves. In that case the documentation issue is the one worth fixing, since the variables are currently discoverable only by reading the Rust.