with_high_performance() documents one of the twelve values it sets, and none of the memory cost

[danielhanchen]

What the doc comment says

xet_runtime/src/config/xet_config.rs, on with_high_performance() (checked on main, workspace version 1.5.4, and identical in 1.5.2):

/// This method sets the following values to their high performance defaults:
/// - data.max_concurrent_file_ingestion: 8 -> 100

What the method actually does

self.data.max_concurrent_file_ingestion = 100;
self.client.ac_max_upload_concurrency = 124;
self.client.ac_max_download_concurrency = 124;
self.client.ac_min_upload_concurrency = 4;
self.client.ac_min_download_concurrency = 4;
self.client.ac_initial_upload_concurrency = 16;
self.client.ac_initial_download_concurrency = 16;
self.reconstruction.min_reconstruction_fetch_size = ByteSize::from("1gb");   // default 256mb
self.reconstruction.max_reconstruction_fetch_size = ByteSize::from("16gb");  // default 8gb
self.reconstruction.download_buffer_size         = ByteSize::from("16gb");   // default 2gb
self.reconstruction.download_buffer_perfile_size = ByteSize::from("2gb");    // default 512mb
self.reconstruction.download_buffer_limit        = ByteSize::from("64gb");   // default 8gb

Twelve values, of which the documented one is the only upload-ingestion field. Read literally, the comment says high performance mode is an upload-concurrency flag. In practice it is mostly an 8x download-buffer preset.

Why it matters beyond tidiness

Two consequences we hit directly:

1. The memory cost is invisible. download_buffer_size 16gb and download_buffer_limit 64gb are not mentioned in this comment, nor in the huggingface_hub docs, which describe the flag as maximising "network and disk resources". Measuring a 96.8 GB download on a 192-core / 1996 GiB host, peak RSS was 18.0 GB with the flag against 4.1 GB without, which matches download_buffer_size almost exactly. Someone enabling this on a memory-constrained machine on the strength of the current documentation has no way to anticipate that.

2. It is read as all-or-nothing. Because the preset is documented as a mode rather than as a set of ordinary knobs, the natural reading is that you either take it or you do not. Every value it sets has an HF_XET_* variable of its own, which means a caller can adopt the parts their machine can afford. That is a much better answer for a 16 GB laptop than either extreme, and the current comment gives no hint that it is available.

Suggestion

List all twelve fields with their default and high-performance values, and note the resulting worst-case buffer allocation (download_buffer_size + max_concurrent_file_downloads * download_buffer_perfile_size, capped at download_buffer_limit) so the memory implication is visible at the point where someone decides to enable it.

Two smaller things noticed while checking, happy to split them out if you would prefer separate issues:

• with_high_performance() does not set data.max_concurrent_file_downloads (it stays at 8, only the ingestion twin goes to 100). Raising that variable by hand is therefore going outside the preset, which is not obvious from the name symmetry.
• The huggingface_hub environment-variable docs document HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY, which does not exist in this tree. The field is reconstruction.use_vectored_write, variable HF_XET_RECONSTRUCTION_USE_VECTORED_WRITE.

Measurements above come from a full A/B on the same repo, each run in its own HF_HOME, order interleaved, with the reference configuration re-measured at the start, middle and end to bound CDN drift. Happy to share the raw numbers if useful.