Uploading large files on huggingface fails.

[EugeoSynthesisThirtyTwo]

Describe the bug

Whether I try to upload a 400GB file or 8 separate 50GB files, it fails each time. I tried many times and the faulty files are always different (it's random). Each attempt is extremely long due to the size of the files, and it's a real pain to have the upload fail.

I fixed it by uploading only two files at a time and pray that it doesn't fail.

Reproduction

Upload a folder contaaining 8 files of 50GB each.

from huggingface_hub.hf_api import HfApi
api = HfApi()

api.upload_folder(
    repo_id="AliceThirty/GLM-4.7-PRISM-Unsloth-GGUF",
    folder_path="GLM-4.7-PRISM/UD-Q8_K_XK",
    path_in_repo="UD-Q8_K_XK",
    token="abcdefgh",
)

Logs

Processing Files (4 / 4)      : 100%|█████████████████████████████████████████████████████|  197GB /  197GB,  0.00B/s
New Data Upload               : |                                                         |  0.00B /  0.00B,  0.00B/s
  ..._K_XL-00004-of-00008.gguf: 100%|█████████████████████████████████████████████████████| 49.8GB / 49.8GB
  ..._K_XL-00002-of-00008.gguf: 100%|█████████████████████████████████████████████████████| 49.0GB / 49.0GB
  ..._K_XL-00003-of-00008.gguf: 100%|█████████████████████████████████████████████████████| 49.0GB / 49.0GB
  ..._K_XL-00001-of-00008.gguf: 100%|█████████████████████████████████████████████████████| 49.5GB / 49.5GB
Traceback (most recent call last):
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\utils\_http.py", line 657, in hf_raise_for_status
    response.raise_for_status()
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\httpx\_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://huggingface.co/api/models/AliceThirty/GLM-4.7-PRISM-Unsloth-GGUF/commit/main'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\me\sc\ia\quantize_and_upload\upload.py", line 64, in <module>
    api.upload_folder(
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\utils\_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\hf_api.py", line 1751, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\hf_api.py", line 5030, in upload_folder
    return self.create_commit(
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\utils\_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\hf_api.py", line 1751, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\hf_api.py", line 4444, in create_commit
    hf_raise_for_status(commit_resp, endpoint_name="commit")
  File "C:\Users\me\miniconda3\envs\hf\Lib\site-packages\huggingface_hub\utils\_http.py", line 716, in hf_raise_for_status
    raise _format(BadRequestError, message, response) from e
huggingface_hub.errors.BadRequestError: (Request ID: Root=1-697a02a0-4ebffb902f212c1b43f96e97;2d1caa2e-4bc4-4f56-ac80-6edc68c3657d)

Bad request for commit endpoint:
Your push was rejected because an LFS pointer pointed to a file that does not exist. For instance, this can happen if you used git push --no-verify to push your changes. Offending file: - UD-Q8_K_XL/GLM-4.7-PRISM-UD-Q8_K_XL-00002-of-00008.gguf

System info

- huggingface_hub version: 1.3.4
- Platform: Windows-10-10.0.26200-SP0
- Python version: 3.11.13
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: C:\Users\me\.cache\huggingface\token
- Has saved token ?: True
- Who am I ?: AliceThirty
- Configured git credential helpers: manager
- Installation method: unknown
- httpx: 0.28.1
- hf_xet: 1.2.0
- gradio: 6.0.1
- tensorboard: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: C:\Users\me\.cache\huggingface\hub
- HF_ASSETS_CACHE: C:\Users\me\.cache\huggingface\assets
- HF_TOKEN_PATH: C:\Users\me\.cache\huggingface\token
- HF_STORED_TOKENS_PATH: C:\Users\me\.cache\huggingface\stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
- HF_XET_HIGH_PERFORMANCE: False

[Wauplin]

@EugeoSynthesisThirtyTwo have you tried to use upload_large_folder instead of upload_folder? I'm not sure what's happening but the later has better retry mechanism built-in. Otherwise, I would suggest to push each file individually using upload_file since there are only 8 of them and that they are very large.

[EugeoSynthesisThirtyTwo]

I will try thank you

[EugeoSynthesisThirtyTwo]

It worked once, but I can only upload at the root of the repo. And then, when I tried again with more files, it uploaded the files forever (It uploaded them once, then twice, then again... starting over every time, without having them appear in the repo) wtf

I tried to switch back to the first method (upload_folder) but with only a single 50GB file inside. And even now, it says this:

Trying again causes the same error.

I have been drowning in problems to upload files for the past days. Even uploading them through the browser didn't work. And I checked that I have enough space remaining on huggingface.

[EugeoSynthesisThirtyTwo]

I can't upload anything anymore. I thought the repo was broken so I created a new one, and even uploading 20GB files don't work (both through browser and through huggingface_hub). I have enough space.