Infinite retry loop in upload_large_folder when encountering HTTP 403 storage limit errors

[nyuuzyou]

Describe the bug

Based on pull request #3315, the upload_large_folder function gets stuck in an infinite retry loop when hitting storage limits that return HTTP 403 "storage patterns tripped our internal systems" errors. Unlike regular hf upload that fail with HTTPError, this function continuously retries and rehashes the same files during commit creation, even though the storage limitation can't be resolved through retries.

Reproduction

No response

Logs

Failed to commit: (Request ID: Root=1-68b61518-1be15c4329b3678956ed4992;609aa176-d063-47ed-8f23-d440595643e2)

403 Forbidden: Your storage patterns tripped our internal systems! Please contact us at website@huggingface.co so we can verify your account and unlock more storage for your use-case..
Cannot access content at: https://huggingface.co/api/datasets/nyuuzyou/birdwatch/commit/main.
Make sure your token has the correct permissions.
Failed to commit 1 files at once. Will retry with less files in next batch

System info

- huggingface_hub version: 0.34.4
- Platform: Linux-6.12.38+deb13-amd64-x86_64-with-glibc2.41
- Python version: 3.13.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/nyuuzyou/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: nyuuzyou
- Configured git credential helpers: store
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.7.0
- Jinja2: 3.1.4
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.1.0
- hf_transfer: 0.1.9
- gradio: 5.27.0
- tensorboard: N/A
- numpy: 2.2.6
- pydantic: 2.10.6
- aiohttp: 3.11.12
- hf_xet: 1.1.8
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/nyuuzyou/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/nyuuzyou/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/nyuuzyou/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/nyuuzyou/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: True
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ENABLE_HF_TRANSFER: True
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[Wauplin]

Thanks @nyuuzyou for creating this issue instead (and for the clarifications in #3315 (comment). We'll check what we want to do in this case but agree on the fact that it's a bug :)