Hugging Face Hub Dataset Upload CAS Error

[n-bkoe]

Describe the bug

Experiencing persistent 401 Unauthorized errors when attempting to upload datasets to Hugging Face Hub using the datasets library. The error occurs specifically with the CAS (Content Addressable Storage) service during the upload process. Tried using HF_HUB_DISABLE_XET=1. It seems to work for smaller files.

Exact error message :

Processing Files (0 / 0)                : |                                                                                                                                 |  0.00B /  0.00B             2025-09-10T09:44:35.657565Z ERROR  Fatal Error: "cas::upload_xorb" api call failed (request id 01b[...]XXX): HTTP status client error (401 Unauthorized) for url (https://cas-server.xethub.hf.co/xorb/default/7f3abdc[...]XXX)
    at /home/runner/work/xet-core/xet-core/cas_client/src/retry_wrapper.rs:113

Processing Files (0 / 0)                :   0%|                                                                                                                             |  0.00B /  184kB,  0.00B/s  
New Data Upload                         :   0%|                                                                                                                             |  0.00B /  184kB,  0.00B/s  
❌ Failed to push some_dataset: Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/7f3abdc[...]XXX

Workaround Attempts

1. Disabled XET: Set HF_HUB_DISABLE_XET=1 environment variable
2. Updated hf-xet: Use hf-xet==1.1.9 rather than latest
3. Verified Authentication: Confirmed HF token is valid and has write permissions
4. Tested with Smaller Datasets:
   • 100 samples: ✅ SUCCESS (uploaded successfully)
   • 10,000 samples: ❌ FAILS (401 Unauthorized)

Steps to reproduce the bug

from datasets import Dataset, DatasetDict

# Create dataset (example with 10,000 samples)
dataset = Dataset.from_dict({
    "question": questions,
    "answer": answers,
    # ... other fields
})

# Split into train/test
dataset_dict = dataset.train_test_split(test_size=0.1)

# Upload to Hub
dataset_dict.push_to_hub("Org/some-dataset")

Expected behavior

Expected Behavior

• Dataset should upload successfully to Hugging Face Hub
• Progress bars should complete without authentication errors
• Dataset should be accessible at the specified repository URL

Actual Behavior

• Upload fails consistently with 401 Unauthorized error
• Error occurs specifically during CAS service interaction
• No progress is made on the upload (0% completion)
• Dataset is created on Hugging Face Hub with no data folder

Environment info

• Platform: SageMaker (AWS)
• Python Version: 3.12
• Libraries:
  • datasets library (latest version)
  • hf-xet==1.1.9 (attempted fix)
• Authentication: Hugging Face token configured
• Dataset Size: ~10,000 samples, works for smaller sizes (e.g. 100)

[lhoestq]

cc @jsulz maybe ?

[jsulz]

Curious! I took a look at this and was unable to see why this would be occurring on our side. Tagging in @jgodlew and @bpronan since they might have insights.

@n-bkoe just a few questions if you wouldn't mind:

1. What kind of data are you uploading and what is the difference in file size (in bytes) between 100 and 10,000 samples?
2. Could you provide a specific repository where you encountered this so we could look at to attempt to trace this in our systems?
3. I cannot currently reproduce this, but I'm just trying locally; have you tried to attempt this outside of SageMaker? I'm wondering if there is something unique about that environment causing this.
4. How/where did you set HF_HUB_DISABLE_XET?

[n-bkoe]

Hi, and thank you for your quick answer 🙏

1. Its fairly simple string data, four cols, all string, some long. The script works for data up to 8000 samples long, which is two parquet files totalling 260 kb. It breaks at 10k.
2. Unfortunately, both data and code is private for now !
3. I will try
4. I did it both at CLI level when call my script, and tried inside the python script with os.environ["HF_HUB_DISABLE_XET"] = "1"

The load is also partial, it starts for one file, but does not complete and no data file is pushed.

5. Pushing to Hugging Face Hub...
Pushing dataset to YourOrg/dataset-10000-test_set...
Creating parquet from Arrow format: 100%|███████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1235.07ba/s]
Processing Files (0 / 0)                : |                                                                                        |  0.00B /  0.00B             2025-09-11T15:14:37.018887Z ERROR  Fatal Error: "cas::upload_xorb" api call failed (request id 01K4WNFGSQV1FH8846S0DNS91C): HTTP status client error (401 Unauthorized) for url (https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)
    at /home/runner/work/xet-core/xet-core/cas_client/src/retry_wrapper.rs:113

Processing Files (0 / 0)                :   0%|                                                                                    |  0.00B /  291kB,  0.00B/s  
New Data Upload                         :   0%|                                                                                    |  0.00B /  291kB,  0.00B/s  
❌ Failed to push test_set: Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Uploading the dataset shards:   0%|                                                                                                  | 0/1 [00:00<?, ? shards/s]
Pushing dataset to YourOrg/dataset-10000-indic_test_set...
Creating parquet from Arrow format: 100%|███████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1289.10ba/s]
Processing Files (0 / 0)                : |                                                                                        |  0.00B /  0.00B             2025-09-11T15:14:37.721996Z ERROR  Fatal Error: "cas::upload_xorb" api call failed (request id 01K4WNFHFPJ2DC5D6JC93172H9): HTTP status client error (401 Unauthorized) for url (https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)
    at /home/runner/work/xet-core/xet-core/cas_client/src/retry_wrapper.rs:113

Processing Files (0 / 0)                :   0%|                                                                                    |  0.00B /  277kB,  0.00B/s  
New Data Upload                         :   0%|                                                                                    |  0.00B /  277kB,  0.00B/s  
❌ Failed to push indic_test_set: Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Uploading the dataset shards:   0%|                                                                                                  | 0/1 [00:00<?, ? shards/s]
Pushing dataset to YourOrg/dataset-10000-indic_test_set_combined...
Creating parquet from Arrow format: 100%|███████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 1310.04ba/s]
Processing Files (0 / 0)                : |                                                                                        |  0.00B /  0.00B             2025-09-11T15:14:38.685575Z ERROR  Fatal Error: "cas::upload_xorb" api call failed (request id 01K4WNFJDTVAYM9MFTRDSWKTD6): HTTP status client error (401 Unauthorized) for url (https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)
    at /home/runner/work/xet-core/xet-core/cas_client/src/retry_wrapper.rs:113

Processing Files (0 / 0)                :   0%|                                                                                    |  0.00B /  184kB,  0.00B/s  
New Data Upload                         :   0%|                                                                                    |  0.00B /  184kB,  0.00B/s  
❌ Failed to push indic_test_set_combined: Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Uploading the dataset shards:   0%|                                                                                                  | 0/1 [00:00<?, ? shards/s]

Summary:
  Succeeded: None
  Failed: [('test_set', 'Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'), ('indic_test_set', 'Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'), ('indic_test_set_combined', 'Data processing error: CAS service error : Reqwest Error: HTTP status client error (401 Unauthorized), domain: https://cas-server.xethub.hf.co/xorb/default/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')]
❌ Some datasets failed to upload

[jsulz]

Thanks for following up with more details, @n-bkoe

Could you tell me more about your Sagemaker environment and how you are running this script? In testing with your steps to reproduce in a Sagemaker Jupyter notebook instance (and uploading Parquet datasets with splits of anywhere from a few KBs to a few hundred MBs), I've yet to reproduce this error. This makes me believe that it's either something about the Sagemaker environment or the reproduction steps that I'm not yet emulating.

Concerning the HF_HUB_DISABLE_XET flag, you should ensure it is set before any package imports and in the same process where you are running the script itself. If either aren't true, then this environment variable will not work. You could also explicitly uninstall hf-xet from the environment, although that should be unnecessary with the HF_HUB_DISABLE_XET flag.