Hugging Face and SkyPilot have integrated support so models, datasets and Buckets stored on the Hugging Face Hub can be mounted directly into SkyPilot jobs running on any cloud or on‑prem cluster using a single hf:// URL and an existing HF_TOKEN. Because Hugging Face Storage charges no egress or CDN fees, reading those assets onto GPUs in different clouds does not incur per‑cloud transfer charges.
What changed
- Hub data (model repos, dataset repos and Buckets) can now be mounted into SkyPilot tasks using store: hf and hf:// source URLs. Mount modes include MOUNT (lazy FUSE), MOUNT_CACHED and COPY.
- SkyPilot can schedule the same job across 20+ clouds, Kubernetes, Slurm and on‑prem, so a run uses whichever reserved or on‑demand GPUs are available on any vendor.
- Hugging Face Storage charges no egress or CDN fees, so a job running on a different cloud reads models and datasets directly from the same bucket without incurring cross‑cloud egress charges or maintaining per‑cloud copies.
- Buckets are built on Xet: content‑defined chunking (about 64 KB chunks) means incremental checkpoints and model variants only store and transfer changed chunks.
- The feature was developed collaboratively; Hugging Face upstreamed hf‑mount FUSE fixes to allow mounting in unprivileged containers, and SkyPilot integrated the backend.
How it looks in practice
SkyPilot already mounts many object stores (S3, GCS, Azure, R2, etc.). Hugging Face Storage now joins as store: hf reachable via the hf:// scheme. Example file_mounts:
-
A Hugging Face Bucket (read‑write) for checkpoints: /checkpoints: source: hf://buckets/my-org/qwen-sft store: hf mode: MOUNT
-
A model repo (read‑only): /base-model: source: hf://Qwen/Qwen3.5-4B store: hf mode: MOUNT
-
A dataset repo pinned to a revision (read‑only): /data: source: hf://datasets/my-org/my-dataset@main store: hf mode: MOUNT
The hf:// scheme covers the lifecycle: read model and dataset from repos, write checkpoints to a Bucket while training, publish the finished model back to a repo, and pull it onto inference servers when serving. For teams already storing artifacts on the Hub, no migration or new storage account is required.
MOUNT vs COPY and file‑system behavior
- MOUNT uses Hugging Face’s hf‑mount FUSE backend: a bucket or repo appears as a local path, and read() calls fetch only the bytes actually requested from the Xet backend. hf‑mount keeps an on‑disk cache so repeated reads stay local.
- For the hf store, MOUNT and MOUNT_CACHED behave the same — both keep the cache. Lazy reads allow processing to begin on a large file before the whole file is downloaded, which keeps GPUs busy sooner and reduces idle billing, especially on the first epoch.
- COPY downloads data up front through huggingface_hub and requires no special FUSE environment.
Authentication uses the existing HF_TOKEN environment variable; pass it to runs with --secret HF_TOKEN. One token works across AWS, GCP, Azure, Nebius, Lambda and Kubernetes, avoiding per‑cloud bucket credentials.
Cost implications
GPU capacity is typically spread across multiple providers. Object stores are regional and per‑cloud, so feeding GPUs or inference servers in another vendor’s data center traditionally means keeping copies in each cloud or paying egress charges (for example, around $0.09/GB out of AWS). That often forces teams to pin runs to the vendor that holds the data.
Hugging Face Storage removes that read‑side cost: there are no egress or CDN fees, and storage pricing is cited at $12–18/TB/month (compared to roughly $23/TB plus egress on AWS S3 in the article’s comparison). The same bucket is reachable from every cluster and reading is free regardless of where the GPUs run. Writes still incur the compute cloud’s normal egress costs, but most AI workflows are read‑heavy (datasets streamed for many epochs, or model weights pulled onto each new node), so removing read egress is the major benefit.
Quick benchmark
The teams ran a small fine‑tune: Qwen/Qwen3.5-4B on the HuggingFaceH4/Multilingual-Thinking dataset using TRL’s SFTTrainer. The model was lazy‑mounted read‑only from its Hub repo and checkpoints were written to a Hugging Face Bucket. The same SkyPilot YAML ran on AWS, GCP and Lambda; SkyPilot placed each job on whichever GPUs were free and all three read and wrote the same bucket.
Measurements reported:
- The model loaded for free on every cloud. Lazy reads fetched only what from_pretrained touched, allowing training readiness in about 30 seconds (up to 500 MB/s). If the model had resided in S3, cross‑cloud reads would have been billed (e.g. $0.09/GB on AWS).
- Checkpoints streamed to the bucket at up to ~170 MB/s; each checkpoint in the example was about 8.43 GB and persisted beyond the GPU instance.
- Per‑cloud checkpoint write speeds observed in the test:
- AWS (us-east-2), GPU L40S: ~168 MB/s
- GCP (us-central1), GPU L4: ~123 MB/s
- Lambda (us-west-3), GPU H100: ~112 MB/s
Xet‑backed deduplication
Hugging Face Buckets use Xet’s content‑defined chunking so files split into ~64 KB chunks and each unique chunk is stored once. Benefits include:
- Incremental and adapter checkpoints upload only changed chunks.
- Model variants that share a base reuse shared chunks instead of duplicating storage.
- Appending to datasets (e.g., Parquet row appends) only transfers new data: in a test, appending 10K rows to a 100K‑row table moved ~10 MB rather than the full ~106 MB.
- Re‑uploads skip already stored chunks: re‑uploading an 8.43 GB blob already in the bucket took ~8 seconds versus 24 seconds for the first upload because only chunk hashes moved.
Actual savings depend on artifact overlap, but deduplication is automatic: you write as usual and only new chunks are transferred.
Getting started
- pip install "skypilot[huggingface]"
- hf auth login or export HF_TOKEN=<your-token>
- Add an hf:// mount to any SkyPilot task and launch.
Note: MOUNT requires a base image with glibc 2.34+ and /dev/fuse.
Collaboration and open source
The initial store: hf support began as a contribution from Nikhil Jha; the Hugging Face team advanced it and upstreamed hf‑mount fixes for unprivileged containers, and the SkyPilot team integrated it into their storage backend. SkyPilot, hf‑mount and the huggingface_hub client are open source. Documentation and community resources include SkyPilot storage docs, the Hugging Face Storage Buckets guide, hf‑mount materials and the SkyPilot Slack community.
