Hugging Face announced that transformers can now efficiently load and run GGUF-formatted, quantized checkpoints on Apple Silicon. The integration is designed so you can pick a GGUF checkpoint from the Hub, pass it to from_pretrained, and start generating locally through the familiar transformers APIs.
What is GGUF and why it matters
GGUF, created by the llama.cpp team, packages model weights and metadata (including tokenizer info and optional chat templates) into a single file. It supports multiple quantization levels so users can trade precision for lower memory usage. Example file sizes for Unsloth’s Qwen3.5-4B variants are:
- BF16 (unquantized): 8.42 GB
- Q6_K: 3.53 GB
- Q5_K_M: 3.14 GB
- Q4_K_M: 2.74 GB
The authors recommend starting with Q4_K_M for local inference and trying Q5_K_M or Q6_K if more memory is available. The Hub’s GGUF documentation lists the available quantization types.
Loading GGUF models in transformers
Required components to get started:
- An Apple Silicon Mac.
- A PyTorch version supported by the published ggml-quantization kernel builds (typically the two latest PyTorch releases).
- The latest transformers from main (until the next release) and a compatible kernels build.
When calling from_pretrained, pass the Hub model_id and the GGUF filename via the gguf_file argument. If the weights remain packed on Metal, transformers will automatically load compatible ggml/Metal layer kernels and use ggml-org/ggml-attn for attention. If a kernel cannot be fetched, transformers falls back to "sdpa" with a warning; you can also force attn_implementation="sdpa". Without a compatible quantization kernel, the loader may dequantize the model and require more memory.
After the gguf_file step, the rest of the workflow uses the standard transformers APIs: tokenization, apply_chat_template, generate, and decoding behave as usual.
Serving and client integration
You can serve the same GGUF checkpoint with transformers serve, which exposes an OpenAI-compatible API. The model argument takes the form <model_id>:<filename>.gguf so a repository containing multiple quantized files can expose a specific variant. The transformers[serving] extras and kernels package are required to run the server.
Clients like Jan or Pi can connect to the local endpoint to provide a conversation interface while transformers runs the model on the Mac.
Benchmarking vs. llama.cpp
The reference for local inference performance is llama.cpp. The article compares three GGUF checkpoints (a small dense model, a larger dense model, and an MoE model). llama.cpp results come from the llama-bench tool (build 5f55650a7, release b10200, Metal backend from ggml 0.18.0) run as llama-bench -m <file> -p 0 -n 128 -r 3, reporting tg128 (token-generation rate over 128 decoded tokens, averaged across three repetitions, prompt processing excluded).
Transformers measurements come from generate producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and include prefill. Benchmarks were executed on a MacBook Pro M2 Max with 32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0, while plugged in.
Results show transformers approaching llama.cpp performance across the tested checkpoints; note that the measurements are not strictly identical because transformers’ measurement includes prefill while llama-bench reports decode-only throughput.
How the performance gains were achieved
Two main areas produced the improvements:
- Reusing ggml’s Metal kernels
The kernels library distributes compatible ggml Metal kernel builds via the Hub and allows transformers to call them from PyTorch. Specialized kernels can operate on packed quantized weights directly and fuse operations to reduce GPU work. Kernel packages mentioned:
- ggml-quantization: reads packed quantized weights for matrix operations, including selected experts in MoE models.
- ggml-norm: fuses normalization ops (e.g., zero-centered RMSNorm used by Qwen models).
- ggml-attn: provides ggml’s Metal flash attention for prompt processing and decoding.
- ggml-gated-delta-net: accelerates the gated delta network in Qwen3.5/3.8 hybrid layers.
- topk: a custom Metal implementation for MoE top-k routing.
These kernels together reduce the GPU work needed per generated token.
- Reducing synchronization in generate
Faster kernels only help if the GPU has sustained work. Two changes in generate improve CPU/GPU overlap and benefit all transformers models:
- Drop an unnecessary attention mask early: when a supported decoder-only input has no padding, the all-ones padding mask is removed at the start so downstream attention code does not repeatedly inspect it.
- Defer the stopping check: generate copies the stopping decision asynchronously and consumes it on the next step, allowing the CPU to continue scheduling while the GPU runs. This approach also applies to streaming tokens and avoids extra steps in the final result.
Kernels lower the cost of operations; generate changes reduce synchronization points so CPU scheduling and GPU execution can overlap more effectively.
Current limitations and next steps
The initial work targets an interactive conversation scenario on Apple Silicon. Current boundaries include:
- The packed inference path is MPS-only for now.
- GGUF import via dequantization remains available as a separate option; format support does not guarantee packed kernels on every device.
- Padding and batching need further work: unpadded inputs benefit from the mask optimization, while padded batches cannot use the same shortcut and may have lower performance. The team plans to extend the approach to generate_batch on MPS.
- Architecture coverage is limited initially to Qwen3.5 dense and MoE architectures and compatible Qwen3.8 checkpoints; support for other architectures will be expanded gradually.
If you have a GGUF checkpoint you want to run in transformers, open an issue with the checkpoint and your use case to help prioritize support.
Benefits for developers
Loading GGUF checkpoints inside transformers enables several developer workflows:
- Experiment in Python and PyTorch: inspect activations with hooks, modify forward passes, or prototype custom layers using familiar tools.
- Evaluate GGUF models using existing transformers evaluation pipelines.
- Validate GGUF conversions by loading original and converted checkpoints to account for quantization error.
- Try new decoding ideas with custom logits processors or your own generation loop.
- Fine-tune starting from a GGUF checkpoint by dequantizing weights (example: GgufConfig(dequantize=True)).
A broader opportunity beyond GGUF
The larger possibility is bringing ggml’s kernel performance to models that llama.cpp does not (yet) support. A kernel operates on tensors and does not require the entire model to originate from a GGUF file. With ggml kernels and quantization available in PyTorch, the same building blocks could accelerate other architectures, research models, or custom variants without reimplementing whole models in llama.cpp. This path could also extend to other modalities (vision, audio, multimodal) by reusing compatible attention, normalization, and matrix-multiplication kernels, though each architecture needs integration and validation. The initial GGUF examples focus on text generation.
Acknowledgements
The work was initiated and reviewed by Arthur Zucker; Cyril Vallez contributed key generate PRs. The team thanks Sayak Paul, the llama.cpp team, and Bertrand Chevalier for help integrating kernels, plus Aritra Roy Gosthipaty and Pedro Cuenca for reviewing the blog post. Lysandre Debut oversaw the project.
In short: GGUF support in transformers narrows the gap between llama.cpp’s inference efficiency and transformers’ model flexibility on Apple Silicon, and it lays groundwork for broader use of ggml kernels across more models and modalities.
