Diffusers now supports loading Nunchaku-style SVDQuant 4-bit checkpoints natively via a "Nunchaku Lite" integration. Users can call from_pretrained() to load pre-quantized pipelines without a separate inference engine or local CUDA compilation; required CUDA kernels are downloaded from the Hub through the kernels package.
Why this matters
Modern text-to-image transformers in BF16 precision often require 20–30 GB of VRAM, putting them beyond many consumer GPUs. SVDQuant reduces both weights and activations to 4 bits while representing the most problematic parts of weight matrices with a small 16-bit low-rank branch. This lowers peak memory use and can speed the denoising loop compared with weight-only quantizers.
How Nunchaku Lite works
Instead of relying on model-specific fused operators like the original Nunchaku engine, Nunchaku Lite patches compatible torch.nn.Linear modules in a stock Diffusers model at runtime, replacing them with SVDQ or AWQ runtime linear layers before loading the checkpoint. The CUDA kernels used by these runtime layers are provided via the kernels package and downloaded from the Hub when first needed.
Two main kernel families are used:
- svdq_w4a4: 4-bit weights and activations with SVDQuant low-rank correction (available in INT4 and NVFP4 variants). Used for most transformer attention and MLP projections.
- awq_w4a16: 4-bit weights with 16-bit activations, used for adaptive normalization/modulation projections where precision matters.
Because Nunchaku Lite avoids architecture-specific fused kernels, it cannot reach the absolute speedups of the original Nunchaku engine; nevertheless, the lightweight approach typically yields around 30% latency improvement while significantly reducing VRAM usage.
Hardware and precision support
Kernel variants depend on GPU generation and chosen precision:
- svdq_w4a4 (nvfp4): NVIDIA Blackwell (RTX 50 series, RTX PRO 6000, B200)
- svdq_w4a4 (int4): Turing / Ampere / Ada generations (RTX 30 & 40 series, A100, L40S)
- awq_w4a16 (int4): Turing / Ampere / Ada Volta and Hopper GPUs are not currently supported by the 4-bit kernels. The quantizer validates CUDA capability at load time and raises an explicit error on incompatible hardware.
Getting started and an example
After installing diffusers, transformers, accelerate, kernels and bitsandbytes, you can load a pre-quantized pipeline the same way as any Diffusers model with from_pretrained(). An example NVFP4 ERNIE-Image-Turbo Nunchaku Lite checkpoint produces a 1024×1024 image in about 1.7 seconds on an RTX 5090 with peak memory around 12 GB, compared to roughly 24 GB for the BF16 pipeline. NVFP4 checkpoints require Blackwell GPUs; use INT4 variants on older cards.
Benchmarks
Measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024×1024 with rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder:
- BF16 baseline: full pipeline 3.00 s, denoise loop 2.86 s, peak VRAM 31.1 GB (1.0×)
- Nunchaku Lite NVFP4: full pipeline 2.27 s, denoise loop 2.13 s, peak VRAM 20.6 GB (1.35× faster)
- Nunchaku Lite NVFP4 + torch.compile: full pipeline 1.68 s, denoise loop 1.53 s, peak VRAM 20.6 GB (1.8× faster)
- Nunchaku Lite NVFP4 + NF4 text encoder: full pipeline 2.29 s, denoise loop 2.13 s, peak VRAM 16.0 GB (1.35× faster)
These results show up to ~50% peak VRAM reduction while improving latency by about 30%. Much of the remaining overhead is from extra kernel launches; torch.compile can reduce this.
Combining optimizations
- torch.compile: compiling the transformer further improves end-to-end speed (e.g., from ~1.35× to ~1.8× in the benchmark).
- Quantized text encoders: converting a large text encoder (T5, Qwen3) to bitsandbytes NF4 can reduce peak VRAM (in the benchmark, by ~22%).
- Offloading: Diffusers’ existing CPU offload helpers (enable_model_cpu_offload(), enable_sequential_cpu_offload()) remain compatible.
Quantizing and publishing your own model
The diffuse-compressor toolkit provides an end-to-end SVDQuant workflow for Diffusers models: inspect, calibrate, quantize, package a Diffusers pipeline, and push to the Hub. The example in the article walks through quantizing FLUX.2 Klein 4B: inspecting targets, running SVDQuant to produce a safetensors checkpoint (INT4 or NVFP4), converting it into a Diffusers pipeline with a nunchaku_lite entry in transformer/config.json, and verifying outputs locally before calling push_to_hub.
Note: the generic path assumes no structural rewrites are needed. Some models benefit from model-specific structural changes (for example, fusing separate Q, K, V projections into a single to_qkv module). Producing such fused checkpoints requires a model-specific target config during quantization and small runtime adapters to load grouped tensors. The diffuse-compressor examples (and rootonchair/nunchaku-lite) contain patterns and adapters for these cases.
Ready-to-use checkpoints
Available repositories to try immediately include:
- rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder
- rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoder
- OzzyGT/Krea_2_Turbo_nunchaku_lite_nvfp4
- lite-infer (more checkpoints and collections)
Conclusion
SVDQuant kernels used by Nunchaku provide an effective path to run diffusion transformers on consumer hardware. With Diffusers’ Nunchaku Lite integration, pre-quantized checkpoints load via from_pretrained() and diffuse-compressor allows users to quantize and publish new architectures. The W4A4 approach reduces memory usage while improving denoising latency and maintaining image quality close to the BF16 baseline.
Acknowledgements: the integration benefited from contributions by the Diffusers maintainers, the MIT HAN Lab / Nunchaku team (SVDQuant), and other contributors and testers.
