An experimental Transformers implementation has run the Qwen3.8-Flash-Next large language model on a MacBook M5 with 32 GB of unified memory while keeping much of the model's expert weights on an SSD. In a short generation test, the optimized version produced 13.9 tokens per second, about 6% behind the specialized Strata inference engine.
The experiment's author was looking for a local alternative to reduce reliance on online models. Although the author's desktop has two NVIDIA RTX 4090 GPUs, the aim was also to use a larger model on a portable computer. An earlier report found that Strata could run Qwen3.8-Flash-Next at 13.5 tokens per second on a 32 GB Mac M4. That was too slow for the author's coding-agent workflow, but it showed that SSD-based execution was worth investigating.
Why moving experts matters
Mixture-of-experts, or MoE, models activate only a subset of their components, known as experts, for each token. Experts can nevertheless account for more than 90% of a model's weights. Strata keeps frequently used experts in a cache and places the rest in system memory or on disk, allowing a model to run on a device with much less memory than the full model requires.
According to the author, existing approaches make different trade-offs. llama.cpp can store and compute experts on the CPU, which works well when they fit in RAM but can overwhelm the page cache when they do not. vLLM can read selected weights from pinned system memory over PCIe, but that approach does not cache experts or provide a disk tier. The Transformers method based on Accelerate moves an entire layer's expert bank even when only a few experts are needed; on a Mac, moving weights to the CPU alone does not free memory because the CPU and GPU share it.
In a test of Qwen3.6-35B-A3B on a 24 GB A10G accelerator, Strata generated 82–88 tokens per second in Q8_0 format, while llama.cpp reached 63–64 with Q8_0. The best reported vLLM configuration, using FP8 and offloading only experts, reached 20.1 tokens per second; Transformers with Accelerate reached 0.16. The measurements used different formats and settings, and Harry Mellor reran some vLLM tests with improved flags.
A shared cache for selected experts
The developer first tested OLMoE-1B-7B, which fits in memory. The initial implementation loaded experts from disk but ran them individually in a Python loop. Even with a fully populated cache, it achieved only 10.1 tokens per second, compared with about 32 for standard Transformers.
The key change was a preallocated cache shared across layers. By mapping expert identifiers to cache slots, the existing grouped_mm operation could be used without a custom expert matrix multiplication. In a later comparison, the approach reached 34.6 tokens per second against 38.1 for the standard implementation.
The general implementation was submitted as huggingface/transformers#50717. When needed, the system reads selected experts using eight I/O threads and runs the routers for the next two layers early to predict which experts to prefetch. The API remains under discussion with Transformers maintainers. The author says the code was built with AI assistance, while the author chose what to try and which approaches to retain.
The limits of a 32 GB Mac
Each of Qwen3.8-Flash-Next's 48 layers contains 512 experts, ten of which are active at a time; the experts occupy 121 GB in FP8 format. The model also has a 51 GB n-gram embedding table that needed its own disk access path.
For the experiment, the experts were quantized to 3 bits, occupying 49 GB on disk, with 12 GiB held in the device cache. Dense layers were stored at 8 bits, using about 5.4 GB and increasing perplexity by 0.2% compared with BF16. The n-gram table stayed on disk, with only the required rows read as needed.
Conversion took about five hours. After loading, the model used roughly 18 GiB of memory, peaking at 25.4 GiB while processing a 6,000-token prompt. According to the author, that was close to the machine's limit: the browser had to be closed to prevent macOS from swapping.
The first version, which also used model-specific Metal kernels, generated about 10 tokens per second. On the same M5 with identical prompts, Strata reached 14.8; for prefill of a 4,000-token prompt, it reached 220 tokens per second against 68 for the Transformers version. The author also tried vllm-metal, but it could not run this model on the 32 GB machine: although it streams experts from disk, it keeps the n-gram tables in memory.
Closing in on Strata
The developer improved speed by, among other changes, committing GPU work earlier, adding targeted Metal kernels, using Strata's smaller GGUF weights and reading the next layer's experts while the current layer was computing. Changes to attention and the KV cache accelerated long-document processing. For cache misses, prefetching one likely expert per layer proved useful.
In the final measurements on the same machine and prompts:
- For a 200-token generation, Transformers reached 13.9 tokens per second and Strata reached 14.8.
- Prefill of a 4,000-token prompt ran at 228 and 220 tokens per second, respectively.
- Generation after a 6,000-token document reached 12.4 and 13.7 tokens per second, respectively.
The tests used Strata 0.1.60 with IQ3_XXS experts. Both systems used greedy decoding, with thinking, MTP and speculative drafts disabled; prefill ran in chunks of 1,024 tokens.
The author attributes much of the remaining gap to the host and GPU waiting on one another at each layer. Strata can queue the work for an entire token in advance, but PyTorch does not currently expose the necessary control. The experiment has no C++ extension: it uses Python code, Metal kernels compiled at runtime and prebuilt kernels.
Although the result approached Strata's, the author says it remains too slow for everyday coding-agent work, which continues on the desktop. The project nevertheless shows that streaming selected experts from disk through a bounded cache can allow Transformers to handle larger MoE models beyond the available memory.



