Meta releases Muse Spark Open: an 816B-parameter multimodal Mixture-of-Experts model

Meta has published Muse Spark Open, an open-weight multimodal language model in the Muse family. The model uses a Mixture-of-Experts (MoE) architecture and contains roughly 816 billion parameters in total; about 42 billion parameters are active per token during computation.

Capabilities and intent

Muse Spark Open accepts text, images, audio and video as inputs and produces text outputs. Meta says the model is optimized for agentic coding workflows — reading codebases, writing and debugging code, and calling tools — and is designed to plan, act, observe and self-correct over long tasks, including web browsing with image context.

Key technical highlights

• Total parameters: ~816B
• Active parameters per token: ~42B
• MoE experts per layer: 256; experts routed per token: 8
• Inputs: text, images, audio, video
• Output: text
• Context window: up to 1,048,576 tokens (1M)
• Checkpoint formats: BF16 and NVFP4
• Checkpoint names: Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4

Why the MoE design matters here

The language model has 62 layers and a hidden dimension of 8192. Unlike Muse Glimmer (a smaller dense model released earlier in the Muse series), Muse Spark Open replaces each layer's dense feed-forward network with an MoE block. Each MoE block includes 256 experts and the router selects 8 experts per token. Because only 8 experts are executed per token, runtime behaves closer to a ~42B model in compute while keeping all 256 experts in memory, which requires memory comparable to an 816B model. The BF16 checkpoint is approximately 1.5 TB in safetensors format. The NVFP4 checkpoint stores parameters in 4-bit precision, targeting NVIDIA Blackwell GPUs to reduce memory and enable faster single-node runs.

Perception and audio towers

• Perception encoder (images/videos): 1.8B parameters, 50 layers, hidden dimension 1536, 16 attention heads. Images are split into 14×14 patches and transformed to 4096-dimensional features; positional signals use learned absolute embeddings plus 2D RoPE. Some layers use full attention every fourth layer and at the final layer; other layers use sparse attention.

• Audio encoder: converts raw waveform into audio tokens at one token per 80 ms (12.5 tokens/sec). The encoder is a convolutional pyramid with patch embedders and 1D conv blocks; final processing uses RMSNorm and linear projection. Video frames use the same perception encoder as images.

Long context and KV cache

The model is configured with max_position_embeddings set to 1,048,576, so text, image, video and audio tokens share the same 1M-token window. Grouped-query attention helps keep the KV cache smaller: each token adds about 126,976 values to the cache, which in BF16 is roughly 250 KB per token — a full 1M-token context would consume around 266 GB for the KV cache alone.

Inference support and runtimes

Meta provides reference implementations and integration examples across multiple runtimes:

• Transformers (Hugging Face): MuseSparkForConditionalGeneration and MuseSparkProcessor, with DistributedConfig parameters tp_size and ep_size to configure tensor and expert parallelism.
• llama.cpp: prebuilt binaries and several quantized variants (Q2_K, Q4_K_S, Q_8). Tests on Apple M3 Ultra (512 GB) showed a calibrated 2-bit quant running at ~18 tokens/s, with space left for the KV cache.
• SGLang: NVFP4 reference setup running across four GB300 GPUs; the checkpoint includes the chat template and launcher configs.
• vLLM: NVFP4 plugin for Blackwell GPUs; reference configuration supports a 128K token serving context and enables expert parallelism.
• MLX / oMLX: oQ quants in 3-bit and 4-bit formats; an oQ3e test on M3 Ultra used ~344 GB memory and reported prefill/generation throughput numbers for a sample prompt.

Meta also documents how to deploy Muse Spark Open on Hugging Face Inference Endpoints. The NVFP4 variant can be deployed on endpoint instances such as 8×NVIDIA RTX Pro 6000 (98 GB) or 8×NVIDIA H200 (141 GB) depending on desired context length.

Multimodal usage patterns and tool calling

The Muse Spark processor expands image markers to patch tokens, inserts audio tokens between audio boundary markers, aligns video frames with timestamps, and supplies masks to indicate real audio samples versus padding. The chat template supports a reasoning_effort lever (low/medium/high/xhigh mapped to numeric strengths 16/64/256/512). The model cannot disable reasoning entirely, so setting reasoning to "low" is suggested for simpler tasks to conserve tokens.

Muse Spark Open supports multimodal tool calling. Tools are declared to apply_chat_template and the model outputs channel-based messages (Harmony-style) that indicate whether a fragment is internal reasoning (self), commentary, a tool call, or the final user-directed response.

Fine-tuning with TRL and reported results

Meta demonstrates fine-tuning the full 816B checkpoint using TRL's SFTTrainer with LoRA adapters on attention projections while keeping specialists frozen. The example splits the model across 64 GPUs (8 nodes × 8 H100 GPUs). In the run described, they trained for 500 steps (~2 hours on 64 H100s) at around 8,000 tokens/s. Mean token accuracy increased from 72% to 84% and loss decreased from 1.26 to 0.57.

Demos and example applications

Meta and contributors published several demos to highlight Muse Spark Open's agentic and multimodal capabilities:

• 3D asset generator: a loop using quantized Muse Spark on llama.cpp with a Pi coding agent to iteratively generate and refine Three.js assets using screenshots for visual feedback.
• Browser agent demos with OpenEnv and Hugging Face Sandbox: an agent that acts based on system prompts, URLs and screenshots (no DOM access) to demonstrate multimodal web tasks.
• Multimodal shopping agent: audio input and image/video feedback combined with agentic browsing to locate products.

All demos are provided as runnable examples and reference code in the associated repositories.

Conclusion

Muse Spark Open is Meta's full-size open-weight multimodal MoE model in the Muse series. It is available in BF16 and NVFP4 checkpoints, plus multiple community and reference quantizations and integrations (llama.cpp, vLLM, SGLang, MLX). Meta provides recipes for inference, deployment and fine-tuning and invites developers and researchers to try the model and share results in the repository discussions.

(Note: example chat templates in the documentation include a "Knowledge cutoff: 2026-01-04" line.)