Reflection, a US research lab, has released Beam on Hugging Face under the Apache 2.0 license. Beam is a mixture‑of‑experts (MoE) language model totaling 501 billion parameters, but only about 23 billion parameters are active per token. The model accepts text input, produces text output, and supports a 1,000,000‑token context window.
Target use cases
Beam is aimed at coding and agentic workflows, and at teams that want to run a frontier model on their own hardware, infrastructure, or cloud and customize it. It ships with a Rust‑based open‑source harness for inference and day‑0 integrations into common libraries.
Model architecture and attention
The decoder has 52 layers with hidden size 5,120. Most layers use sliding‑window attention over 512 prior tokens; every sixth layer and the final layer attend to the full context. This results in 43 sliding‑window layers and 9 global layers.
- Sliding‑window layers use RoPE positional encodings.
- Global layers omit positional encodings and instead scale each query by a slowly growing scalar (1 + (pos + 1)/2048) ** 0.1, a mechanism Reflection calls scalable‑softmax query scaling to preserve softmax sharpness as context increases. Attention uses grouped‑query attention with 48 query heads and 8 key‑value heads; queries and keys are RMSNorm‑ed. A learned sigmoid gate scales each channel of every head before the output projection.
Mixture‑of‑Experts details
The first three layers are dense SwiGLU MLPs; the remaining 49 layers are MoE blocks with 256 experts each. Each expert is a SwiGLU MLP with intermediate size 2,560 (roughly 39M parameters). Experts account for 98.4% of the model parameters. The router implements a DeepSeek‑V3 group‑limited scheme: it scores all 256 experts with a sigmoid, divides them into 8 groups of 32, keeps the four best groups (by the sum of each group’s top two scores), and then selects the top 8 experts within those groups. Expert outputs are combined with normalized router weights and scaled by 2.5. Because only 8 experts run per token, each token is computed with about 23B parameters. All 256 experts remain resident in memory, so deployment requires memory comparable to a 501B model, while runtime throughput is closer to a 23B model.
Residual stream and numerical decisions
Beam weights residual additions so the residual stream’s scale remains constant across depth: the input embedding is step 1; at step k a sublayer output is added with weight 1/sqrt(k) and the existing stream is kept with weight sqrt((k-1)/k). The squares of these two weights sum to one, so after all additions each sublayer output and the input embedding contribute equally to the final hidden state. Several operations are explicitly computed in FP32 (router, SwiGLU product, norms, RoPE, query scaling, softcap) and final logits pass through a tanh softcap at ±30.
Long context handling and cache growth
Beam’s hybrid attention causes much slower KV cache growth than a fully global design. Each layer caches 2 × 8 KV heads × 128 dims = 2,048 values per token. If all 52 layers were global the cache would grow by ~208 KiB per token; with Beam’s 9 global layers and 43 sliding windows, the cache grows by roughly 36 KiB per token. Reflection provides example cache sizes in BF16 with the transformers dynamic cache: 32K tokens → ~6.5 GiB total / 1.2 GiB effective; 128K tokens → ~26 GiB / 4.6 GiB; full 1M context → ~208 GiB / ~36.08 GiB.
Checkpoints and precision formats
Beam is available in BF16, FP8 and NVFP4 checkpoints. The BF16 safetensors checkpoint is roughly 1 TB. NVFP4 packs most weights into 4‑bit formats for NVIDIA Blackwell GPUs, shrinking the footprint to ~280 GB while keeping some weights in BF16 for accuracy. FP8 checkpoints are recommended for Hopper and Ampere GPUs.
Benchmarks (as reported by Reflection)
Reflection published a set of benchmark numbers across precisions for multiple categories (agentic coding, reasoning, tool calling, general capabilities). Examples include:
- AIME 2026: ~97.5–98.0% (different precisions)
- LiveCodeBench: ~87.4–88.3%
- SWE Bench Pro v2‑Hard: 76.5–78.6%
- SWEBench Verified: ~80.2–81.6% These figures vary slightly by checkpoint precision (BF16, FP8, NVFP4) and task.
Hardware recommendations
Reflection suggests NVFP4 for practical serving because it keeps most BF16 quality while reducing size. NVFP4 makes the BF16 1 TB weights ~280 GB; Blackwell GPUs run NVFP4 natively, older GPUs like H200 do not. Example deployment recommendations:
- 2× B200 (180 GB each) can hold weights but leave little room for activations/KV cache; 4× B200 (720 GB) is safer for long contexts and concurrency.
- 8× B200, 2× B300 (288 GB each), or configurations with RTX Pro 6000 are also possible. If Blackwell hardware isn’t available, Hugging Face Inference Providers (Baseten, Fireworks, Together, etc.) can be used.
Day‑0 inference support
Beam ships with immediate support for transformers, vLLM and SGLang. Examples provided by Reflection include transformers pipelines and AutoModel/AutoTokenizer usage, vLLM server commands (including DFlash2 speculative decoding with a drafter model), and SGLang launch examples for NVFP4 on Blackwell and FP8 on Hopper. The tokenizer includes a response template and parser that outputs structured messages with separate fields for reasoning, answers, and any tool calls.
Tool calling and multi‑node setups
Beam writes tool call arguments in an XML‑style grammar; parse_response can accept tool schemas to cast arguments to their declared types. The model’s configuration supports expert‑parallel layouts where each expert is placed on a device and attention/dense/embeddings are replicated. For distributed loading the number of devices must divide the number of experts (256). Reflection provides example DistributedConfig usage and torchrun launch guidance.
Fine‑tuning and reinforcement learning recipes
Reflection published recipes for supervised fine‑tuning (SFT with TRL and LoRA/PEFT adapters) and for RL training using Miles with GRPO. Highlights:
- LoRA SFT: a LoRA adapter training run on Multilingual‑Thinking enabled Beam to follow requested reasoning language on 16/16 held‑out prompts vs 3/16 for the base model; a LoRA SFT fit on a single 8× B300 node achieved ~19k tokens/s in the example setup.
- GRPO with Miles: recipes for both full‑parameter and LoRA GRPO, with reference GPU counts (e.g., 96× H200 for full GRPO, or smaller LoRA setups). The workflow uses Megatron for training, SGLang for rollouts, and standard optimizer settings described in the recipes.
Built demos and community incentives
Reflection and partners showcased several builds: a native macOS dictation menubar app (BeamSpeak) using Beam to drive a development harness; multi‑agent experiments splitting programming tasks across agents using a shared Git workflow; and a short experiment where Beam helped generate a synthetic dataset and a decision model pipeline for incident triage. Reflection is offering $10,000 in Hugging Face Hub credits to the best Beam projects shared on the model’s repo page (spaces, model repos, dataset repos).
Conclusion
Beam is a large, open, MoE language model designed for long‑context reasoning and agentic coding tasks. It is distributed in multiple precision checkpoints (BF16, FP8, NVFP4), integrates with transformers, vLLM and SGLang from day‑one, and includes recipes for fine‑tuning and RL. Reflection and Hugging Face provide deployment examples and community incentives to encourage experimentation and adaptation.
