Baseten returned to the public conversation at the peak of the 2026 Open Weights debate with a focused message: inference engineering has become an independent engineering discipline. Philip Kiely and Ali Taha — both active in Baseten’s inference work and public outreach — stressed that the field now addresses a different problem than model training: how to transform trained weights into production systems that are fast, reliable, and cost‑effective at scale.
The industry context includes growing amounts of capital and infrastructure: the conversation referenced very large funding rounds in the space and placed inference providers among the primary beneficiaries of a wider “inference inflection” alongside major hardware vendors.
Practical levers for production inference
Baseten’s practitioners break down the practical techniques that are commonly applied in production to lower latency and increase throughput:
- Cache‑aware routing and KV‑cache reuse: for very long contexts (the pod discusses examples like 200,000‑token requests), routing a request to an instance that already holds relevant KV cache entries can avoid expensive re‑prefill work.
- Disaggregated prefill and decode: separating the GPUs that create the KV cache (prefill) from those that perform decoding can improve resource utilization and throughput in many deployments.
- Speculative decoding: small, fast draft models generate a few tokens ahead; the main model validates drafts. When traffic is domain‑specific (e.g., coding), speculators can achieve high draft‑acceptance rates and large speedups.
- Quantization and layer selection: quantization is the primary lossy optimization — but careful per‑layer choices, calibration and KL‑divergence‑based evaluation of logit distributions can preserve fidelity while increasing speed. Baseten’s research shows that quantizing certain combinations of layers can cause quantization errors to cancel and thus sometimes even improve measured fidelity vs. naive quantization strategies.
- Model grafting and retrofits: for multimodal features, teams can graft vision encoders or other small adapters onto an otherwise unchanged language model (for example, attaching a Kimi encoder onto GLM‑5.2 and training a lightweight projector), preserving original language behavior when no visual input is provided.
These optimizations are complementary and can be stacked; Baseten argues that by combining quantization, speculative decoding, disaggregation and kernel/runtime improvements, practical stacks can achieve multi‑fold improvements — in some cases approaching an order of magnitude improvement over a non‑optimized baseline.
Benchmarks, failure modes and nondeterminism
- Many inference optimizations are lossless (cache reuse) while quantization remains lossy; fidelity checks include not only standard benchmarks but also distributional comparisons (e.g., KL divergence between original and quantized logit distributions).
- Mode collapse (repeating the same token) and nondeterministic failures can be caused by software, kernel race conditions, or hardware/network differences between clusters. The same model weights run on different inference engines or different clusters can behave differently because of kernel bugs, timing/race issues or differences in interconnect speed for KV cache transfer.
- Economics: high‑volume consumers commonly switch from pay‑per‑token public APIs to dedicated deployments (hourly box rentals) once usage is stable, both for cost reasons and for control over tuning (speculators, batch sizing, quantization choices).
Hardware, Dynamo and Rubin
Baseten’s engineers discussed the hardware and systems side at length. NVIDIA’s Dynamo is described as a toolkit for moving KV cache and coordinating data across a cluster, not a plug‑and‑play speedup; it is useful for building KV‑aware routing and offloading solutions. Meanwhile, next‑generation hardware (Rubin) was discussed as emphasizing tighter system integration (faster memory hierarchies, new tensor core designs) that will shift some inference work from low‑level kernel hacking to higher‑level infra orchestration.
Large multimodal or frontier models require hardware with very large HBM pools. For example, models with trillions of parameters may only fit on GB300‑class nodes or multi‑node configurations; in such cases KV‑offloading and network speed become critical constraints.
Parallelism: tensor, expert, pipeline
Baseten explains typical parallelism choices:
- Tensor parallelism shards model matrices across GPUs and requires very high‑bandwidth interconnects (NVLink); useful for latency‑sensitive setups.
- Expert (MoE) parallelism places experts on individual GPUs and can yield high throughput with reduced inter‑GPU bandwidth needs.
- Pipeline parallelism is generally a last resort for multi‑node inference when a model is too large to fit within a single node’s memory; pipeline is combined with tensor/expert techniques inside nodes.
Auto‑tuning (evaluating many parallelism and batch/parallel configurations on real traffic) is the practical way providers find best performance for a given hardware topology.
Speculation taxonomy and stacking gains
Speculative decoding has evolved quickly: early ideas like Medusa taught intuition; modern approaches (DFlash, EAGLE, and further spec‑spec ideas) are faster and more complex. Practical stack effects were described in rough multipliers: shifting from BF16 to NVFP4 quantization yields substantial speedups (cumulative from 16→8→4 bit reductions), speculation adds another ~2×, disaggregation another ~2× in some conditions, plus various double‑digit percent gains from kernels and runtime tuning. Stacked together, these can produce 4–10× improvements in many real‑world configurations, though exact speedups depend on hardware, traffic shape and model architecture.
Multimodality and video: why video is hard
Video generation remains a demanding frontier because even short clips translate to very large token counts, and attention is quadratic in sequence length. Approaches are:
- Dense attention with vast compute (expensive, often in closed systems),
- Sparse attention or locality‑constrained attention (saves compute but can degrade quality),
- Autoregressive video (scales better but currently often degrades quality and accumulates drift over long sequences).
Baseten notes a clear gap between open‑source video models (e.g., Wan family earlier checkpoints) and closed, high‑quality systems (Veo, Kling). The industry is likely to pursue hybrid architectures that combine autoregressive and diffusion elements for long‑form coherent video, but open solutions lag the closed ones today.
Training ↔ Inference: a converging loop
The speakers emphasized the convergence of training and inference: inference traces feed continual fine‑tuning, quantization‑aware distillation and speculator training. This creates a feedback loop where live product traffic improves models and runtime speed; Baseten has already experimented with automatic profiling and even model‑assisted kernel suggestions in test environments. Two practical approaches to continual learning were contrasted:
- Weight updates (pushing new weights/LoRA layers into serving),
- Persistent KV‑cache compaction and long‑term memory mechanisms (preferred for rich, multi‑hop reasoning needs).
Takeaways
Inference engineering now spans kernel‑level optimizations, model adaptations, and cluster/hardware orchestration. The most impactful gains come from stacking complementary techniques — quantization, speculators, disaggregated prefill/decode, KV‑aware routing and hardware‑aware parallelism — and by validating fidelity with distributional metrics, not only task benchmarks. As models grow and new hardware generations arrive, inference engineering will remain central to making large models practical in production, and the field will increasingly blend with training work to maintain continuous improvement.
Key names and references in the discussion: Philip Kiely (Baseten), Ali Taha (Baseten), GLM‑5.2, Kimi K3/K2, vLLM, SGLang, NVIDIA Dynamo, Rubin, GB300, and concepts such as speculative decoding, KV‑cache, disaggregation and quantization. The conversation took place in the context of the 2026 Open Weights debate and reflects practical experience and research shared publicly by Baseten engineers.
