On the first day of PyTorch Conf, DeepSeek released DeepInfer, the inference stack it says powers the V4.1 Flash service. The release includes runtime and API-processing components and a production engineering report; it prompted broad community interest because the published production figures and architectural details point to a cache-heavy, high-throughput serving design.
What was released
- DeepInfer: a high-performance inference framework targeting large-scale, low-latency serving. The codebase advertises support for distributed inference, speculative decoding (DSpark-style), SWA Bounded Replay, MoE expert parallelism, structured outputs, and explicit accelerator targets of NVIDIA SM100 and Huawei Ascend 950.
- deepseek-recipe: a Rust library with Python bindings for protocol conversion and model-output parsing across API formats such as Chat Completions, Messages, and Responses.
The repository is positioned to accept future model-inference implementations and optimizations, but the company notes that not all production orchestration elements were published.
What remains non-public or deployment-dependent
DeepSeek acknowledges that key production components remain tied to internal infrastructure. In particular, full KV-cache management, the disaggregation/PD layer, and production load-balancing logic are not fully open-sourced. As a result, the published engine and tooling are valuable, but they do not on their own reconstruct DeepSeek’s complete online service.
The headline production claims
DeepSeek’s engineering notes and related posts include several concrete production figures (company-reported):
- 155 trillion tokens/day from a single cluster
- 96% input-cache hit rate
- 400 ms average time to first token
- 230 tokens/s generation (300 tokens/s for coding workloads)
- 1.5M cache-inclusive input tokens/s per prefill GPU (production data cited)
- 14,000 output tokens/s per decode GPU (reported)
- $38 revenue per accelerator-hour at off-peak prices (company figure)
Community analyses emphasize that these numbers describe a service that is heavily cache-dependent—155T/day includes cached prefixes and is not the same as 155T newly generated tokens/day. The 96% input-cache hit rate is crucial context for interpreting throughput and economics.
Architecture and scheduling principles
DeepInfer separates vision encoding, prefill, and decode so each stage scales against its own bottlenecks. Deployment guidance includes ordinary eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode. Key design points published or discussed in the engineering notes:
- prefill scheduling uses estimated attention-compute rather than raw request length to allocate work and fit time budgets
- cached prefixes live in shared 3FS storage, so requests need not be routed back to the worker that previously served a prefix (no strict cache affinity)
- decode routing balances running-request count and accumulated KV-cache length (capturing both scheduling and memory/state pressure)
- expert-parallel replicas are redistributed periodically to mitigate routing skew at the expense of additional HBM usage
- asynchronous CPU scheduling overlaps host orchestration with accelerator execution to reduce idle gaps
A notable configuration detail is the apparent lack of tensor parallelism in the published Blackwell setup; community commentators trace that choice to earlier interconnect constraints in DeepSeek’s evolution, but note this is an interpretation rather than a general condemnation of tensor parallelism.
Cache hierarchy and state management
The cache design links throughput figures, agent workloads, and the reported revenue per accelerator:
- persistent block KV resides in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity
- reported retention: three days for agent traffic, seven days for chatbot traffic
- any prefill worker can fetch a cached prefix, decoupling cache locality from compute placement and enabling independent scaling of storage and compute
- sliding-window attention state stays in host memory (shorter lifetime, useful within active sessions)
- SWA Bounded Replay reconstructs missing short-lived state from only the latest 128 tokens, trading an approximation for much lower prefill computation (DeepSeek reports nearly halving total prefill work)
The practical consequence: high cache-retention and reuse make many tokens cheap to serve, but cache hits and newly computed prefill/decode tokens impose very different costs. Commentary also notes diminishing returns from very long retention tails (roughly 0.5–1 percentage point of extra hits), illustrating a trade-off between storage capacity and recomputation.
OpenAI’s separate announcement of 12-hour API prompt caching is a contemporaneous comparison point—the two products are not identical but both target persistent agents that reuse context across long gaps.
Portability and model-specific optimizations
DeepInfer includes many model/runtime co-design choices: a smaller prefill backbone activating fewer parameters per token, wide expert parallelism, highly compressed KV, speculative decoding, and multi-bit quantization patterns. Community explanations mention numbers such as a 196B Engram backbone, 552B expert parameters, DP64/EP64 layouts, and claimed large KV compressions; however, those secondary claims lack sufficient public detail (replication, shared-weight layout, quantization scheme) to fully reconstruct or generalize the reported reductions. The implementation therefore appears model-specific and may not transfer unchanged to arbitrary checkpoints.
vLLM reproduction and what it means
vLLM published a comparative follow-up: acknowledging DeepInfer’s Apache-2.0 release and crediting DeepSeek, vLLM reported decode-throughput within ~6% of DeepInfer’s published results on GB200 when configurations were matched. Important caveats:
- the within-6% parity covers decode-stage throughput, not the entire production service (cache operations, routing, retention economics, and end-to-end latency under real traffic are not reproduced by that measurement)
- reported per-GPU input/output accounting in production headlines must be interpreted with the service’s cache/input/output mix
vLLM’s openness and published recipes materially qualify claims that DeepSeek’s efficiency is inaccessible to existing open-source engines, but they do not invalidate production-scale infrastructure differences.
Economic discussion and debates
The most consequential financial disclosure is the $38/accelerator-hour revenue at off-peak prices. Community arithmetic explored inference margins under different assumptions and produced a range of inferred margins highly sensitive to token mix, hardware rental costs, and included expenditures:
- early public estimates (with specific cache/uncached splits and GPU-cost assumptions) produced inferred inference margins in the 80–95% range under optimistic inputs
- alternative cost assumptions ($5–$10/GPU-hour rental) compress the accelerator-only margin to roughly 74–87%
Critically, these accelerator-only calculations omit SSD cache fleet costs, CPUs, networking, idle/reserve capacity, engineering and support costs, and other operating expenditures. They also rely on a favorable cache-heavy traffic mix. Large company-level losses can coexist with high inference contribution margins if other costs are significant.
Some community extrapolations projected very large revenue or gigawatt-scale deployments under optimistic scenarios; these are scenario calculations, not DeepSeek-audited disclosures.
Hardware compatibility and performance notes
The repo lists NVIDIA SM100 and Huawei Ascend 950 as accelerator targets. Community analysis compared reported decode figures for GB200 and Ascend 950DT under different constraints and found wide variation in tokens/s/device ratios (examples cited range from roughly 5× to 7.4× differences depending on workload assumptions). Those comparisons combine different measurements and procurement price assumptions and are not direct end-to-end hardware benchmarks.
A commentator projected substantial future Ascend allocations and deep Blackwell-equivalent compute by end-2027 for DeepSeek, but clarified these are projections rather than procurement announcements.
Community reactions and implications
Reactions span endorsement, skepticism and qualified support:
- some researchers (e.g., Tim Dettmers) judged DeepSeek’s inference stack comparable to major providers at user scale; others praised API speed and price
- some experienced inference engineers found the engineering write-up more valuable than the repo alone, since important deployment components remain absent
- other community members suggested the release will inform open-source engines and hosting providers and could increase competitive pressure on hosting economics
Overall, the public release supplies reusable engineering lessons and practical recipes to the community, while also making clear that reproducing the exact production economics and scale requires more than just the code in the repository.
Why this matters
DeepInfer’s publication is a meaningful step for transparency: it surfaces production optimizations, cache strategies and scheduling heuristics that have driven a cache-heavy, high-throughput service. The repository and the engineering report together let the open-source inference ecosystem test and adapt these techniques. At the same time, the debate about which parts are general-purpose versus model- and infra-specific will determine how widely others can realize similar throughput and economics.
Next steps for reproducibility
Public reproductions (such as vLLM’s decode parity work), community benchmarks and further disclosures from DeepSeek about orchestration and cache operations will clarify which gains are portable. For now, DeepInfer is a substantive contribution of methods and code, but not a turnkey replacement for a production-scale, fully orchestrated inference service.
This article synthesizes DeepSeek’s published repository and engineering notes, community analyses and vLLM’s comparative measurements; the production numbers cited are DeepSeek’s own reported figures and have not been independently audited.
