On the first day of PyTorch Conf, DeepSeek published DeepInfer — an inference framework it says underpins DeepSeek V4.1 Flash — together with a companion deepseek-recipe library for API compatibility. The release targets high-throughput, low-latency production inference and is licensed under Apache 2.0, but not every component of DeepSeek’s production service was exposed; certain service-level elements remain tied to internal infrastructure.
What DeepInfer includes
- DeepInfer implements distributed inference, speculative decoding, SWA Bounded Replay, structured outputs and support for NVIDIA and Huawei Ascend accelerators. The deepseek-recipe component provides a Rust implementation with Python bindings to translate between common API formats (Chat Completions, Messages, Responses) and handle tokenization and output parsing.
- DeepSeek credits vLLM, SGLang and TRRTLLM as upstream contributors and reports additional optimizations layered on that base. However, the company states that strategic pieces — notably its ultra-optimized KV cache management and some load-balancing orchestration — are not yet fully open because they are integrated with DeepSeek’s infrastructure.
Production metrics published by DeepSeek
DeepSeek’s engineering report and announcement include the following production figures: a single cluster serving 155 trillion tokens/day; a 96% input-cache hit rate; average 400 ms time to first token; 230 tokens/s generation throughput general case and 300 tokens/s for code generation. The report also cites $38 revenue per accelerator-hour at off‑peak pricing.
These are first‑party production claims from DeepSeek, not independently audited measurements.
Architecture and operational patterns
- DeepInfer separates vision encoding, prefill and decode stages so each can scale against different bottlenecks. Recommended deployment shapes in the report are eight‑accelerator machines for prefill and 64–128‑accelerator supernodes for decode.
- Prefill scheduling estimates attention compute and allocates work by estimated compute cost (compute tokens), allowing chunk sizes to adapt to a time budget rather than enforcing a single chunk size across heterogeneous requests.
- Prefill routing does not require cache affinity because cached prefixes live in shared storage; any prefill worker can fetch a prefix, decoupling cache locality from compute placement.
- Decode routing balances running-request count and accumulated KV-cache length to capture both scheduling pressure and memory/state pressure.
- Expert-parallel replicas are periodically redistributed to reduce wait caused by routing skew; replication increases HBM use with workload‑dependent diminishing returns.
- CPU scheduling overlaps with accelerator execution via asynchronous orchestration to reduce idle gaps between decode steps.
Cache hierarchy and retention
DeepSeek describes a persistent block KV cache stored in a separate 3FS cluster with roughly 100 PiB of SSD capacity. Reported retention windows are three days for agent traffic and seven days for chatbot traffic. Because any prefill worker can retrieve a cached prefix from shared storage, cache locality is decoupled from compute placement, enabling independent scaling of storage and compute.
The retention policy and the high cache-hit rate are crucial to interpreting throughput: the advertised 155T tokens/day includes cached tokens, so it does not equal 155T newly generated tokens/day. DeepSeek’s reported 96% input-cache hit rate is therefore fundamental context for the headline throughput numbers.
Short‑lived attention state (sliding‑window) remains in host memory rather than on SSD and is useful mainly within active sessions. SWA Bounded Replay reconstructs missing short-lived state using only the last 128 tokens and, according to DeepSeek, nearly halves total prefill computation compared to full reconstruction.
Reproduction and independent comparisons
vLLM responded with reproduction work: under a matched configuration on GB200 hardware, vLLM reported decode‑stage throughput within about 6% of DeepSeek’s published numbers. That result addresses decode-stage performance parity but does not reproduce the end‑to‑end production stack (100 PiB cache operations, routing, retention economics, and cluster-level availability), so it does not invalidate the importance of the missing infrastructure components.
Three caveats apply to the vLLM comparison: it is a decode‑stage parity result (not whole-service reproduction); it does not validate DeepSeek’s cache or routing subsystems; and the benchmark’s token accounting (input/cache/output) must be matched before comparing throughput headlines.
Economics and margin discussion
DeepSeek’s $38 per accelerator‑hour revenue figure at off‑peak prices is widely discussed. Community arithmetic using DeepSeek’s production numbers (e.g., 1.5M cache-inclusive input tokens/sec per prefill GPU, 96% input cache hit, 14,000 output tokens/sec per decode GPU across ~2400 GPUs) yields very high inference‑level contribution margins under favorable hardware cost assumptions (examples cited $5–8 per card/hour), sometimes exceeding 80% inferred margin. Those calculations are sensitive to assumptions about GPU rental cost, cache accounting, SSD fleet cost, CPUs, networking, idle capacity, failure overheads, engineering costs, and workload token mix.
Different community posts use different cost assumptions and reach different margin ranges (e.g., 73.7–86.8% at $5–$10/GPU‑hr). Some extrapolations project rack counts and revenue at large scale but are scenario exercises, not audited corporate disclosures.
Community reactions and caveats
- Enthusiastic commentators argue DeepSeek’s serving stack is comparable to top closed systems for user scale; others emphasize that missing infrastructure code and operational know‑how limit direct reproduction.
- Technical commentators highlighted that many of DeepSeek’s efficiency advantages are model/runtime co‑design decisions: a 196B Engram parameterization with a smaller prefill backbone, wide expert parallelism, heavy KV compression, speculative decoding and hardware/quantization choices. The blog post did not publish all replication-specific details (replication factors, quantization layout, shared weights), so independent reconstruction is nontrivial.
Takeaway
DeepInfer’s open release and DeepSeek’s detailed engineering writeup contribute valuable production knowledge to the community. The decode engine and API‑processing recipe are useful artifacts, and independent teams (vLLM) have shown that core decode throughput is reachable with open tooling. However, the most consequential production claims depend on a cache‑heavy accounting model and on service orchestration components that DeepSeek has not fully open-sourced. As a result, while the release advances transparency and may accelerate open production-inference engineering, validating the full set of performance and economic claims requires reproducing large-scale cache, routing and operational infrastructure that are not included in the repository.
(Information in this article is drawn from DeepSeek’s release and engineering report and from community reproductions and analyses; production claims cited are DeepSeek’s own.)
