OpenAI has introduced Jalapeño, its first custom inference chip and the rack-scale system built around it. According to the company, test results show a notable step forward: Jalapeño can serve more AI work per unit of power while returning responses faster. OpenAI says the chip achieves both higher throughput and lower latency with a single architecture, whereas many existing hardware systems must trade one for the other.
What this means for users
OpenAI states the gains can translate to faster responses, more responsive agents, and more reliable access under growing demand. The company frames the development as aligned with its mission to ensure artificial general intelligence benefits all of humanity, arguing that higher efficiency can make more capable AI more affordable and more broadly available.
Measurement approach and benchmark
Performance was evaluated by matching user experience and measuring how much useful AI work a system can complete per unit of power while meeting latency requirements for users and interactive agents. OpenAI used the public InferenceX benchmark from SemiAnalysis, which measures the full process of serving an AI request. Jalapeño was compared to leading commercially available AI systems across the tested operating range, from high-throughput serving to highly interactive, low-latency use.
Results on three public models
Across three public models, Jalapeño delivered 1.5–1.9× more AI work per watt at peak throughput and 1.7–3.6× lower end-to-end latency than the comparison systems. For highly interactive workloads, performance improved by 2.1–4.1×.
The comparisons were normalized using each accelerator’s published chip power rating. Jalapeño is rated at a 700 W package TDP, although measured sustained power during the tested workloads remained at or below 550 W.
Key InferenceX results (Jalapeño vs. comparison system; comparison entries reference higher-TDP GB-series accelerators):
- GPT-OSS 120B (nominal 8k/1k, STP): end-to-end latency 0.69 ms vs. 1.87 ms (1,459 vs. 535 tok/s/user); 22,935 vs. 427 mixed/kW (at 535.28 tok/s/user).
- DeepSeek R1 MXFP4 (nominal 8k/1k, STP): end-to-end latency 1.43 ms vs. 5.90 ms (700 vs. 169 tok/s/user); 12,258 vs. 118 mixed/kW (at 169.41 tok/s/user).
- Kimi K2.5 MXFP4 (nominal 8k/1k, STP): end-to-end latency 1.44 ms vs. 5.48 ms (694 vs. 182 tok/s/user); 6,744 vs. 120 mixed/kW (at 182.46 tok/s/user).
On the largest tested public model, Kimi K2.5, Jalapeño delivered roughly 1.5× higher peak performance per watt and about 3.4× lower end-to-end latency than the comparison system.
OpenAI also reports that Jalapeño’s advantage increased further in internal tests on frontier OpenAI models, suggesting the architecture scales its benefit as workloads grow larger and more demanding.
Full-stack design and architecture choices
OpenAI emphasizes that the gains stem from co-design across chip, memory, network, software, and rack-scale system, tuned to real language-model workloads. Language-model inference moves through distinct phases with different bottlenecks: prefill (prompt processing) is compute-intensive, while decode (token-by-token generation) is more constrained by memory bandwidth. Communication overhead can add latency when state or data must move between cores and chips, leaving processors idle.
Jalapeño was designed to minimize data movement and communication delays: model state, including the KV cache used during generation, can be explicitly placed and kept local while the system activates appropriate combinations of compute, memory, and networking for each phase. The network is integral to the architecture, with a large domain that keeps workloads within a single connected system and reduces data movement across components. The result is a balanced, fungible accelerator capable of supporting changing model architectures and performing well in both prefill and decode phases—important for agentic workloads.
AI-assisted chip development
OpenAI says AI models played a direct role in Jalapeño’s development. Earlier model generations helped guide chip design and bring-up, while newer models accelerated programming and optimization. The team moved from initial design to tapeout in nine months by using AI to explore implementations and shorten design, measurement, and verification loops. AI also assisted in optimizing arithmetic circuits to fit more compute performance onto the chip on schedule.
The chip was designed as a predictable programming target for both human engineers and AI: work is described via local tensors, explicit communication, and predictable synchronization so AI can optimize mapping, placement, scheduling, and coordination across the system. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5–1.8× faster than prior human-expert-written implementations (the company stresses these figures apply to selected blocks, not whole models).
Deployment plans and roadmap
OpenAI plans to begin deploying Jalapeño within its compute infrastructure by the end of the year. Jalapeño is positioned as the first generation of a multigenerational roadmap: Gen 2 is deep in development and Gen 3 is taking shape. Each generation will build on lessons learned to further improve efficiency and speed.
Meeting growing AI demand will require compute from every available source; OpenAI says it will continue to widely deploy accelerators from NVIDIA and other partners for both training and inference workloads. As Jalapeño is prepared for deployment, the company is continuing production qualification, maturing software, preparing to operate at scale, and validating performance across additional models. The reported results illustrate what OpenAI argues is possible when the full system is designed together: more responsive, capable, and agentic AI delivered more efficiently to more people.
