SWE-Serve is a benchmark designed to evaluate repository-scale changes across inference-serving stacks by exercising the full serving path rather than only repository-level checks. Developed with input from the SGLang team, SWE-Serve converts 83 merged SGLang pull requests into 53 executable tasks that reflect real engineering work on model serving.
What SWE-Serve tests
Unlike repository-level benchmarks that cover general software-engineering tasks or inference benchmarks that focus on kernels and performance, SWE-Serve covers the full inference-serving stack: model and backend enablement, decoding strategies, caching and runtime state, scheduling, serving APIs, runtime correctness, and performance.
The 53 tasks are grouped into six engineering families:
- Speculative and advanced decoding: 14 tasks
- Model and backend enablement: 12 tasks
- Kernels, quantization, and performance: 8 tasks
- Serving APIs and runtime correctness: 8 tasks
- Caching and runtime state: 7 tasks
- Distributed execution and scheduling: 4 tasks
Twelve tasks run on CPU, and 41 use a single NVIDIA H100. This first release does not evaluate other inference engines, multi-GPU execution, or multi-node serving.
Tasks and verification
The benchmark draws on 83 merged SGLang pull requests: 37 tasks stem from a single upstream PR and the other 16 combine two to six related changes. Each task provides an instruction and a containerized SGLang checkout from before the target change. A patch passes only if it satisfies the task’s hidden verifier on the declared hardware; the submission is never compared to the reference implementation.
Reference changes are substantial: the median reference solution edits 553 lines across seven files. A typical verifier contains around seven tests for the new behavior and 10 regression tests. Nineteen tasks start a real server, and three enforce a calibrated performance gate on an H100.
Example task
One task asks an agent to add serving support for dense and mixture-of-experts (MoE) Qwen3.5 models. Starting from a revision without Qwen3.5 support, the agent must enable both a 0.8B dense model and a 35B-A3B MoE model to load and serve through SGLang’s normal interfaces on one H100. The verifier checks model registration, configuration and weight loading, image and video inputs, OpenAI-compatible requests, native batched generation, log probabilities, and MoE expert routing.
What live-serving tests catch
Some failures only appear once a real server starts and the model is loaded. The SGLang team suggested end-to-end verification ideas, and SWE-Serve includes 19 tasks that load the required model and exercise the patch through a live serving interface.
Across those tasks, the same 627 patches pass 45.9% of the time under the complete verifier. When the live serving tests are excluded, the pass rate rises to 69.4%. In other words, 147 patches transitioned from fail to pass when live-serving checks were omitted — roughly one in three patches that passed other checks failed a live serving test.
The 19 tasks include 276 live serving tests; 242 of those tests were sourced or adapted from SGLang, and the remaining 34 cover behavior introduced by the merged changes for which there was no suitable upstream test.
The Gemma 4 MoE task illustrates the difference: of 33 patches, 16 passed every other check but failed at least one live-serving test covering model loading, expert routing, text and image serving, or batched generation with correct ordering and log probabilities.
Note that a SWE-Serve pass has a narrow meaning: the patch satisfies the benchmark verifier. SWE-Serve’s tests are not SGLang’s upstream review process and do not imply that a patch or reference solution is deployable, ready to merge, or endorsed by SGLang maintainers.
Where agents struggle
The request-to-output path was divided into four runtime domains: request handling and I/O; scheduling and request lifecycle; model execution; and KV-cache and runtime-resource management.
Using the best setting for each of 11 models, tasks confined to a single runtime domain (26 tasks) had a 69.0% pass rate, while the 27 tasks spanning multiple runtime domains had a 47.7% pass rate — a 21.3 percentage-point drop. Every model setting showed the same directional difference.
Variation across models and settings
Performance varies widely across models and effort settings. Across each model’s best tested configuration, mean pass@1 ranges from 34.6% to 75.5%. The evaluation used mini-swe-agent, a minimal software-engineering agent that uses only Bash in closed-book conditions. Two Claude and three GPT-5.6 models were tested at five effort levels; the other six models ran at one setting each, producing 31 model-effort configurations in total.
Each configuration ran the full 53-task benchmark three times. Agent sessions were capped at 210 minutes and 350 steps. A task counts as solved only when the agent’s patch passes the complete verifier on the task’s declared hardware.
Among the best-performing configurations, Claude Opus 5 (max) and GPT-5.6 Sol (max) reached around 75% mean pass@1. However, cost and wall time do not map cleanly to performance: configurations that tied at 64% mean pass@1 had mean costs per task ranging from $0.95 to $7.24 and mean wall times from 25.5 to 99.9 minutes. No single model led all six engineering families, and similar overall scores can mask different strengths.
Benchmark validation and integrity
The creators screened 786 potential task sources, built 156 executable candidates, and admitted 53 tasks. Every admitted task was tested on its declared hardware; the unmodified repository had to fail the new-behavior tests while still passing regression tests, and a reference patch had to pass the complete verifier. Verifiers were stress-tested with agent-created patches, and tasks were repaired, narrowed, or excluded when concrete issues arose.
Reported evaluations are closed-book: public web and upstream source repositories were blocked, while Hugging Face access for model weights was allowed because an open-network pilot showed models retrieving task-specific upstream code. The team audited all 1,749 trials behind the leaderboard; 196 prohibited retrieval attempts were blocked and none succeeded. The paper documents the full qualification and evaluation-integrity process.
Run SWE-Serve
SWE-Serve includes task environments, verifiers, and baseline configurations and makes the gap between passing local checks and working through the full serving path measurable. Researchers and engineers can explore the leaderboard and run SWE-Serve on GitHub to evaluate coding agents on SGLang inference-engineering tasks.
