NVIDIA’s benchmark shows why a patch that passes local tests may still fail when real models handle real requests.
SWE-Serve is a benchmark for AI coding agents changing inference-serving software—the systems that load models and answer requests. NVIDIA developed it with input from the SGLang team, using 53 executable tasks based on 83 merged SGLang changes.
The benchmark checks work across model support, decoding, caching, scheduling, serving interfaces, and runtime performance. Nineteen tasks start a real server, load the required model, and test the patch through its live interface rather than only running local checks.
That difference changed the results. Across the live-serving tasks, the same 627 patches passed 45.9% of the time with the complete verifier. Removing the live-server checks raised the rate to 69.4%; 147 patches moved from failing to passing. NVIDIA reports that the live tests cover 276 checks, including model loading, request handling, batched generation, expert routing, and log probabilities.
In practical terms, successful inference engineering means more than producing the right code or passing isolated tests. The model must load, the server must accept requests, scheduling must behave correctly, and the returned output must follow the expected interface.
The benchmark’s pass condition remains narrow: a patch only has to satisfy SWE-Serve’s verifier on the declared hardware. A pass does not show that the change is ready for production or for merging into SGLang.
This first release covers 12 CPU tasks and 41 tasks on one NVIDIA H100. It does not test other inference engines, multi-GPU execution, or multi-node serving, so teams should treat it as a focused evaluation of SGLang-style, single-device work—not a general measure of deployment reliability.