NVIDIA and Princeton researchers report faster high-batch serving by caching recent inputs instead of repeatedly storing a full recurrent state.
NVIDIA and Princeton’s Dao AI Lab propose ReplaySSM, a decoding method for models that combine attention with state-space-model layers. NVIDIA says the method targets a practical problem: state-space layers can become the slowest part of serving when many requests run together.
A state-space model keeps a running summary of earlier inputs. Conventional decoding reads and writes that full summary for every new token. ReplaySSM instead keeps a checkpoint and a small buffer of recent input records. It computes outputs from both, then folds the buffer into the checkpoint only when the buffer fills.
That reduces memory traffic, which matters because these workloads often wait on memory movement rather than raw computation. It also makes speculative decoding easier to roll back: the system can discard unaccepted input records instead of restoring a separate full-state snapshot for each draft token.
NVIDIA tested four hybrid models from the Nemotron 3 and Qwen3.5 families, using one H100 or Blackwell Ultra B300 GPU per model. Against tuned vLLM kernels, NVIDIA reports no speedup at batch size one. At batch size 256, the ReplaySSM kernel ran 1.46 to 1.80 times faster, while the full decode step ran 1.20 to 1.47 times faster.
For speculative decoding at batch size 512, NVIDIA reports 1.85 and 2.10 times the throughput of vLLM on two 120B-class models. The report also describes keeping a high-precision checkpoint for updates while using lower precision for output reads, limiting error buildup.
NVIDIA says ReplaySSM is integrated into several inference frameworks and serves GLM-5.3-Flash. The next practical test is whether these gains hold across other hardware, models, batch sizes, and production workloads.