Its strongest result came from a synthetic test; a GSM8K evaluation showed a smaller, more practical gain.
NVIDIA describes three software techniques for reducing the cost of running Ling-3.0-flash on its DGX Spark computer. The work matters to AI developers because inference speed depends heavily on moving model data through memory, not just on raw computing power.
In a batch-size-one evaluation, NVIDIA found that decoding was memory-bound. The mixture-of-experts calculation and the output layer—which scores every possible next word—were notable costs.
Humming replaces the standard mixture-of-experts execution path with specialized GPU kernels. It compiles code at runtime for the model’s shapes and hardware, then stores the compiled binaries for reuse. On a synthetic workload, NVIDIA reports that Humming raised throughput from 36.09 to 39.45 tokens per second, a 9.3% increase.
The second technique quantizes the output layer from BF16 to FP8, using smaller numeric values. NVIDIA says this halves the layer’s weight-read volume. Combined with Humming, the synthetic test reached 43.94 tokens per second, up 21.7% from baseline.
NEXTN takes a different approach. Its built-in prediction layer drafts three tokens, while the main model verifies four positions in each round. With all three techniques enabled, NVIDIA reports 102.88 tokens per second in the synthetic test—185.1% above baseline.
That headline result is workload-specific. Across all 1,319 GSM8K questions, NVIDIA reports a 42.5% throughput increase and 24.5% lower mean latency. Accuracy shifted from 96.89% to 96.82%, while draft acceptance fell from 3.90 tokens on the synthetic test to 2.68 on GSM8K.
Integration also needs testing: without the required shared output-layer interface, the system can silently become slower than baseline. Developers should reproduce the results on their own workloads before budgeting for the gains.