TensorRT Edge-LLM completed a multi-turn benchmark 6.4 times faster than the published llama.cpp reference run.
NVIDIA says its TensorRT Edge-LLM software completed the MLPerf Inference v6.1 Edge Agentic benchmark in 24 minutes and 36 seconds on one Jetson AGX Thor Developer Kit. The system ran Qwen3.6-27B at 52.33 tokens per second and finished all 1,007 generated turns.
The benchmark replays software-engineering agents that call tools, receive results, and continue the same conversation. Across 20 conversations, the input grew to about 23,500 tokens, testing how well the system handles long histories rather than isolated prompts.
The main time savings come from three techniques. First, NVFP4 stores the model’s weights and activations in a smaller four-bit format, reducing the memory traffic that can slow edge devices. NVIDIA says the setup also used FP8 for the model’s attention cache.
Second, the runtime reuses the cached attention data and recurrent state from earlier turns. Instead of processing the entire conversation again, it processes only the new ending. NVIDIA reports that roughly 96% of prompt tokens used a “hot” cache during this workload.
Third, tree-based multi-token prediction drafts several likely continuations, including alternative tool-call arguments. The target model checks those candidates together, accepting multiple tokens when they match. NVIDIA says this approach could deliver about 40% more decoding performance than its linear three-step version on this workload.
The comparison was against a llama.cpp reference run on the same Jetson hardware, but with a different quantization setup. Developers can inspect NVIDIA’s release/0.9.1-mlpinf branch and attempt the benchmark themselves; the result is a benchmark demonstration, not proof of production performance across vehicles, robots, or other deployments.