# MLPerf Inference v6.1 Introduces Edge Agentic and End-to-End RAG Benchmarks for Real-World AI Evaluation
The latest iteration of the industry-standard MLPerf Inference benchmark marks a significant evolution in how artificial intelligence performance is measured. Version 6.1 moves beyond isolated, single-turn queries to evaluate complex, autonomous systems. The update introduces two entirely new workloads: the Edge Agentic Inference test and the End-to-End Retrieval-Augmented Generation (RAG) benchmark. These additions reflect the AI community’s shift toward multi-step, multi-component pipelines that mirror real-world production environments rather than simple prompt-and-response interactions.
## Why These New Benchmarks Matter
The newly added Edge Agentic Inference workload evaluates autonomous software-engineering capabilities under realistic constraints. Unlike traditional tests that assess a model on a single prompt, this workload requires systems to execute consecutive turns, manage external tools, and parse incoming feedback while operating within tight local power and memory limits. Context histories grow significantly across turns, placing severe pressure on local DRAM bandwidth.
Meanwhile, the End-to-End RAG benchmark addresses the reality that modern query-answering has evolved beyond simply relying on a large language model trained on a static corpus. Stakeholders now need to understand the real-world performance of sophisticated, multi-step pipelines that integrate retrieval mechanisms with generative models to provide accurate, up-to-date answers.
## NVIDIA Jetson Thor Dominates Edge Agentic Inference
The Edge Agentic benchmark measures an OpenAI-compatible endpoint through both performance and accuracy phases. During the performance phase, systems replay twenty recorded software-engineering trajectories covering 1,007 turns. The accuracy phase deploys standard prompts in single-turn mode to verify tool-call validity without active reasoning steps.
NVIDIA’s Jetson AGX Thor Developer Kit delivered exceptional results under the MAXN power profile using 128 GB of unified memory. Running the Qwen3.6-27B model with TensorRT Edge-LLM in SingleStream mode, the board achieved 52.33 tokens per second. The entire 1,007-turn suite was completed in just 24 minutes and 36 seconds—a 6.4x reduction in execution runtime compared to a baseline llama.cpp implementation with Q4_K_M quantization on identical hardware, which took 2 hours and 37 minutes. The system also reached an impressive 87.94 percent accuracy score on function-calling tests.
## Technical Innovations Driving the Speed
Engineering teams secured this dramatic performance advantage through sophisticated memory compression and execution caching strategies. The submission formatted weights and activations in NVFP4 on the Blackwell architecture, while keeping the key-value cache in FP8 precision. This 4-bit layout conserved unified memory, allowing the system to handle long context sequences efficiently.
Furthermore, TensorRT Edge-LLM retained cached attention pages and recurrent states across turns. Because the system only prefilled new suffixes rather than recalculating entire histories, a warm cache served approximately 96 percent of prompt tokens. This mechanism avoided recalculating 13.1 million tokens out of the 13.6 million prompt tokens contained in the workload. Additionally, a tree-based multi-token prediction algorithm drafted eight steps forward across a 16-node verification tree, lifting decoding throughput by roughly 40 percent compared to traditional linear drafting methods.
## Datacentre Gains and Broader Hardware Diversity
The v6.1 release also introduced edge silicon options alongside traditional datacentre hardware. The results showcase rapid gains across the industry: per-accelerator throughput on one prominent reasoning model rose 5.7x over results from twelve months ago, while Visual Language Model performance climbed 2.99x from six months prior.
The largest single cluster scaled to 512 accelerators, and submitters demonstrated novel heterogeneous environments—one crossing an ocean with distributed nodes, and another combining high-speed networking with accelerators from two distinct hardware vendors. The benchmark also introduced speculative decoding into the GPT-OSS task, which verifies multiple candidate tokens within a single forward pass to test the efficiency of interactive services.
Participation climbed to thirty organizations, with over half of entrants adopting an API-centric harness that uses client-server requests to reflect datacentre environments. This interface is expected to serve as the foundation for upcoming testing suites focused on endpoint deployment.
## Frequently Asked Questions (FAQ)
**Q1: What is the primary purpose of the new Edge Agentic Inference workload?**
A1: The Edge Agentic Inference workload evaluates an AI system’s ability to handle autonomous, multi-step software-engineering tasks. It tests how well models manage tools, maintain context over long sequences, and parse feedback while operating within the strict power and memory constraints of edge computing devices.
**Q2: How does the NVIDIA Jetson Thor achieve such a significant speedup over baseline configurations?**
A2: The Jetson Thor leverages memory compression—using NVFP4 for weights and FP8 for the key-value cache—and execution caching. By retaining cached attention pages and recurrent states across turns, the system avoids recalculating the vast majority of prompt tokens. It also uses a tree-based multi-token prediction approach that drafts multiple future tokens at once, significantly increasing decoding speed.
**Q3: What is speculative decoding, and why was it added to the benchmark?**
A3: Speculative decoding is a technique where a smaller, faster model drafts multiple candidate tokens in a single forward pass. These candidates are then verified by the primary, larger model. It was added to the GPT-OSS task to measure the efficiency of interactive services and to validate how effectively systems can process multiple candidate tokens simultaneously.
**Q4: Why are API-centric harnesses important for modern AI benchmarking?**
A4: API-centric harnesses use client-server requests to mirror how AI services are actually deployed in datacentre environments today. They provide a more realistic measure of real-world performance compared to isolated local execution, and they help standardize how future endpoint-focused AI systems are evaluated.
## Conclusion
The introduction of Edge Agentic Inference and End-to-End RAG benchmarks in MLPerf Inference v6.1 signals a pivotal moment for AI evaluation. As models move from simple query-answering to complex, multi-step autonomous tasks—both at the edge and in the datacentre—benchmarks must evolve to match. The dramatic performance improvements showcased by cutting-edge hardware, driven by memory compression and intelligent caching, demonstrate that the industry is successfully meeting these challenges. With standardized API-centric testing frameworks and growing participation from diverse organizations, the AI community is better equipped than ever to measure, compare, and improve real-world inference deployments.
Thank you for reading
