Evaluating AI Agents via Tool Calls and Task Completion
NVIDIA details a new framework that scores AI agents on both step‑level process and end‑to‑end outcomes using executable environments.
Original source published: September 21, 2026
NVIDIA's technical blog notes that agent evaluation has moved from scoring isolated function calls to measuring full task completion in executable environments that run each tool call and track state across steps. This shift enables verification of whether the final environment state matches the intended goal.
The proposed framework uses two scoring layers: step‑level process scoring to identify invalid, irrelevant, or redundant calls, and end‑to‑end outcome scoring that checks the final state. Metrics roll up through a hierarchy (benchmark → trial → task → turn → step) and are reported on accuracy, verbosity, and cost axes, often paired (e.g., success‑rate ranges). Nemotron 3.5 Lightning is cited as achieving 86 % accuracy on PinchBench while completing tasks 30 % faster. The article advises enterprises to build domain‑specific evaluations from real tickets and APIs, focusing on environment‑state outcomes rather than isolated call accuracy.