NVIDIA AIPerf: New LLM Inference Benchmarking Tool

NVIDIA AIPerf is presented as the successor to GenAI‑Perf, rebuilt with a multiprocess architecture that avoids the client‑side bottleneck seen in single‑process tools. The tool supports more than 15 endpoint types—including chat, response ranking, and image generation—and can ingest public datasets such as ShareGPT and trace‑replay formats from Mooncake, Baseten, and WEKA AgentX.

AIPerf lets engineers shape traffic with configurable arrival patterns—constant, Poisson, and gamma distributions—with tunable burstiness, enabling tests that mirror production request characteristics. It reports core latency and throughput metrics (TTFT, ITL, request latency, output token throughput) with percentile breakdowns, and, when DCGM or pynvml is available, adds GPU power, utilization, and memory telemetry. Results are emitted to the console and saved as CSV and JSON files.

For AI practitioners, AIPerf offers a quick‑to‑configure load client that can saturate a server without becoming the limiting factor, providing reliable data for capacity planning and performance tuning of LLM deployments.