---
title: "Introducing the Inference Performance Index to Benchmark AI Chip Capacity"
slug: "introducing-the-inference-performance-index-to-benchmark-ai-chip-capac"
date: "2026-11-10"
category: "AI & Automation"
author: "Dr. Kai Wei"
correspondent: "Dr. Kai Wei"
correspondentRole: "Lead Frontier AI Correspondent"
readingTimeMinutes: 3
tags: ["AI"]
originalSourceUrl: "https://epoch.ai/publications/inference-performance-index"
canonicalUrl: "https://aurav.ae/introducing-the-inference-performance-index-to-benchmark-ai-chip-capac"
publisher: "Aura-V Intelligence (aurav.ae)"
---

# Introducing the Inference Performance Index to Benchmark AI Chip Capacity

> Epoch AI’s Inference Performance Index quantifies how memory and inter-chip bandwidth shape real-world inference throughput.

## Key Takeaways (TL;DR)

- The IPI measures concurrent request capacity across output-speed requirements in GB300 equivalents.
- Nvidia’s GB300 NVL72 outperforms the H200 HGX by roughly eightfold in modeled capacity per GPU.
- Memory bandwidth and interconnect scale, not raw FLOP/s, drive the biggest gains in agentic inference workloads.

## A new lens on inference throughput

Epoch AI argues that raw FLOP/s figures mask the true determinants of inference capacity, especially for agentic workloads that dominate token usage on platforms like OpenRouter. Their analysis shows the Nvidia GB300 can deliver about 25 × the inference capacity of an H100 chip while offering only 2.5 × the 8-bit FLOP/s, a disparity they attribute to memory and communication limits. To capture this, Epoch introduces the Inference Performance Index (IPI), which reports capacity in “GB300 equivalents” (GB300e). A score of one represents the baseline GB300; a score of two means a chip can serve twice as many simultaneous requests on average across the modeled workloads and output-speed targets. [Source]

## How the index is built

The IPI combines four representative agentic models and evaluates both prefill and decode phases. Using Nvidia’s AISimulate, Epoch feeds each chip’s arithmetic throughput (BF16, FP8, FP4), HBM capacity, memory bandwidth, and inter-chip bandwidth into an analytical “speed-of-light” model. Concurrency (requests per chip) and output speed (tokens / s / request) are traded off: raising concurrency from two to eight requests reduces output speed from roughly 210 to 80 tokens / s / req, but total throughput rises about 50 % (from 420 to 640 tokens / s / chip). The IPI integrates this trade-off by plotting capacity curves across a 50–400 tokens / s / req range, then calculating the area under each curve on a logarithmic axis. The resulting area, divided by the GB300 baseline, yields a chip’s IPI score.

## Early rankings and what they reveal

According to the report, the GB300 NVL72 scores 1 GB300e by definition. Within the same Blackwell Ultra family, the B300 HGX scores 0.54, while the larger-scale GB300 NVL72 reaches 1.9 × that figure, underscoring the impact of its 72-GPU interconnect. The MI355X lands at 0.55 and the Vera Rubin (VR200) chip at 2.18.

Epoch notes that InferenceX’s own benchmarks echo this ordering, attributing Hopper-based H200’s weaker Kimi K3 performance partly to missing software optimizations.

## Practical implications for data-center capacity

Because the IPI expresses capacity in GB300e, operators can translate a fleet’s hardware mix into a common unit. This scaling insight helps estimate how many agents or users a data center can serve simultaneously, given a target output speed. Epoch plans to update the index as inference demand evolves, adding new models and workload categories.

Our read: The IPI reframes inference benchmarking around memory and interconnect bandwidth, exposing a performance frontier that FLOP-centric metrics overlook. Its early chip rankings suggest that investments in larger HBM pools and high-density interconnects yield disproportionate capacity gains for agentic workloads, a trend that could reshape hardware roadmaps and procurement strategies.

## Aura-V Editorial Verdict

The IPI offers a hardware-aware metric that corrects the over-reliance on FLOP-based comparisons. Its early results highlight the strategic advantage of chips with larger HBM and high-bandwidth interconnects.

---

*Original Think-Tank Source: [Inference Performance Index: changing the way we measure inference capacity](https://epoch.ai/publications/inference-performance-index)*