---
title: "Visa Benchmark: 23.1% of Agent Runs Bought the Wrong Product"
slug: "visa-benchmark-23-1-of-agent-runs-bought-the-wrong-product"
date: "2026-11-19"
category: "FinTech"
author: "Tariq Al-Mansoor"
correspondent: "Tariq Al-Mansoor"
correspondentRole: "Gulf & Capital Correspondent"
readingTimeMinutes: 2
tags: ["Fintech"]
originalSourceUrl: "https://linas.substack.com/p/fintechpulse1153"
canonicalUrl: "https://aurav.ae/visa-benchmark-23-1-of-agent-runs-bought-the-wrong-product"
publisher: "Aura-V Intelligence (aurav.ae)"
---

# Visa Benchmark: 23.1% of Agent Runs Bought the Wrong Product

> Visa tested 22 AI shopping agents built on models from Google, Meta, OpenAI, Anthropic and eight other labs; in 23.1% of runs the shopper bought a product that did not meet their needs.

## Key Takeaways (TL;DR)

- Visa recently tested 22 AI shopping agents; in 23.1% of all runs the shopper ended up buying a product that didn't meet their needs.
- Those purchases would settle like any other transaction, and nobody has yet agreed on who pays when the shopper disputes the charge.
- The author's framing is that the best AI models still fail, and that a more expensive model doesn't fix that.

## Mis-purchases in 23.1% of runs

Visa recently tested 22 AI shopping agents built on models from Google, Meta, OpenAI, Anthropic and eight other labs [Source](https://linas.substack.com/p/fintechpulse1153). In 23.1% of all runs, according to the newsletter, the shopper ended up buying a product that didn't meet their needs. The 57.2% headline pass rate is noted, and the author points to a consistency figure as mattering more than that headline number.

In a live deployment — the author gives Grok Bot, Claude Duet and others as examples — each of those purchases would go through checkout like any other. The payment is authenticated, the card is tokenized, and the transaction settles.

Read that way, the author writes, the benchmark is basically an early estimate of the dispute bill agentic payments will run up. It also hints at where Visa wants to sit when that bill arrives.

## Model cost is not the fix

The author's framing of the results is that the best AI models still fail, and that a more expensive model doesn't fix that. The benchmark is presented as relevant to agent builders, merchants, issuers and investors.

Our read: A benchmark that sizes dispute exposure is more useful to issuers and agent builders than a pass rate, because the cost arrives after settlement, where the payment record carries no signal of what the shopper intended. Until the parties agree on allocation, that exposure sits wherever the chargeback lands.

## Aura-V Editorial Verdict

Visa's Agentic Commerce Benchmark amounts to an early estimate of the dispute bill agentic payments will run up, and it hints at where Visa wants to sit when that bill arrives. The liability question itself remains open.

---

*Original Think-Tank Source: [Visa tested 22 AI Shopping Agents & 23% bought the wrong thing 🤖🛍️; Nubank’s Q3 2026 earnings: half the profit growth came from taxes, the 12% margin that’s really 10% & what NU stock is worth 🏦💜](https://linas.substack.com/p/fintechpulse1153)*
