Visa's Agentic Commerce Benchmark: AI Shopping Agents Pass 57.2% of Runs

Visa has published a benchmark showing that AI shopping agents are not yet reliable buyers. In its first round, announced on November 5, 2026, 22 models from 12 labs tried 15 shopping scenarios. The overall pass rate was 57.2%. The best models passed 82.2% of attempts and the worst passed 28.9%. Most striking, 54% of failed runs still ended with the agent buying something. The benchmark is Visa's own, and we could not load Visa's results site, so this post relies on Fortune, WWD and Digital Transactions, all of which quote Visa.

This matters because agents that shop are a live product bet. We have covered Uber's rides from third-party agents, DoorDash's agent orders and Grok's shopping cards. Visa's numbers are among the first public attempts to measure how often such agents get it right.

TL;DR: what Visa reported

How the test works

Per WWD's account of Visa's statement, the benchmark asks an AI model to identify a consumer's needs from a conversation. If nothing in the catalog fits, the right answer is to not buy. Visa gives the example of a customer who wants a television: a relative recommended Samsung, the customer holds a membership with a third-party supplier, and the agent must uncover the budget and further needs such as Dolby Vision and gaming capability. Fortune describes a similar setup, with a list of attributes such as compatibility with a child's gaming console, a 30-day return window and Dolby Vision.

Two design choices stand out:

• The shopper holds back information. As with real consumers, the simulated shopper volunteers some requirements but not all. The agent has to ask the right questions. Fortune notes that the shopper is also an agent.
• No lucky guesses. Visa says the agent must uncover enough information to justify its decision. A correct purchase made without asking the right questions should not count.

Every model got three attempts per scenario. That is how Visa can say that all three attempts passed in only 34.2% of cases: the average pass rate is not the same as dependable behavior.

The results in context

Fortune adds a cost angle: Visa's chart plots success against cost per shopping inquiry, and Visa notes that the five most expensive models succeeded on average 67% of the time, while the cheapest five scored 50%. Aggregator reports quote per-run costs for the top two models, but we could not trace them to Visa's own page, so we do not repeat them here.

The ranking Fortune gives is limited to the top. Google's Gemini and Meta's Muse agents are tied near 82%, with exact versions not confirmed in what we could read, a Claude agent is third at 76%, and an OpenAI agent is sixth at 64%. Fortune does not name the exact versions of Claude or OpenAI models, and we do not know where Grok or others placed. Digital Transactions says the field included Gemini, Claude, Muse and Grok. For comparison with how Google's and Meta's flagship models do elsewhere, see our posts on Gemini 4 Argon's launch benchmarks and Muse Wonder versus GPT-6.1 Sol, Opus 5.5 and others.

Small cart with a green price tag, representing the Visa Agentic Commerce Benchmark for AI shopping agents

Why 54% of failures still buying is the number to watch

A shopping agent can fail in two ways. It can do nothing, which is annoying. Or it can buy the wrong thing, which costs money and can be hard to undo. Visa's finding is that, among failed runs, more than half were the second kind. In Visa's words, as quoted by WWD, the AI either "recommended a purchase that did not meet the customer's requirements, or it failed to uncover all the customer's requirements."

That points to the real weakness. The problem is not browsing a store or clicking a checkout button. It is calibrating when to stop and ask, and when to say "nothing here fits." Models trained to be helpful tend to complete the task. In shopping, a helpful-looking purchase can be the wrong action.

For builders, this suggests concrete design rules:

1. Make "do not buy" a first-class outcome in your agent's prompt and tests.
2. Require a requirements checklist before purchase: budget, must-haves, return window, brand rules.
3. Confirm with the human for purchases above a set amount, as Fortune's reporting and the trust surveys suggest consumers want.
4. Log the evidence the agent used so a failed purchase can be audited.
5. Test with withheld information, as Visa's simulated shopper does.

What Visa has not shown, and what to be careful about

• It is Visa's own benchmark. Visa sells payments and has an interest in agentic commerce growing. That does not make the test wrong, but independent replication would help.
• Simulated shoppers. The shopper is an agent, and scenarios are scripted. Real people are messier.
• First round only. Visa says it wants to expand to other categories. Three product types do not represent all of retail.
• Narrow slice of commerce. Fortune quotes Visa's Rubail Birwadker saying that product discovery is just one element of agentic commerce. Payments, fraud, and liability are separate.
• Versions matter. Models change fast. A result for one version tells you little about the next.
• We have not seen the raw data. We could not load Visa's results site, so the model-by-model table is incomplete here.

Fortune also notes that liability for rogue purchases or fraud by agents remains unresolved. That is outside the benchmark but central to deployment.

What this means for what you build or pay

If you build a shopping or purchasing agent, the benchmark is a free test harness and a reference for failure modes. Run your own agent against it before you ship, and track two numbers instead of one: the average pass rate and the share of scenarios where all three attempts pass. Visa's gap between 57.2% on average and 34.2% for full consistency shows why a single demo proves little.

If you pay for model access, the cost chart Fortune describes suggests that more expensive models did better on average, but the top two tied at 82%, so price alone does not decide the winner. Match the model to the stakes: use a stronger model where a wrong purchase is costly, and a cheaper one where a human reviews every cart.

How consumers and shops should read this

WWD cites a Visa survey from last May in which only 23% of US consumers said they trust AI to handle payment transactions on their behalf. The benchmark gives that distrust a number: agents that mostly succeed, but not consistently. For shoppers, the practical advice is to treat an agent's pick as a recommendation until you confirm it, especially for items with returns rules or compatibility needs.

For retailers, the finding that agents often miss requirements suggests that clear, structured product data and visible return policies will matter. Agents that cannot find the answer may guess. If you run a shop, test how popular agents describe your products. Our guide on Cloudflare's trusted agent delegation covers one effort to let sites identify and trust agents, and Paul Graham's view on Amazon banning agents shows the other side of the debate.

What to watch next

• Visa's next rounds and whether it adds categories such as groceries or travel.
• Lab responses. Visa says it has shared the benchmark with leading model providers for feedback. Watch for labs citing their scores.
• Independent runs of the public benchmark.
• Holiday data. Visa wants people to try it before Black Friday, so expect anecdotes soon.

Bottom line

Visa's first Agentic Commerce Benchmark says AI shopping agents get the job right a little more than half the time, with a wide gap between the best and the worst. The sharpest finding is that when agents fail, they often still buy. The test is Visa's own, narrow and early, but it gives builders a clear target: teach agents to ask more, and to decline when nothing fits.

Related reading

• Uber: four rides from third-party AI agents
• DoorDash: several hundred agent orders, a reality check
• Grok Bot shopping product cards and Shop Pay
• AWS SageMaker 4B shopping agent via SeqKD distillation
• Paul Graham on Amazon banning agents
• Gemini 4 Argon launch benchmarks and pricing
• Cloudflare trusted agent delegation

Figures are accurate as of November 5, 2026, based on Visa statements as reported by Fortune, WWD and Digital Transactions.