GemNews

All stories · Article

Visa’s AI shopping benchmark: 42.8% of AI shopping runs failed, and only 34.2% met shoppers’ requirements in all three attempts

*Visa’s new benchmark found that AI models failed 42.8% of shopping test runs. The bigger concern is consistency: models got all three attempts right in just 34.2% of cases.*

By Janie Nguyen · · 6 min read

Share: WhatsApp · Telegram · X · Facebook · LinkedIn · Email

The headline number is 42.8%. The reliability problem is 34.2%.

On November 5, 2026, Visa introduced its Visa Agentic Commerce Benchmark, testing 22 AI models from 12 labs across 15 shopping scenarios involving products such as TVs, laptops and personal care appliances.

For merchants preparing for agent-led commerce, that distinction matters. An AI agent that makes the right recommendation once is not necessarily an agent that can do it reliably.

Source: Visa - Agentic Commerce Benchmark, November 5, 2026

What did Visa actually measure?

This was not simply a test of whether AI could find a product or complete a purchase.

Visa designed the benchmark to evaluate two capabilities:

  • Understanding the shopper: The AI must uncover every requirement defined in the scenario through conversation, including details the customer does not initially disclose.
  • Making the right decision: It must select a product that meets those requirements or decline to buy if no suitable option exists.

A run only passes when both conditions are satisfied.

Consider Visa’s TV-shopping scenario. The shopper initially provides incomplete information. The AI must ask questions to uncover the budget, format compatibility, HDR10+ support and gaming requirements before choosing a product.

Finding a TV that looks suitable is not enough. The agent must establish why it is suitable.

Three numbers reveal the reliability gap

42.8% - Test runs failed

Across the 22 models and 15 shopping scenarios, only 57.2% of runs passed. The best-performing models reached 82.2%, while the weakest achieved 28.9%.

34.2% - All three attempts succeeded

Every model received three attempts at each scenario. Yet all three attempts passed in only 34.2% of cases.

This is the more important reliability signal. A model may understand a shopper's needs and make the right decision once, then fail when asked to repeat the same task.

54% - Failed runs still ended in a purchase

More than half of unsuccessful runs still resulted in the AI making a purchase. Visa says this happened either because the selected product did not meet the identified requirements or because the AI had not uncovered all the customer's requirements.

That distinction matters: an AI can fail without recognizing that it should stop.

Source: Visa's published benchmark results

Asking better questions mattered more than simply spending more

Visa found a clear relationship between requirement discovery and benchmark performance.

Model groupPass rate
Five models uncovering the most hidden buying factors70.2%
Five models uncovering the fewest44.4%
Five most expensive models per run67.1%
Five least expensive models per run50.2%

The results suggest that uncovering what shoppers actually need is an important part of successful AI shopping.

More expensive models performed better on average, but asking the right questions showed a substantial performance difference too.

These are benchmark comparisons, not proof that asking more questions alone will guarantee better results.

What this means for Shopify merchants

Visa tested AI models, not Shopify stores. It did not evaluate product-page quality, catalog structure, product feeds or merchant checkout systems.

But its findings raise a practical question for merchants: Can an AI agent find enough reliable product information to establish that an item matches a shopper's requirements?

In the TV scenario, those requirements include compatibility, HDR10+ support and gaming capabilities. These are not generic marketing claims. They are specific product attributes that need to be verified.

For merchants, that creates three areas worth checking:

Product specifications

Make important attributes explicit. Compatibility, dimensions, supported formats, materials, operating requirements and other purchase-critical details should not be buried in vague descriptions.

Structured product information

Keep product attributes consistent across product pages, variants, metafields and catalog feeds. An attribute that appears in one place but conflicts with another creates ambiguity.

Information gaps

Review the questions an AI shopping assistant would need to ask before recommending a product. If the answer cannot be found in your published product information, the catalog may be missing something important.

These are practical recommendations drawn from the benchmark's requirements-discovery problem, not findings that Visa directly tested on Shopify stores.

AI shopping still has a trust problem

The reliability issue also sits within a wider consumer-confidence gap.

Visa's September 2026 Trust Index found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted generative AI to handle payment transactions on their behalf.

People may be comfortable using AI to explore products without being ready to let it make purchasing decisions independently.

Visa plans to expand its benchmark to more product categories and eventually include checkout, payment and order placement. Those stages have not yet been evaluated in the first round.

Source: Visa Trust Index, September 9, 2026

What merchants should do now

The benchmark does not mean Shopify stores need to rebuild their checkout for AI agents today.

A more useful first step is to examine how clearly your catalog communicates product suitability.

  • Pick a product category where compatibility or specifications influence purchasing decisions.
  • Explore Visa Payments Frontier and watch the benchmark replays.
  • List the questions the AI needs to ask before it can make a correct recommendation.
  • Check whether your product pages and structured catalog data answer those questions clearly and consistently.

The next stage of agentic commerce will not be determined by how many products AI can find. It will depend on whether AI can understand the requirements, identify the right product and make that decision consistently.

Discuss on GemCommunity