Visa: AI Models Failed Nearly Half of Tests to Make Suitable Shopping Decisions

Visa's Agentic Commerce Benchmark, designed to assess AI models from 12 labs across 15 shopping scenarios, found that the 22 models tested achieved an average pass rate of 57.2 percent. These tests evaluated the AI's ability to identify customer needs through conversation, make suitable purchasing decisions for items like TVs and laptops, and crucially, to avoid purchases when no appropriate product was available. Individual model performance varied significantly, from 28.9 percent to 82.2 percent.

A key finding was that more than half (54 percent) of the failed attempts still resulted in a purchase. This occurred either because the AI selected a product that did not meet identified requirements or because it failed to uncover all of the customer's needs. Models that demonstrated a greater capacity to identify more hidden customer requirements generally performed better, achieving an average pass rate of 70.2 percent compared to 44.4 percent for those that identified the fewest.

Consistency proved to be a significant challenge, with all three attempts at a given scenario successful in only 34.2 percent of model-scenario combinations. While higher-cost models showed better average performance, passing 67.1 percent of attempts versus 50.2 percent for the least expensive, the overall results suggest that current AI models are not consistently reliable for nuanced retail decision-making. Visa plans to expand the benchmark to include more product categories, checkout, and payments, making the research publicly available to boost industry standards.