Asking for sources changed which retailers AI shopping assistants recommended
In a paired exmxc study, asking for supporting URLs increased visible evidence links and reduced Amazon’s appearance in top-five recommendation lists.
More evidence links, fewer Amazon recommendations
When exmxc asked AI shopping assistants for supporting URLs, their answers displayed an average of 3.83 more distinct evidence URLs. Amazon appeared in 18.1 percentage points fewer top-five lists, even as Amazon-owned evidence links appeared more often. A retailer can become more visible as a source while appearing less often as a recommendation.
What we tested
Retrieval Attribution v0.1 captured 192 responses across eight U.S. shopping intents, four assistants and three replicates. For each intent, platform and replicate, we paired a natural shopping prompt with a version that explicitly asked for verifiable supporting URLs or an acknowledgment that the information could not be verified. The prompts, order rules and analysis plan were frozen before collection. Results below describe this study’s shopping intents and sessions.
| Assistant | Pairs | Natural prompt | Asked for sources | Change |
|---|---|---|---|---|
| ChatGPT | 24 | 1.92 | 8.00 | +6.08 |
| Gemini | 24 | 0.79 | 4.46 | +3.67 |
| Claude | 24 | 5.33 | 9.21 | +3.88 |
| Perplexity | 22 | 24.32 | 25.82 | +1.50 |
| All pairs | 94 | 7.74 | 11.57 | +3.83 |
The exploratory 95% interval for the pooled increase was +3.02 to +4.62 URLs per response, using 10,000 bootstrap resamples clustered by shopping intent. Interfaces expose different kinds of source lists. When we omitted identifiable source-panel-only entries, the pooled increase was still +3.82 URLs per response.
Where Amazon and Walmart appeared
| Measure | Natural prompt | Asked for sources |
|---|---|---|
| Amazon in the top five | 75 | 58 |
| Walmart in the top five | 57 | 50 |
| Amazon-owned evidence URL visible | 12 | 26 |
| Walmart-owned evidence URL visible | 15 | 30 |
Amazon dropped out of 19 top-five lists and entered two when sources were requested. The exploratory 95% interval for the change was −26.4 to −8.5 percentage points. In the natural arm, the difference between Amazon’s and Walmart’s source visibility was small and its direction varied by platform. The retailer-access audit delivered one of 40 planned product-page observations, which leaves the proposed crawler-access explanation open.
Coding and limits
We counted affirmative retailer recommendations and omitted entries that only said the retailer’s information could not be verified. We distinguished evidence URLs from purchase-only links, using how the answer presented each link. The independent coding check covered 24 responses across all four platforms, both prompt arms and all three replicates. It identified verification-only retailer rows and evidence-section links that needed consistent treatment. We applied those corrections before computing the published figures.
Both analyses omitted one pair because the first completed answer changed in the interface and was not preserved. The source analysis also omitted a pair with incomplete source capture. The top-five analysis instead omitted a pair with a truncated ranking. The evidence-URL analysis measures what was displayed. Destinations were not independently verified, so the preregistered verified-link endpoints remain unscored. These captures cannot reveal which pages an assistant read, whether a retailer feed was involved, or why the assistant selected a particular retailer.