AI Product Recommendation vs Brand Mention: What Shopify Merchants Should Measure

A brand can appear in an AI answer without any specific product being selected. A product can also be listed as an option without the answer saying that it fits the buyer’s needs. Only a positive, purchase-oriented suggestion of the tracked product should count as a recommendation.

OpenAI says shopping products are selected when ChatGPT considers them relevant to the user’s intent, and that not every available product is shown. Selection can consider the query, context, price, reviews, availability, and other product information. OpenAI’s shopping documentation

Counting every brand mention as a recommendation inflates visibility and hides whether a concrete product actually matched the shopper’s request.

Give every completed model test one primary outcome for the confirmed target product. Keeping the states mutually exclusive prevents the same answer from being counted twice.

Recommended

The answer positively suggests the tracked product for the buyer’s stated need.

Mentioned

The tracked product is identifiable, but the answer gives no positive recommendation.

Negative

The answer identifies the product and advises against it or states a material mismatch.

Not found

A valid completed answer contains no reliable reference to the tracked product.

Indeterminate

The answer is ambiguous or the product identity cannot be resolved with confidence.

Provider failures, timeouts, and blocked requests are not “not found.” They are test errors and should be excluded from product-outcome metrics.

Product identity should first be matched through stable evidence such as product URL, Shopify ID, handle, and full title. A language-model classifier is useful only when the deterministic signals remain ambiguous.

Product Recommendation Rate should count valid tests where the confirmed product was positively recommended. Product Mention Rate should count valid tests where it appeared at all, including recommendations, neutral mentions, and negative mentions.

Always show the raw counts and excluded tests beside the rate. A percentage without its denominator makes small or unreliable samples look stronger than they are.

Save the detected product, the relevant answer excerpt, the matching signal, and the confidence. Low-confidence cases belong in “indeterminate,” not automatically in “not found.” Historical runs without product classification should remain visible as legacy data but stay outside current rates.

Shopify’s own Catalog search preview is also directional: Shopify notes that agentic storefronts can re-rank results using their own logic. The preview can support a diagnosis, but it cannot reproduce the exact consumer experience. Shopify’s preview guidance

“Not found” proves only that the tracked product was absent from that valid response. It does not prove Catalog ineligibility, a crawling problem, or poor product data. Shopify explicitly separates Catalog syndication, crawling, and merchant feeds. Shopify’s discovery documentation

Diagnose in sequence: channel status, Catalog observation, product-data completeness, buyer-requirement coverage, and verified technical blockers. If none explains the result, report that no cause is proven instead of inventing one.

1. 1

   Confirm the target Shopify product before running the prompt.

2. 2

   Store the exact prompt, model, provider route, market, and timestamp.

3. 3

   Resolve product identity from URLs and product identifiers before semantic classification.

4. 4

   Save the outcome with a short answer excerpt and confidence.

5. 5

   Exclude provider errors and disclose indeterminate results separately.

Use the separate prompt-selection guide to build the test set, and the product visibility guide for the broader testing process.