We Compared AI Fashion Search With Traditional Site Search

The headline describes the study FashionMember plans to run, not a completed platform contest. We have built the query protocol, relevance ledger, and deterministic audit. We have not submitted those queries to an authorized live storefront, observed shoppers, or verified that an AI interface outperforms conventional search.

That distinction matters because a conversational answer can feel better while retrieving worse products. A keyword result page can look primitive while exposing more of the catalog, preserving filters, and making availability easier to verify. The comparison has to measure the same task, not the drama of the interface.

Start with one frozen catalog

Both conditions must use the same product snapshot. Archive every product and variant with a durable ID, title, category, material, color, pattern, fit, size system, care, price, availability, image, and last-updated time. Save the source export and hash it.

If the AI system has a richer feed than the conventional engine, the study tests data access as well as retrieval. That can be an operationally useful result, but it is not proof that the ranking method alone is better. Run a second analysis after the conventional index receives the same governed attributes.

Shopify’s current predictive-search documentation shows how even a familiar implementation has explicit fields and limits. Its Storefront API can return products, collections, pages, articles, and query suggestions, with controls for searchable fields and unavailable products. Shopify also documents term parsing, partial matches, typo tolerance, and variant-specific behavior. “Traditional search” is therefore not one universal baseline; its configuration must be archived.

Build queries from real tasks

A test made only of exact product names will favor literal search. A set made only of long lifestyle requests can favor a system designed to rewrite queries. Use a balanced taxonomy:

• exact product and attribute requests;
• category plus price or availability constraints;
• material, care, climate, or occasion needs;
• fit and variant requirements;
• negative constraints such as “not dry clean only”;
• substitutions for unavailable products;
• ambiguous discovery prompts;
• misspellings, synonyms, and regional language;
• no-good-answer cases where abstention is correct.

Freeze the original text. If an AI condition asks a follow-up question, record the question and the shopper’s scripted answer. Do not silently give one condition extra information.

Include product claims carefully. A query for waterproof outerwear, a hypoallergenic material, or a garment made in a particular country cannot be satisfied merely because generated copy says so. FTC advertising guidance requires truthful, non-deceptive, substantiated claims. The evaluator should open the source product record and confirm the evidence.

Judge relevance before seeing the rankings

Qualified merchants should label which frozen product IDs satisfy each task and why. Use at least two reviewers for a meaningful live study, record disagreements, and adjudicate against the source data. Relevance can be graded: fully satisfies, partially satisfies, useful alternative, or not relevant.

The relevance set must include variant truth. A dress may be relevant in style but unavailable in the requested size. A visually similar jacket may fail the care constraint. An AI explanation cannot convert a non-matching SKU into a match.

Keep the reviewers blind to condition when possible. Remove interface styling and system labels from the ranked lists. Otherwise a reviewer may reward the more fluent presentation.

Measure retrieval and the journey

For every query, capture:

• whether any relevant item appears in the first positions;
• recall within the visible result set;
• reciprocal rank of the first relevant item;
• graded ranking quality such as NDCG when judgments support it;
• catalog coverage and repeated concentration on a few popular products;
• unsupported attributes or claims;
• unavailable, duplicate, or wrong-variant results;
• response time, failure rate, and cost;
• number and quality of clarifying turns;
• keyboard, screen-reader, zoom, and mobile usability;
• path from result to the correct product variant.

Amazon Personalize’s official evaluation documentation is useful measurement context even though it describes recommendations, not this FashionMember search study. It defines rank-aware metrics, coverage, precision at K, and the danger of comparing models trained on different data. Its impact guidance also separates offline model metrics from online interaction outcomes.

A high offline score does not guarantee customer success. A live test should measure search exits, refinements, product views, add-to-cart events, selected variants, purchases, cancellations, and returns, with a predeclared attribution rule. A click is not a sale, and a sale is not proof that the shopper received a suitable product.

Accessibility belongs in the comparison

An AI answer that requires a mouse or moves keyboard focus unpredictably is not an improvement. W3C’s combobox pattern describes accessible names, popup relationships, autocomplete behavior, keyboard interaction, and the ability to dismiss suggestions without losing the previous value. Shopify’s predictive-search guidance likewise points implementers to accessible listbox patterns and clear controls.

Test both conditions with the same devices and assistive-technology plan. Record whether dynamic results are announced, headings explain groups, price and availability are perceivable, focus remains visible, and the user can return to the query.

A reproducible fictional fixture

FashionMember created eight invented fashion queries and fictional product IDs in content/data/FM-003-fashion-search-comparison.csv. Every row uses the same made-up catalog hash and contains five traditional and five AI result IDs. The script scripts/fm003-fashion-search-comparison.php checks the fixed snapshot, unique rankings, review flag, and fictional boundary, then calculates hit rate, mean recall, and mean reciprocal rank at five.

In this deliberately constructed fixture, both conditions achieve a hit in the first five results for all eight queries. The traditional list has mean recall at five of 0.812 and MRR at five of 0.750. The AI list has 1.000 for both. Those values were designed to prove the arithmetic and show why several metrics are needed. They are not outputs from Shopify, Google, Amazon, OpenAI, a FashionMember system, or any retailer. They provide no evidence that AI search is better.

What can go wrong in a live test

The query set can leak into tuning. Relevance labels can favor the team’s preferred merchandising logic. One condition can receive fresher inventory. Personalization can create different results for repeat sessions. An AI system can cite product facts from a stale page instead of the frozen feed. A conventional interface can expose filters that are omitted from a stripped-down comparison.

Control cookies, location, language, login status, inventory, price, device, and test order. Preserve screenshots or structured result exports where terms allow. Record model, index, application, theme, and API versions. Repeat at planned intervals because both systems can change.

NIST’s AI Risk Management Framework calls for testing in the intended context, documenting privacy and fairness risks, using relevant benchmarks, recording uncertainty, involving domain experts, and monitoring deployed behavior. Search evaluation should include poor-result and no-result cases, not only the prompts that make the demo look intelligent.

The live gate remains open

To publish the promised comparison as a result, FashionMember must obtain an authorized storefront and product export, freeze both conditions, recruit qualified relevance judges and consented users where applicable, preregister the query taxonomy and metrics, run accessibility and claim checks, archive configurations and outputs, and complete privacy, security, merchandising, legal, and statistical review.

The useful question is not whether AI can write a persuasive answer. It is whether a shopper can reach an available, evidence-backed, appropriate variant more reliably—and whether the retailer can explain how that conclusion was measured.

Sources and verification

• Shopify Storefront API: predictiveSearch — official current fields, arguments, resource types, limits, and unavailable-product controls.
• Shopify: Add predictive search to a theme — official matching behavior, implementation, and accessibility context.
• Shopify API search syntax — official query grammar and term behavior.
• W3C WAI-ARIA Authoring Practices: Combobox Pattern — official interaction, keyboard, role, and state guidance.
• Amazon Personalize: Evaluating a recommender — official rank-aware metric definitions and comparison limitations; used as measurement context, not search-test evidence.
• NIST AI RMF Core — official voluntary guidance on context, measurement, privacy, fairness, uncertainty, expert input, and monitoring.
• FTC advertising and marketing basics — official truth, non-deception, and substantiation guidance.