How to Evaluate an AI Vendor Before Signing a Contract

The worst time to discover how an AI vendor handles your data is after the design team has uploaded an unreleased collection, the service has become part of the weekly merchandising process, and the contract is approaching renewal.

Vendor evaluation should begin before the demonstration. Write down the fashion task, the people and information it touches, the decision it will influence, and the cost of failure. Then ask the vendor for evidence that fits that use—not a generic assurance page.

The objective is not to eliminate risk. It is to decide whether the expected benefit justifies the documented residual risk, under controls the company can actually operate.

Step 1: define the use case narrowly

“Generative AI for ecommerce” could mean rewriting approved product facts, generating a fictional campaign concept, translating care instructions, answering customer questions, or changing prices. Those tasks have different failure costs.

Create a one-page use-case record:

• intended user and reader or customer outcome;
• approved inputs and prohibited inputs;
• expected output and any decision it affects;
• affected customers, workers, suppliers, creators, and partners;
• required human review and override;
• material legal, privacy, security, intellectual-property, accessibility, labor, and reputation questions;
• success measures, failure thresholds, and stop conditions.

If the use cannot be explained clearly, the team is not ready to compare vendors.

Step 2: draw the data flow

Ask where data enters, where it is processed, who can access it, what subprocessors receive it, where it is stored, how long it remains, whether it trains or improves any model, and how deletion works. Include prompts, uploads, outputs, feedback, logs, embeddings, support tickets, and integration data.

Map real fashion information: unreleased designs, tech packs, wholesale price lists, supplier contacts, customer profiles, fit feedback, model releases, employee records, and campaign plans. “We do not store prompts” does not answer whether another service retains uploads or logs.

NIST’s AI RMF Core calls for mapping third-party software and data risks and documenting controls. The map should be understandable to the people who own the business process, not only the security team.

Step 3: separate evidence from sales language

For every important promise, ask what evidence supports it and where that promise appears contractually.

If the vendor says its tool reduces returns, request the study design, sample, baseline, product category, time period, exclusions, and definition of “return.” If it says it is more accurate than human work, ask which humans, tasks, datasets, measures, and uncertainty were involved.

The FTC has warned businesses not to exaggerate AI capabilities, make unsupported comparative claims, or ignore reasonably foreseeable risks. A polished demonstration does not substantiate a business outcome.

Security evidence also needs context. A certification can be useful, but it has a scope, audit period, exclusions, and set of systems. Ask for the current report or appropriate summary, remediation status, penetration-testing approach, vulnerability disclosure route, identity and access controls, encryption design, incident history, and breach-notification process. Qualified reviewers should determine what is appropriate for the data and use.

Step 4: perform supplier due diligence

NIST SP 1326, published in July 2026, describes due diligence as research into pertinent information about a supplier or product so an organization can make informed acquisition decisions. Its scope is ICT cybersecurity supply chains, and its assessment components include ownership or control, provenance, resilience, foundational cyber practices, and supply-chain tiers.

A small fashion company can adapt the questions without pretending to run a federal assessment:

• Who owns and controls the supplier?
• Which critical services and models does it depend on?
• How would an upstream outage or policy change affect the product?
• Does the company publish meaningful security and reliability information?
• What is its financial and operational capacity to support the promised term?
• Has it changed pricing, features, model providers, or data terms in ways relevant to customers?
• Can the brand verify key statements independently?

Unknown information is not automatically disqualifying, but it should remain visible in the decision rather than being converted into a neutral score.

Step 5: test representative fashion work

Run a bounded pilot with cleared or synthetic data before connecting production systems. Build cases from ordinary work and difficult edges.

For a copy tool, include missing fiber content, conflicting color fields, regulated claims, nearly identical SKUs, long care instructions, and prohibited urgency language. Count unsupported facts, omitted facts, variant confusion, editing time, and approval failures.

For image generation, test product fidelity, anatomy, protected marks, culturally sensitive styling, disclosure, and whether synthetic output could be confused with documentary product evidence.

For recommendations, test cold starts, sparse history, changed inventory, unusual sizing, and customers outside the most common patterns. Compare the tool with a simple baseline. Preserve inputs, settings, outputs, reviewer decisions, and version information so the test can be revisited.

A vendor-selected benchmark can inform the pilot but cannot replace it.

Step 6: inspect human control and impact

Who sees the output before it affects a person? Can they understand why the system made the recommendation? Can they correct, override, or stop it? Is there an alternative route for someone who cannot or does not want to use the automated path?

“Human in the loop” is not enough if the reviewer has seconds to approve hundreds of outputs or is punished for disagreeing. The contract and operating plan should support realistic review, logs, escalation, correction, and appeal where the impact requires them.

Include accessibility in the test. A customer-service agent that produces inaccessible links or prevents a person from reaching human support may reduce service even when its average answer score looks good.

Step 7: make the contract match the decision

The contract should address the points the team relied on: allowed use, data roles, retention, training and reuse, subprocessors, security measures, incident notice, audit evidence, service levels, model or feature changes, intellectual-property terms, claims handling, confidentiality, accessibility commitments, pricing, liability, termination, export, and deletion.

NIST SP 800-161 Rev. 1 frames cybersecurity supply-chain risk across the lifecycle and emphasizes visibility into how acquired products and services are developed, integrated, and operated. For a fashion buyer, the key translation is that procurement is not complete when the purchase order is signed. Risk continues through updates, integrations, renewal, and disposal.

Do not assume a website statement controls if the contract says something different. Qualified counsel and privacy, security, procurement, finance, accessibility, and labor reviewers should assess terms relevant to the specific deployment.

Step 8: design the exit now

Ask how to export prompts, templates, product mappings, evaluations, logs, and other business records. Define the format, timing, cost, and deletion evidence. List downstream workflows and integrations that would break.

Create a manual or alternative process for critical functions. A pilot can be successful and still be rejected if the brand cannot exit without losing essential records or operational continuity.

The NIST Cybersecurity Framework supply-chain guidance includes examples such as supplier requirements in agreements, verification, ongoing monitoring, vulnerability disclosure, and service-level expectations. Exit is part of that lifecycle discipline.

Use a scorecard without worshipping the score

FashionMember’s reusable worksheet is stored at content/resources/FM-047-ai-vendor-scorecard.md. It covers intended use, data flow, training and reuse, security, privacy, performance, human control, rights, accessibility, reliability, contract, and exit.

The sheet uses three practical states:

• Hold when high-impact evidence is missing.
• Test when a bounded pilot can answer defined questions safely.
• Proceed with controls only when named owners accept residual risk, contract language matches the evidence, and monitoring and exit routes are active.

A high numeric total should never cancel a critical failure. Unclear customer-data reuse, an impossible deletion process, or a consequential decision with no appeal can be a stop condition on its own.

What a scorecard cannot decide

This guide adapts public risk and supplier-management frameworks to fashion operations. It does not approve any vendor, prescribe contract language, or replace professional review. NIST SP 1326 focuses on ICT cybersecurity supply-chain due diligence; a real purchase may involve additional laws, industry duties, worker protections, accessibility requirements, and financial considerations.

Sources and verification

• NIST SP 1326: Due Diligence Assessment Quick-Start Guide — current supplier due-diligence structure and assessment components.
• NIST SP 800-161 Rev. 1 — lifecycle cybersecurity supply-chain risk management for acquired systems and services.
• NIST AI RMF Core — mapping third-party components, controls, measurement, human oversight, and ongoing management.
• NIST Cybersecurity Framework supply-chain resources — supplier agreements, monitoring, disclosure, verification, and lifecycle examples.
• FTC: Keep your AI claims in check — evidence for capability and comparative claims and attention to foreseeable risks.
• FTC: Advertising and marketing basics — truthfulness, non-deception, and substantiation.