I have watched good catalogs get flooded with AI descriptions that look fine in a spreadsheet and feel wrong on the product page. The model was not broken. The review step was missing.
Evaluating AI product copy is not a vibe check. It is a boring rubric you run before anything reaches the storefront — claims, banned words, required schema, and brand voice. I would rather reject fifty drafts than explain to a merchant why half their SKUs now say “luxury” about a mid-range water bottle.
Why storefront copy fails differently from chat demos
In a chat demo, one polished paragraph feels like progress. On Shopify, that paragraph sits next to price, shipping, reviews, and a metafield block that feeds Google Shopping. One invented material claim can become an ad complaint. One missing care instruction can become a return.
So I treat generation as draft labor and evaluation as the product. The model proposes. A checklist decides.
The four-part rubric I actually use
I keep it short enough that a junior merchandiser can apply it without a meeting.
1. Claims that need a source
Anything measurable or regulated gets flagged:
- Materials and composition percentages
- Certifications (organic, OEKO-TEX, FDA-adjacent language)
- Performance numbers (“lasts 48 hours,” “blocks 99%”)
- Medical or safety implications
If the vendor feed does not contain the claim, the draft must not invent it. Soft language is allowed (“designed for everyday wear”). Hard language needs a field.
When I wire this into a pipeline, the evaluator is often a cheap model with a JSON schema returning each claim, whether it is supported, and a short reason. Humans still spot-check the first few hundred SKUs and anything with certification language.
2. Banned words and stuffed SEO
Every brand has a list. Mine usually starts with:
- luxury, premium, revolutionary, game-changing (unless the brand owns that voice)
- best, #1, unmatched (comparative claims without proof)
- Keyword stuffing in the first sentence
I paste real voice examples into the prompt and reject drafts that ignore them. Prompting alone is not enough. The rubric catches the drift when the model gets bored and starts writing brochure copy.
3. Schema and metafield completeness
For catalog work, “good prose” is incomplete if the structured fields are empty. I check:
- SEO title length and uniqueness
- Meta description without duplicate boilerplate
- Required metafields (care, fit, material, country of origin)
- Bullet list length (I often cap at 4–6)
Structured outputs help here — return JSON the admin can write to Shopify, then render Liquid from metafields. Free-text essays that “also happen to include bullets” are how pipelines break at 2 a.m.
4. Brand voice on a short sample set
I do not score “creativity.” I compare against five approved product pages: sentence length, second person vs third, humor allowed or not, how specs are introduced. If the draft reads like a different brand, it fails even when claims are clean. Flagship models help on rewrite; they are not required for the fail/pass gate.
A workflow that fits a real catalog night
Typical sequence for a merchant with a dirty vendor CSV:
- Normalize source fields (title, vendor bullets, attributes).
- Generate with a small model plus a schema (description, bullets, SEO, metafields).
- Run the rubric automatically — claim check, banned list, required keys.
- Promote failures to a stronger model once, with the failure reasons in the prompt.
- Human review on a sample (new category, high-AOV SKUs, certification language).
- Write to draft products or a staging metafield, not live.
Retries belong in the budget. If you model only the happy path, plug the volume into the AI API cost calculator and add 20–40% for rewrites. Evaluation tokens are usually cheap next to a bad claim reaching Google Shopping.
What I refuse to automate as a quality score
I avoid single-number quality scores. “87/100 brand alignment” looks scientific and means nothing in a dispute with a brand manager. Pass/fail on claims, pass/fail on schema, and a short human note on voice scale better.
I also refuse to ship copy that invents social proof (“loved by thousands”) or invents shipping promises. Those belong to policy pages and systems that know inventory — not the copy generator.
Connecting evaluation to cost and risk
Evaluation adds tokens. A cheap classifier on every draft is usually worth it: you catch invented claims before a human spends twenty minutes hunting them. A second-pass rewrite on roughly a third of SKUs is also a line item — model it, do not discover it on the invoice.
If your pipeline is “one flagship call per product, no cap, no checklist,” you are buying both spend and risk. Start small, measure fail rates by category, then widen. Fashion accessories fail differently from supplements; your rubric should know that.
How I sample before trusting the fail rate
I never judge a generator on ten hero SKUs the brand already loves. Those are the easy ones. I pull a stratified sample:
- New category with thin vendor data
- Best sellers (AOV and traffic)
- Edge SKUs with certifications or restricted claims
- Bundles and gift sets where attributes collide
- Non-English source fields if the catalog mixes locales
Run the rubric. Count fails by reason code: unsupported claim, banned phrase, missing metafield, voice mismatch, schema parse error. The distribution tells you whether to fix the prompt, the source feed, or the model tier.
If sixty percent of fails are missing care instructions because the CSV never had them, that is not an LLM problem. That is a data problem wearing an AI costume. Fix the feed mapping before you burn tokens on rewrite loops — and keep using the AI API cost calculator so the rewrite tax stays visible.
Storefront QA that is not a spreadsheet forever
Spreadsheets work for the first batch. They do not work when merchandisers live in Shopify admin. I prefer:
- A staging metafield like
custom.ai_review_statuswith valuespass,fail,needs_human - A simple admin filter or a thin internal view listing fails with reason codes
- Draft theme preview on a development theme so marketing can read copy in context next to price and images
Context changes judgment. A paragraph that looks fine in a cell can clash with a lifestyle image or duplicate the title. Evaluating AI product copy includes looking at the page, not only the string.
When to let a stronger model rewrite
Promotion rules I like:
- Fail on claims or schema → fix with structured instructions, often still on a small model
- Fail on voice after two small-model attempts → one flagship rewrite with approved examples in the prompt
- Fail on regulated language → human only; do not keep sampling the model until it invents safer wording
Unlimited regenerate buttons teach teams to spend. A ticket that says “failed: unsupported SPF claim” teaches teams to fix the source.
Closing the loop with shopping channels
If descriptions sync to Google Merchant Center or Meta catalogs, your rubric is also a channel-risk tool. Invented attributes become disapprovals. Keep the same claim rules in the feed path. One source of truth for material and certification fields beats three prompts that drift.
Ship the rubric before you scale the generator
The storefront does not care that the model is impressive. It cares that the description matches the product in the warehouse. Write the banned list, the required metafields, and the claim rules first. Then turn the model on.
If you want help wiring that checklist into a Shopify catalog pipeline with a real monthly ceiling, that is the work I do from Karachi — themes, metafields, and AI that is allowed to fail closed.