Demos never return HTTP 429. Sale days do.
LLM rate limits are not a footnote. They are a product constraint: tokens per minute, requests per minute, and concurrent connections that look infinite until Black Friday traffic and a retry storm meet the same API key.
What a 429 actually means in an AI feature
Providers throttle by model and tier. You can be under your monthly budget and still get blocked for ten seconds because you bursted. Chatbots feel this as “the bot is thinking forever.” Catalog jobs feel it as a half-updated metafield run.
I treat 429 as expected weather, not an outage. The system should back off, avoid double-applying side effects, preserve user-visible progress, and alert when queue depth says you are about to miss SLA.
Backoff that does not make things worse
Naive retry: sleep one second, try again, hammer the API, amplify the outage.
What I ship instead:
- Exponential backoff with jitter so every worker does not wake in sync
- Respect
Retry-Afterwhen the provider sends it - Cap attempts (usually 3–5 for interactive; higher for overnight batch with a deadline)
- Separate budgets for user-facing versus background jobs
Interactive support turns should fail soft into a human handoff message after a short wait — not spin for ninety seconds. Batch catalog rewrites can wait minutes. Different paths, different patience.
Idempotency: the part people skip
Retries without idempotency create duplicate Shopify writes, double emails, and two assistant messages in the thread.
Patterns that help:
- Idempotency keys on write paths (same key yields the same result)
- Upsert metafields by namespace and key instead of append-only blobs
- Deduplicate outbound messages by conversation ID plus turn hash
- Store “already completed” markers before you acknowledge success to the UI
If your agent can call “create draft order” twice because the first response timed out, you need a key — not hope.
Queues beat inline bursts
For anything that is not a single user typing, I push work to a queue:
- Catalog description generation
- Embedding refresh
- Nightly eval suites
- Bulk rewrites after a vendor feed lands
Workers pull at a rate the provider tier can sustain. The storefront stays fast. The merchant sees a progress bar or a “ready for review” email, not a browser tab praying.
Sale-day support bots still need concurrency limits. Cap parallel LLM calls per shop. Shed non-critical features (nice-to-have upsell suggestions) before you shed “where’s my order.”
Timeouts, partial failures, and token tax
A model that streams for forty seconds and then dies mid-JSON is worse than a fast failure. I set hard timeouts, validate structured outputs, and retry only when safe.
For RAG support answers: if retrieval succeeded but generation returned 429, do not re-retrieve blindly on every attempt if the chunks are cached for that turn — you save latency and token cost. Model that cost in the AI API cost calculator; retries are a tax on both time and money. A fifteen percent retry rate is a fifteen percent tax on the generation line.
Observability that is useful at 2 a.m. Karachi time
Log, with PII redacted:
- Provider, model, status code
- Tokens in and out when available
- Queue wait time versus generation time
- Retry count and final outcome
- Shop or tenant ID — not raw customer message bodies by default
Alert on sustained 429 rate and on queue lag, not on every single throttle. One spike during a deploy is noise. Ten minutes of climbing lag during a flash sale is a page.
Designing for the tier you actually bought
If the business plan assumes flagship model quality for every turn, buy headroom or route cheap first. Multi-model routing helps: classify with a small model, escalate hard cases. That cuts tokens and spreads limit pressure across endpoints when providers separate them.
I also keep a kill switch: disable nonessential AI widgets from the theme without a full deploy when the provider is having a bad hour.
Multi-tenant and multi-shop gotchas
Agencies and productized services often share a provider account across shops. One noisy client can starve the others if you do not isolate quotas. I prefer per-shop API keys or clear concurrency partitions in the worker pool. Fairness is a product feature when you run AI for more than one storefront.
Likewise, do not let a bulk catalog job for Shop A sit on the same priority queue as Shop B’s live support. Separate queues or priority lanes. Sale-day traffic should not wait behind a thousand description regenerations.
Provider outages versus your outages
A wave of 500s is different from 429s, but the user experience rhymes. Circuit breakers help: after N failures in a window, open the circuit, serve cached FAQ answers or a static “we are routing you to a human” message, and probe half-open cautiously. Blind retries during a provider incident turn your workers into a self-DDOS.
Document the customer-facing copy for that state in advance. Support teams hate improvising mid-incident.
Idempotent webhooks and Shopify side effects
Shopify webhooks retry. Your LLM pipeline that “enriches then writes metafields” must tolerate duplicates. Combine webhook idempotency keys with LLM result caching keyed by product revision. If the product did not change, do not pay for another generation — return the prior enrichment. That saves money and reduces write races.
Load testing without melting your free tier
Before a campaign, I replay traffic shapes against staging with a cheaper model or a stub that returns canned JSON at production latency. You are testing queue depth, UI timeouts, and handoff paths — not model quality. Save the real-model soak for a smaller canary percentage on production with alerts armed.
Budgeting retries in the proposal
I literally add a retry percentage to the cost section. Ten to twenty percent is a starting assumption for flaky networks and validation failures; tune from logs. Put it next to the AI API cost calculator estimate so finance sees the same meter engineering sees.
Client communication when the bot is slow
Engineering can be correct and still lose trust if the storefront sits on a spinner. I surface partial UI states: “Still checking your order,” then a timed offer to email a human. That copy is part of the rate-limit design. So is training support staff that a 429 storm is not “the AI being dumb” — it is capacity.
After incidents I send a short note: what tripped, what we capped, what we will change in the queue. Merchants forgive weather. They do not forgive silence.
One more practical habit: keep a read-only status flag in the theme so you can disable the chat drawer without a full deploy when provider limits are red. Pair that with the cost view in the AI API cost calculator so finance and engineering argue from the same sheet after the spike.
Production checklist before you launch
- Backoff plus jitter plus Retry-After
- Idempotent writes
- Queue for batch; concurrency caps for chat
- Soft fail to human handoff on interactive paths
- Timeouts and structured-output validation
- Redacted metrics for 429 rate and queue depth
- Cost model that includes a retry percentage
Rate limits are how the API reminds you it is a shared meter. Design for that on day one and sale day feels like traffic — not like a meltdown.
If you want a support or catalog pipeline scoped with retries and a monthly ceiling, that is standard work for me: Shopify plus AI with production habits, not just a prompt in a notebook.