Retrieval-augmented generation is the right default when truth lives in your docs — policies, visa checklists, product manuals, SOPs. It is also easy to overbuild into a science fair.
Here is how I ship a budget RAG pipeline: small models where they work, refuse when retrieval is empty, and a cost ceiling you can defend in a proposal. Some of this came from document-QA work in the VisaChamp shape — not invented client metrics, just the engineering lessons that repeated.
What “budget” means here
Budget is not “worst model, hope for the best.” It means:
- You know tokens per question at p50 and p95
- Embeddings and re-indexes have a cadence and a price
- The system fails closed instead of hallucinating policy
- You are not paying a flagship model to rewrite the same PDF chunk five ways
Run the generation side through the AI API cost calculator. Add embedding volume separately — it is usually smaller than generation, but refresh jobs sneak up.
Chunking without the lore
I start simple:
- Split on headings and logical sections before raw token windows
- Aim for chunks that answer one question, not a whole chapter
- Keep a little overlap so sentences are not cut mid-thought
- Store source title, URL or file id, and section path with every chunk
Giant chunks waste context. Tiny chunks retrieve noise. I tune with a golden set of twenty questions before I tune with vibes.
For tables and forms (common in visa and compliance PDFs), I preserve structure in the text representation instead of flattening into poetry. The model cannot cite what you destroyed.
Retrieve less, not more
Stuffing ten chunks “to be safe” is how you burn input tokens and confuse the model. I prefer:
- Top k small (often 3–5)
- A relevance floor — if scores are weak, refuse
- Optional rerank only when the corpus is large and quality demands it
Empty or weak retrieval → “I do not have that in the docs” plus a human handoff. That single rule prevents more support disasters than a bigger model.
Generation on a leash
Prompt pattern that works:
- Answer only from the provided chunks
- Quote or cite the section id
- If the chunks do not contain the answer, say so
- Cap max output tokens
Use a mid or small model for most turns. Escalate when the user is angry, the topic is high risk, or the first draft fails a citation check. You do not need a flagship model to say “the fee is on page 2 of the checklist” when the chunk is sitting there.
Refresh cadence and ops
Docs change. Stale embeddings are polite hallucinations.
Budget ops look like:
- Re-embed on publish or nightly for active corpora
- Delete vectors for removed pages
- Log which chunk ids were used when an answer was wrong
- Keep eval questions that must pass before you widen traffic
I do not build a multi-agent research swarm to answer “what is the refund window.” I build a boring index and a strict prompt.
Cost levers that actually matter
- Shorter system prompts; put policy in the index
- Smaller k and refuse-when-empty
- Cheap model default; escalate rarely
- Prompt caching on the stable instruction prefix
- Batch re-embeds off-peak
- Cap agent tool loops if you wrap RAG in an agent (see the token tax post)
What I skip on v1
- Fancy graph RAG
- Fine-tuning for voice (usually later, if ever)
- Multimodal everything when the PDFs are text-first
- A custom vector DB when a managed store is fine at your scale
Complexity is a subscription you pay in incidents.
Proposal language I reuse
“We will index N document types, answer with citations, refuse when retrieval is empty, hand off to a human on low confidence, and keep generation under $X/month at the planned volume including a 3× spike case.”
If a stakeholder wants unconstrained chatting over legal PDFs with no refuse path, that is not a budget RAG problem — that is a risk problem.
Indexing product catalogs versus policy PDFs
Catalog RAG and document RAG share bones but not skin. Products change with inventory and seasonality; policies change with legal review. For catalogs I lean on structured fields plus short text blobs, and I am aggressive about SKU ids in metadata. For policies I lean on section paths and version dates.
Mixing both corpora in one undifferentiated index is how “return window” answers start citing a jacket description. Separate indexes or strong metadata filters are cheap insurance.
Citations users can trust
“According to our docs” is weak. “Refunds → Domestic returns → paragraph 2” is stronger. I require the model to return chunk ids that the UI turns into links. If the id is missing, the answer does not ship. That is an evaluation rule as much as a UX rule — related to how I treat storefront copy claims.
Multilingual and messy scans
Budget pipelines struggle with scanned PDFs and mixed languages. Before fancy models, I ask whether OCR quality is good enough. Garbage chunks guarantee garbage answers. Sometimes the budget decision is “pay for better PDF cleanup” rather than “pay for a bigger LLM.”
Security and tenancy
If you embed client documents for multiple merchants, partition vectors by tenant. Retrieval leakage is a trust-ending bug. Budget does not mean shared everything. It means lean tech with correct boundaries.
A minimal architecture I reuse
- Ingest → clean → chunk → embed → store with metadata
- Query → embed → retrieve → relevance gate → generate with citations → log redacted traces
- Eval set on a schedule
- Dashboard: refuse rate, citation miss rate, token p95, cost vs ceiling
That is enough for VisaChamp-style document QA and for store policy bots. You can always add rerankers later when the corpus — and the budget — earn them. Keep generation estimates honest with the AI API cost calculator.
Human review sampling after go-live
Even with refuse-when-empty, sample production answers weekly. Ten randomly selected threads beat a gut feel. Tag errors: retrieval miss, bad chunking, model overreach, stale doc. Each tag points to a different fix. Budget RAG stays budget when you fix the right layer instead of jumping to a larger model every time someone forwards a bad screenshot.
Publish the sampling habit in the runbook so it survives handoff. Pair cost dashboards with the AI API cost calculator assumptions you sold, and adjust the ceiling when the corpus grows.
Ship truth, not theatre
A budget RAG pipeline is chunking discipline, a refuse rule, and honest metering. That is enough to help merchants and operators trust the bot. Fancy diagrams can wait.
If you need document QA or store-policy RAG scoped with a ceiling, that is work I do from Karachi — practical retrieval, Shopify-aware support flows, and no fake accuracy percentages in the pitch deck.