Agents look cheap in a demo because the demo takes three steps and the system prompt fits on a slide. Production agents take twelve steps, drag tool results back into context, and quietly multiply your input tokens.
That is the multi-step agent token tax. You think you bought a chatbot. You accidentally bought a research assistant that re-reads its own homework on every turn.
Why loops explode the meter
A normal completion is: system + user → answer. Billed once.
An agent turn is often:
- System + user + instructions → model decides to call a tool
- Tool result appended → model thinks again
- Another tool → another completion
- Final answer
Each hop resends the growing transcript. Input dominates. Output looks small until a reasoning model “thinks” for two thousand tokens between tools.
If step one sent 2,000 input tokens, step four might send 8,000–15,000 because the tool payloads (search hits, JSON records, page scrapes) piled up. Four steps is not 4× one step. It is a triangle number of context.
A worked sketch (order of magnitude)
Suppose a store ops agent:
- Stable system prompt: 1,200 tokens
- User ask: 80 tokens
- Each tool result: ~1,500 tokens
- Model “scratch” output between tools: ~200 tokens
- Six tool hops before the final answer
By the last hop you are carrying the system prompt, the user ask, five prior tool blobs, and the chatter between them. It is easy to land north of 10k–20k input tokens for a single user question — before the final prose answer.
Run that through the AI API cost calculator with your real model mix. Then multiply by daily questions. The demo that felt like pennies becomes a line item you should have put in the proposal.
Caps that save the product
I do not ship unbounded agents for merchant-facing work. Guardrails I use:
- Hard step cap (often 4–8). On hit: summarize what you know, ask the user, or hand off to a human.
- Tool result truncation. Store the full payload server-side; send the model a slim summary plus IDs.
- Retrieve, do not paste. Policies and catalogs belong in RAG chunks, not a full PDF in the prompt every hop.
- Cheap model for routing and tool choice; stronger model for the final user-visible answer.
- Stop conditions for loops that re-query the same tool with the same arguments.
An agent that can browse forever will. Give it a budget in steps and in tokens.
Agents versus workflows
Many “agent” requests are workflows with lipstick. If the steps are knowable — fetch order, check policy, draft reply — encode the graph. Use an LLM at the nodes that need language, not as the operating system.
Reserve open-ended agents for messy research, internal ops with a human in the loop, or problems where the tool set truly varies. Storefront support and catalog pipelines usually want boring graphs with escape hatches.
Caching and summarization
Prompt caching helps when the system prefix is stable — good. It does not erase the tax of stuffing fresh tool results every hop.
Summarize older steps aggressively: “Tools already tried: X, Y. Key facts: …” Keep raw blobs out of the context window unless the next decision needs them. Your future invoice will thank you.
What I put in the SOW
When a client asks for an agent, the proposal lists:
- Default model and upgrade model
- Max steps per request
- Estimated tokens per successful task (and the 3× case)
- What happens on cap / empty tool / low confidence
- Human handoff path
If we cannot agree on a step cap, we are not ready to build. That conversation is cheaper than a surprise bill after a viral week.
Tool design that respects the context window
Every tool schema you expose is also a prompt tax — descriptions, parameters, enums. Bloated tool lists teach the model to waffle. I keep tools few, names boring, and responses slim. Prefer get_order(id) returning a tight summary over search_everything(query) returning novels.
When a tool must return a large payload (full policy PDF text), stash it and return {ref, excerpt, chars}. Let a second dedicated retrieval step pull only what the next decision needs. Agents that pass megabyte JSON between hops are how you meet your rate limit and your CFO on the same afternoon.
Parallel tools are not free
Some frameworks fire multiple tools at once. Latency improves; token accounting still adds every result into the next thought. Parallelism is a performance tactic, not a discount. Cap parallel fan-out the same way you cap depth.
Human-in-the-loop as a cost control
For irreversible actions — emails, store credit, draft orders — require approval before the agent continues. That pause prevents expensive wrong loops and protects the brand. It also clarifies product boundaries: the agent prepares; the human commits.
I write that into the SOW so nobody expects a fully autonomous refund robot on day one.
Evaluating agents without fake win rates
I track: median steps per success, percent hitting the step cap, percent handed to humans, token p95 per task, and a small set of golden tasks that must stay green. I do not publish “94% autonomous resolution” without defining resolution. Internal dashboards can be honest even when marketing copy stays humble.
When to delete the agent
If after a few weeks the successful paths are always the same four steps, replace the agent with a workflow. You will keep quality, cut tokens, and sleep better. Agents are a probe for unknown structure. Workflows are how known structure should run in production. Model both with the AI API cost calculator before you fall in love with autonomy.
Prompt patterns that shrink the triangle
Two patterns cut input growth:
- Scratchpad outside the chat — keep working memory in server state; tell the model only the current goal and the last tool summary.
- Deterministic pre-steps — fetch order or customer tier in your code before the LLM runs, then ask the model to draft language. Fewer tool hops, fewer tokens.
I still use agents when the path is unknown. I just do not pretend every customer email needs a twelve-step research loop. Price the loop with the AI API cost calculator, put the step cap in the UI for internal tools, and make the cap visible in admin so operators understand why the agent stopped and asked a question.
Design the agent like it pays rent
Multi-step agents are powerful and easy to romanticize. Treat context as expensive memory, not free whiteboard space. Cap the loop, slim the tool payloads, and price the triangle — not the happy-path screenshot.
If you want help turning an agent idea into a capped workflow with a monthly ceiling, that is core work for me: Shopify integrations and AI systems that are allowed to stop.