Building Agents That Don't Break in Production

Every agent project follows the same emotional arc. Day one, you wire a model to three tools and watch it plan, call them, and produce a sensible answer. It feels like magic. Day thirty, it has silently double-charged a customer, looped forty times on a malformed API response, and produced a confident summary of a document it never actually read.

Nothing was wrong with the model. What was missing was everything around it.

The demo is 10% of the system

An agent is not a prompt. It is a distributed system with a non-deterministic scheduler. Once you see it that way, the engineering requirements stop being surprising:

• Steps fail, so you need retries, and retries mean side effects must be idempotent.
• Runs are long, so state has to live somewhere other than a variable in a process that might restart.
• Actions cost money and touch real records, so you need budgets, permissions, and approvals.
• Behaviour changes with every model and prompt version, so you need regression tests.

None of that is exotic. It is the same discipline you would apply to a payment processor. The novelty is only that the control flow is chosen at runtime by a model.

Bound the plan

The most common failure we inherit is an unbounded agent: given a goal, allowed to keep going until it decides it is done. That works until the day it doesn't, and then it burns $400 in tokens discovering that an endpoint returns a 500.

Bound three things explicitly:

1. Steps. A hard ceiling on iterations, with a defined behaviour when it's hit: escalate to a human, don't silently return a partial answer as if it were complete.
2. Money. A per-run token and dollar budget, checked before each model call, not tallied afterwards.
3. Time. A wall-clock deadline. An agent that has been thinking for six minutes is not about to succeed.

These limits are also enormously useful in development, because they turn "it hung" into a specific, loggable event.

Make tools boring

Tools are the agent's only contact with reality, so they deserve more care than the prompt does.

Give every tool a strict typed schema and validate arguments before executing. A model will eventually pass a string where you expected a date, and you want that to be a clean validation error the agent can recover from, not a 500 halfway through a write.

Make every write idempotent. Pass an idempotency key derived from the run and step. This single change converts "the retry created a duplicate invoice" from an incident into a non-event.

Return errors as data, not exceptions. {"ok": false, "error": "no_such_customer"} gives the agent something to reason about. A stack trace gives it something to hallucinate around.

Keep results small. An agent that receives 8,000 tokens of JSON from a search tool will spend its context on noise and its attention on the wrong rows. Filter and summarise at the tool boundary.

Put a human at the risky edges

The question is never "should a human be involved?" but "involved where?" Score actions by reversibility and blast radius, then gate accordingly:

• Reversible, small: just do it. Reading, searching, drafting.
• Reversible, large: do it, but notify. Bulk updates with an undo path.
• Irreversible: ask first. Payments, external messages, deletions.

The gate needs to be a real pause: checkpoint the state, notify the approver with enough context to decide in ten seconds, and resume cleanly on approval, possibly hours later. An approval that forces the agent to restart from scratch will be turned off within a week.

Trace everything, from the first commit

You cannot debug an agent from logs that record only inputs and outputs. You need the whole tree: the plan, each tool call with its arguments and result, each model call with its prompt version and token counts, and the cost and latency of every step.

Add this before you need it. Retrofitting tracing onto an agent that is already misbehaving in production is a miserable week, and you will be doing it while someone asks you hourly whether it's fixed yet.

Once traces exist, two things become possible that were not before. You can attribute cost to individual steps, and the answer is almost always that one badly-scoped retrieval call is eating 60% of the budget. And you can build an evaluation set out of real runs, which is the only kind worth having.

Test the agent, not the prompt

Prompt tweaking without measurement is superstition. Take thirty to a hundred real tasks, write down what a correct outcome looks like for each, and score the agent against them on every change.

Grade what you can deterministically: did it call the right tool, did the record end up in the right state, did it stay inside budget. Use a model as a judge only for the genuinely subjective parts, and check the judge against human labels before you trust it.

The payoff is that "should we upgrade the model?" becomes a ten-minute question with a number attached, instead of an argument.

What good looks like

An agent we would leave running unattended has: bounded plans, typed and idempotent tools, permissions scoped to exactly what it needs, approval gates on anything irreversible, checkpointed state that survives a restart, a full trace of every run, and a regression suite that gates deploys.

That list is unglamorous, and it is the entire difference between a demo and a system. The model was never the hard part.