Building AI Agents: A Practitioner's Handbook, what you learn after the tutorials run out.

Choosing a Framework

Every framework for building agents makes the same core tradeoff: convenience against control. Higher abstractions let you build faster but constrain what you can do; lower abstractions give you full control at the cost of writing more code. The right choice depends on how well-specified your task is, how much you need to inspect and tune the agent's behaviour, and how tightly the framework's opinions match your own.

The framework landscape

LangGraph expresses agents as typed state graphs: nodes process state, edges route between them, and checkpoints persist state to durable storage. The graph structure forces you to think carefully about state transitions and makes the agent's control flow inspectable and debuggable. It's the right choice when you need fine-grained control over execution flow, have complex conditional branching, or need human-in-the-loop interrupts at specific points. The learning curve is real — you're writing a graph, not a function.

AutoGen models agents as conversational actors that message each other. The GroupChat abstraction manages turn-taking and termination conditions. It shines for tasks that are naturally dialogue-shaped: debate, critique, iterative refinement. It's harder to use for tasks that require structured sequential pipelines or tight state management, because the conversational model can make control flow implicit rather than explicit.

CrewAI provides opinionated higher-level abstractions — Agent, Task, Crew — with built-in role assignment, goal specification, and process templates (sequential, hierarchical). It reduces boilerplate dramatically and is the fastest framework to get a multi-agent prototype running. The trade-off is that when something goes wrong, the abstractions can obscure what's actually happening. Use it for prototypes and well-understood workflows; be ready to reach for something lower-level when you need to debug deeply.

The Claude Agent SDK and the raw Anthropic API with tool use represent the lowest abstraction level: you call the model directly, handle tool dispatch yourself, manage conversation history yourself, and build exactly what you need. This is the right choice when you have a narrow, specific task where a framework's opinions would fight you, when you need maximum transparency into every model call, or when you're building infrastructure that other systems will sit on top of.

Pydantic AI, smolagents, and the OpenAI Agents SDK are newer entrants that compress the ReAct-style loop into smaller surface areas. Pydantic AI leans on Pydantic models for structured input/output validation and works well when you want tool signatures enforced by types. Hugging Face's smolagents emphasises code agents — the model writes Python that is executed in a sandbox, rather than emitting JSON tool calls — which tends to produce shorter trajectories on tasks where multiple tool calls can be composed into a single script. The OpenAI Agents SDK wraps the OpenAI Responses API with built-in handoffs, guardrails, and tracing, and is the lowest-friction choice when you are already on the OpenAI stack.

Rolling your own — a thin wrapper over direct API calls — is underrated for simple agents. A single ReAct loop over a fixed tool set can be implemented in under 100 lines of Python. If your task doesn't require state persistence, parallel execution, or multi-agent coordination, the simplest implementation is often the most reliable. Below is roughly what that loop looks like; reading it once is worth more than reading any framework tutorial, because everything else you'll encounter is an elaboration of this core structure.

Everything a framework adds sits on top of that loop: state persistence, parallel dispatch, sub-agent spawning, interrupt handling, tracing. Decide which of those you actually need before you commit to a framework — many production agents never need more than the loop above plus a loop detector and a token ceiling.

Designing Robust Tool Schemas

Tool schemas are the interface between the model and the outside world. A poorly designed schema produces hallucinated arguments, incorrect parameter values, and calls to the right tool with semantics the tool's implementer never intended. The model reads your schema at every step where it considers using the tool — schema quality is not an afterthought.

The anatomy of a good tool schema

Schema design principles

Name tools for their action, not their implementation. search_knowledge_base tells the model what the tool does. kb_search_v2 tells it nothing useful. Tool names appear in the model's reasoning and influence when it decides to call the tool — descriptive names improve selection accuracy.

Make descriptions do double duty. Every description should answer two questions: when should the model call this tool, and when should it not? The "when not" is as important as the "when." Models that lack negative guidance will call tools in situations where a different tool (or no tool) would have been better.

Prefer enums over open strings wherever possible. If a parameter has a fixed valid set of values, enumerate them. The model will then pick from the list rather than generating a value that might not match what your implementation expects. An open string parameter like "format": "json" is reliably called with "JSON", "Json", "application/json", and other variants that your parser may not handle.

Make required vs. optional deliberate. Every field in the required array will be generated on every call. If a parameter is optional in your implementation but you put it in required, the model will hallucinate values for it when they don't apply. Only mark a field required if the tool cannot meaningfully execute without it.

Design for idempotency where you can. If the model calls a tool twice with the same arguments — which happens more than you'd expect — the second call should either return the same result or fail gracefully rather than creating duplicate side effects. This is especially important for write operations: creating a record, sending a message, incrementing a counter. Design tools to be safe against double-calls.

Returning results the model can use

Tool design doesn't end at the input schema. The result returned to the model is equally important. Results should be structured, compact, and unambiguous. A search tool that returns raw HTML or an enormous JSON blob forces the model to parse irrelevant content and wastes context tokens. Return the minimal structured representation needed: title, snippet, URL, relevance score. Include explicit signals for edge cases: "results": [] is better than an empty response that the model might interpret as a tool call failure.

For tools that can legitimately return a lot of data — a file read, a database query, a paged search — cap the payload at the tool boundary and signal truncation explicitly. A result like {"rows": [...first 50...], "truncated": true, "total_matches": 1,847, "next_cursor": "..."} tells the model exactly what happened and how to request more. Tools that silently drop data create a class of bug where the agent confidently reasons over an incomplete view of reality.

Parallel tool calls

Modern frontier models can emit multiple tool calls in a single assistant turn. When the tools are independent — three separate searches, checks across different services, lookups in different tables — dispatching them in parallel cuts end-to-end latency dramatically. A well-designed agent loop treats the model's response as a list of calls and runs them concurrently:

Two caveats. First, the model cannot always tell which calls are truly independent; tools that share state (e.g., two calls that both modify the same record) can race if the model requests them in parallel. Mark such tools as serial-only in their description and defensively serialise them in the dispatcher. Second, parallel dispatch amplifies errors: a batch of five calls where one fails can confuse the model more than a single failed call, because the rest succeed and the failure is buried in the response. Return a consistent shape for both success and failure so the model can reason about partial outcomes.

MCP and portable tool definitions

The Model Context Protocol (MCP) is an open standard for exposing tools to language models across vendors. A tool defined as an MCP server can be consumed by Claude, ChatGPT, Cursor, Claude Code, or any other MCP-aware client without rewriting the schema for each. For any tool you expect to reuse across more than one agent or product, implementing it once as an MCP server and letting each client mount it is the path of least resistance. The protocol handles discovery, schema negotiation, and transport; your server owns only the tool logic.

MCP does not replace the schema design principles above — a poorly described MCP tool is just as unreliable as a poorly described inline tool — but it changes where tools live and how they're maintained. For in-house tools used by a single agent, inline definitions remain simpler. For tools that need to be shared across an organisation or with external users, MCP is now the default.

Tool budgets and circuit breakers

Tools cost money, take time, and can fail. A production agent needs per-tool guardrails beyond what the schema can express. Three worth implementing from day one:

• Per-run call limits. Cap the number of times any single tool can be invoked within one agent run (e.g., search_web limited to 10 calls per run). This catches loops where the agent repeatedly searches without progress.
• Per-tool circuit breakers. If a tool has failed more than N times in the last M minutes, short-circuit further calls and return a structured error to the model. This prevents the agent from hammering a failing dependency and gives the model a clear signal to try something else.
• Cost tracking. For tools that cost money per call (paid APIs, LLM-backed sub-tools), track cumulative cost per run and surface it in the trace. A tool that looks cheap in isolation can be expensive when an agent calls it 40 times.

System Prompt Design for Agents

Tool schemas tell the model how to act. The system prompt tells the model who it is, what it is trying to accomplish, and which choices are out of bounds. Tool design and prompt design are two halves of the same interface; a great tool set attached to a vague prompt produces confident nonsense, and a great prompt attached to inscrutable tools produces an agent that cannot actually do anything.

Agent prompts differ from chat prompts in three ways that matter for design. They run for many turns without user correction, so ambiguity in the first turn compounds. They steer autonomous action rather than conversation, so vague guidance becomes visible in the form of inappropriate tool calls. And they are read alongside tool results and conversation history, so information that would be obvious in a chat ("don't send any emails today") has to be stated explicitly enough to survive dozens of intervening turns.

The anatomy of an agent system prompt

A well-structured agent system prompt has five parts that each do distinct work. In production agents they are nearly always explicitly labelled — either with section headings in plain text or with XML tags — because the model reasons better about structured content than about prose where roles blur together.

The ordering matters. Role comes first because it sets the frame for everything that follows; a model that has internalised "I am a triage agent with narrow scope" makes different choices than one that thinks of itself as a general assistant. Capabilities come next so the model knows the edge of its action space. Process is the main procedural guidance. Constraints come late and negatively framed — after the model has the positive picture, negative guidance lands more clearly. Output format comes last, because it governs the exit path.

Why XML tags

Frontier models — Claude in particular — are trained to attend to XML structure in prompts. Wrapping sections in <role>, <constraints>, and so on improves the model's ability to follow each section's directives independently, and makes the prompt itself easier to edit programmatically. If you find yourself concatenating strings to build prompts, switch to a template that renders sections into XML-tagged blocks; the cost is low and the reliability gain is real. The same content rendered as unstructured prose is measurably worse in long-running agent trajectories.

Negative specification and the anti-goal list

The single most underused construction in agent prompts is an explicit list of things the agent should not do, framed as actions rather than attributes. "Don't be careless" is useless. "Never call refund_order without first verifying the customer's identity via lookup_account" is actionable. Specific negative directives function as guardrails in a way general exhortations do not; they give the model something concrete to check itself against before each action.

Start the anti-goal list empty. Every time the agent does something you did not intend, ask: what negative directive in the prompt would have prevented this? Add it. After a few weeks, the anti-goal list becomes a compressed record of your failure modes, and the agent's behaviour tightens along with it.

Few-shot examples for agents

Adding one or two complete example trajectories — a full sequence of user request, tool calls, tool results, and final output — dramatically anchors the model's behaviour. A single strong example outperforms a paragraph of abstract guidance, because the model can pattern-match against the example's structure directly. The trade-off is token cost: trajectories are long, and they live in the system prompt on every run. Reserve few-shot examples for your hardest cases and prune aggressively.

Iterating on prompts without overfitting

The tempting workflow when the agent does something wrong is to add another sentence to the prompt pointing at the specific error. Done a few times, this is fine. Done habitually, it produces a prompt that is a patchwork of special cases, none of which interact cleanly, and each of which was added in response to a single observed failure. The agent gets worse on unobserved failures even as it gets better on the specific ones you patched.

The discipline is the same as for software regression testing: every prompt change should be validated against an eval set that covers previous success cases, not just the new failure. If a prompt change makes the agent better on the new case but worse on the regression suite, the change is net-negative even when it fixes the visible problem. Good prompt iteration is aggressive on the regression suite, conservative on the prompt itself.

Memory and Context Engineering

Every production agent has a memory system, even if it wasn't designed as one. The conversation history is short-term memory. The system prompt is procedural memory. Any vector store, database, or structured state object the agent reads from is long-term memory. Designing these deliberately — deciding what gets stored, for how long, and how it is retrieved — is the difference between an agent that improves over time and one that forgets everything at every session boundary.

Four memory timescales

Production agents need memory at four distinct timescales, each with different write and read patterns. Blurring them together is a common design mistake — storing everything in a single vector index, for example, produces an agent that can retrieve nothing well because the index is dominated by irrelevant chatter.

Short-term: context compression

The agent's context window is finite. Over a long trajectory, tool results and intermediate reasoning accumulate faster than the model can productively attend to them. Effective compression happens at two points:

At ingestion. Tool results should be shaped for the model at the moment they are generated, not retrospectively. A raw HTML page becomes a summary. A 500-row query result becomes the first 20 rows plus aggregates. A file listing becomes the structure, not the contents. Compression at ingestion is cheap because you have only one result to process; compression at the window boundary is expensive because you have a trajectory to fold down.

At the window boundary. When the context usage crosses a threshold, older tool results are summarised into a "progress summary" block while recent turns remain verbatim. Pseudo-code for a rolling-summary strategy:

The specific instruction given to summarise_trajectory matters more than the summary length. A summary that preserves decisions made and facts established is useful; a generic "what has happened" summary is not. Anchor compression to the kind of information the agent needs for its next step, not to a word count.

Session state: the underrated workhorse

Most agent builders reach for vector memory before they have tried a simple structured state object, and it is almost always the wrong order. A session-scoped dictionary maintained by the orchestration layer — not by the model — is the most reliable way to give an agent persistent context within a run. It holds the things the model has verified ("customer is on the Pro plan"), the artefacts it has produced ("draft reply stored at draft_id=4817"), and the decisions it has made ("classified as billing, routed to team B"). It is injected into the context on every turn as a small, structured block.

Two properties make the state object work. First, it is typed: the orchestration layer validates writes, so the agent cannot store malformed data that will confuse it later. Second, it is the source of truth for facts the agent acts on. When the agent wants to know what plan the customer is on, it reads the state object, not the full conversation — which may have contradicted itself across turns. This is exactly the mitigation for the stale state failure mode from the previous section, promoted to a first-class architectural element.

Long-term: vector memory, done well

Cross-session memory is where most agents reach for a vector database. The common failure is to embed everything the agent ever saw and hope retrieval will sort it out. It will not. Effective vector memory design starts from the question: what exact objects does the agent retrieve by similarity? If the answer is "prior resolved tickets, indexed by the problem description," your embeddings should be problem descriptions, not full ticket logs. If the answer is "knowledge-base articles, indexed by the situation they apply to," you embed situation descriptions authored for that purpose, not article titles.

The practical shape of a retrieval-augmented agent memory:

• Write path. At the end of each session, a lightweight extraction step identifies which facts, decisions, or artefacts are worth storing. Not every session produces a memory entry — most do not. Selective writes keep the index small and the retrieval precise.
• Read path. At session start (or when the agent explicitly calls a recall tool), a query is built from the current situation and used to retrieve the top-k most similar entries. Retrieved entries are formatted into a <relevant_past> block and added to the context.
• Pruning. Memory entries have TTLs or are demoted based on staleness. An agent that retrieves a customer preference from two years ago without knowing it's stale is worse than one with no memory at all.

Procedural memory: prompts and skills

The other long-lived memory is the agent's procedural knowledge — how to do things. This lives partly in the system prompt, but in mature agent systems, it increasingly lives in skill files: reusable instruction blocks the agent can load on demand. A skill for "handle a refund request" contains the procedure, the required checks, the output format, and the escalation rules. It sits in a skills directory, is version-controlled, and is loaded by the agent when the current task matches the skill's description.

This pattern, pioneered by tools like Claude Code and now standard in the Claude Agent SDK, solves a real problem: as agents accumulate more procedures, the single system prompt balloons until it either doesn't fit in context or drowns out the specific situation at hand. Skills keep the resident prompt small and expand on demand.

Handling Failure Modes

Six failure modes account for the vast majority of production agent incidents. Each has detectable signals, known causes, and proven mitigations. Building defences against them before they appear is far cheaper than debugging them after they've affected users — and the mitigations are mostly generic, so an investment in one production agent transfers to the next.

The first three (loops, context overflow, stale state) are failures of the agent's own reasoning over a long trajectory. The next three (tool hallucination, prompt injection, cost runaway) are failures at the boundary between the agent and the outside world. The distinction matters because the mitigations are different: trajectory failures are prevented by the orchestration layer, boundary failures are prevented by the tool layer and the deployment environment.

Testing with Evals

The standard software testing instinct — write a unit test, run it, fix the failure — transfers to agent development only partially. Agent behaviour is probabilistic and context-dependent in ways that deterministic unit tests can't capture. Effective agent testing requires a layered eval suite that covers different granularities of behaviour.

Eval-driven development

The productive loop is: write a failing eval that captures the behaviour you want, implement the change that makes it pass, and lock the eval as a regression test. This keeps your eval suite expanding over time without manual curation effort. Every bug that reaches production should be converted to an eval case before the fix is shipped; otherwise the same class of failure will reappear.

A minimal eval harness

An eval harness is a runner that takes a set of cases, runs the agent against each one, scores the outputs, and produces a report you can compare across versions. The scoring layer is the interesting part; the runner is mostly mechanical. Below is a shape that has held up across many production agent projects — it is small enough to understand in one reading, and it grows naturally as the eval suite grows:

The key design decision here is separating case from scorer: a case is a task, and each case can carry multiple scorers that each check one property of the output (did it classify correctly? did it stay within scope? did it cite correctly?). A case either passes all of its scorers or it fails. This makes failures diagnosable ("the agent passed classification but failed scope compliance") and makes it cheap to add new properties to check over time without re-running anything.

The harness should also record tokens, cost, step count, and prompt version on every run. Without these, you cannot tell whether a regression was caused by a prompt change, a model change, or a tool change — and in a fast-moving agent project, all three change often.

Evaluating non-deterministic outputs

Agent outputs are rarely identical across runs. Three techniques handle this. Execution-based evaluation runs generated code or queries and checks the results rather than the outputs themselves — whether the SQL returns the right rows matters more than whether the SQL is identical to a reference. LLM-as-judge uses a separate model to score outputs against a rubric; this scales but requires calibrating the judge's own biases. Semantic equivalence embeds outputs and reference answers and checks cosine similarity — useful for text outputs where multiple phrasings are equally correct.

For safety-critical behaviours — refusals, scope compliance, PII handling — always use deterministic programmatic checks rather than semantic equivalence. You need to know with certainty whether the agent sent an email to the wrong recipient, not approximately whether it seemed to.

Calibrating an LLM-as-judge

LLM-as-judge is powerful at scale but unreliable if taken at face value. Judge models have biases — they prefer longer answers, answers that restate the question, answers with confident tone — and the bias shifts when you change the judge model, the rubric wording, or the order of candidates. A judge that has not been calibrated is a noisy signal that can move your eval metric without the agent's behaviour actually changing.

The minimum calibration procedure: collect 100+ human-labelled cases covering the full score range (not just pass/fail), run the candidate judge over the same cases, and compute agreement with the human labels. Acceptable agreement depends on the use case — for gate-keeping critical behaviours, you want >90% exact-match on the key axis; for trend tracking, lower rank correlations may suffice. Log the judge's prompt, model, and temperature alongside every score, because any of those changing invalidates the calibration.

Two practical anti-patterns to avoid. First, do not use the same model family as both agent and judge for the same rubric — they share biases, and the judge will over-credit the agent's reasoning style. Second, do not change the judge's prompt without re-running the calibration set; a "clarifying" edit to the rubric can silently shift the score distribution.

Production Deployment

A working agent and a shippable one are separated by a set of engineering decisions that don't show up in benchmarks. Most production agent failures are not model capability failures — they are infrastructure failures: runaway cost, rate limit exhaustion, tool API instability, missing rollback capability, and absent observability that makes debugging impossible.

Prompt and model version pinning

An agent's behaviour is determined by the combined configuration of its system prompt, model, tool set, and orchestration code. All four can change; changes to any one of them can shift behaviour; and without explicit versioning you cannot tell which change caused a regression. Two concrete disciplines make this tractable in production:

Pin the model. Never call a model by a rolling alias in production. claude-sonnet-4-6 and gpt-5-turbo point at specific model versions today and potentially different ones next month. Pin to the dated model string emitted in the API response and upgrade deliberately as part of a release, after running the regression suite on the new version. A silent model upgrade has broken more production agents than any other single cause.

Version the prompt alongside the code. The system prompt lives in version control, is reviewed in pull requests, and emits a version identifier into every agent trace. The cheapest implementation is a single string constant in the repo with a semver-style version variable; the more mature implementation is a prompts directory with one file per agent role, each with a semver header. Either way, the prompt version and the model version together are the minimum provenance information needed to debug a production incident weeks later.

Cost engineering

Cost is a first-class production concern, and treating it as such is the difference between an agent that breaks even at scale and one that silently burns through the budget. Three techniques do most of the work:

Prompt caching. Most providers (Anthropic, OpenAI, Google) support caching of long, stable prompt prefixes — system prompts, tool schemas, few-shot examples — so they are charged at a steep discount on subsequent calls within a window. Structuring your prompt with the stable content first, then the volatile content (user message, recent history), and enabling cache breakpoints reduces input cost by 60–90% on repeated calls. For an agent that makes 15 model calls per run, caching is frequently the largest single cost optimisation available.

Model routing. Not every step needs the frontier model. Classification, simple extraction, and routing decisions often work well on smaller, cheaper models; only the hard reasoning steps need the expensive one. A lightweight first-stage model that handles 70% of tasks end-to-end, escalating the rest to a stronger model, produces order-of-magnitude savings with marginal quality impact — provided the routing decision itself is accurate and tested.

Cascade evaluation. When the agent produces a structured answer, a cheap verifier can check it before the result is committed. If the verifier passes, ship it. If not, either retry with a stronger model or escalate to a human. Cascades work especially well for extraction and classification tasks where the verifier can check a concrete property ("does the extracted amount appear in the source document?") cheaply.

The tuning point in any cascade is the confidence threshold that triggers escalation. Measure it on your calibration set rather than guessing. A threshold that's too low wastes money escalating easy cases; too high wastes quality on hard ones. Revisit the threshold after every model change and every prompt change, because the underlying confidence distribution can shift.

Monitoring in the Wild

Agents degrade in ways that traditional software doesn't. A web server that goes down produces an error rate spike; an agent that gradually loses quality in a specific task category produces a subtle decrease in user satisfaction that no single metric captures. Effective monitoring triangulates across multiple signal types simultaneously.

The signal stack

Distributed tracing for agents

Each agent run should emit a structured trace that records the complete trajectory: every model call with its input token count and output tokens, every tool call with its arguments and result, every state transition, and the final outcome. This trace is the foundation of all post-hoc debugging. Without it, when a user reports that the agent did something wrong, you have no way to reconstruct what happened.

Use a consistent run ID that propagates through every component of the trace. When a sub-agent is invoked, its trace should reference the parent run ID. This makes it possible to reconstruct the full causal chain of a multi-agent interaction, not just the leaf-level tool calls.

Anomaly detection and drift

The most important long-run monitoring signal is distribution shift in incoming requests. Track an embedding of user requests over time and alert when the distribution drifts significantly from your evaluation set. Distribution shift is the earliest detectable signal that your agent is encountering task types for which it wasn't designed and hasn't been evaluated — often weeks before the quality degradation becomes visible in user feedback.

Observability tools worth knowing

The agent observability ecosystem matured rapidly through 2025. You do not have to build traces, dashboards, and replay UIs from scratch. A few tools worth knowing:

• Langfuse and LangSmith are the two most widely used purpose-built LLM observability platforms, with trace capture, prompt versioning, dataset management, and replay. LangSmith is tightly integrated with LangChain/LangGraph; Langfuse is open source and framework-agnostic.
• Arize Phoenix is an open-source alternative focused on traces and evals, with a strong local-first workflow for teams that want to keep data in-house.
• OpenTelemetry (OTel) GenAI conventions define a vendor-neutral schema for LLM traces. Emitting OTel spans from your agent code means any OTel-compatible backend (Honeycomb, Datadog, Grafana Tempo) can ingest them — useful if you already have an observability stack and don't want a second one.

The core requirement regardless of tool is that every agent run is traceable end-to-end with a stable run ID, that prompt and model versions are recorded, and that traces can be replayed against new prompt or model versions without re-running the original user interaction. Lock in the trace format before you pick a backend; switching backends is cheap, rewriting your trace schema is not.

Feedback loops from users back into evals

The highest-value monitoring signal in most agent products is user feedback. A thumbs-down on a specific agent response is more informative than a week of aggregate dashboards — it points at a specific failure with a specific trace. The production pattern that separates mature agent teams from the rest is a closed loop from user feedback back into the eval set:

1. User flags a bad response (thumbs-down, "this wasn't helpful," explicit report).
2. The corresponding trace is automatically tagged and surfaced in a review queue.
3. A human (on-call engineer, or a rotating reviewer) labels the failure and, for recurring patterns, promotes the trace to a permanent eval case.
4. The regression suite grows by roughly one case per material user complaint, and the same failure cannot ship twice.

This loop is mechanical but often neglected. Teams spend heroic effort on post-hoc incident analysis and then fail to convert the learnings into regression tests, which means the same failure mode resurfaces six weeks later. The discipline is simple: no incident closes without an eval case.

Incident response

Every production agent system will eventually have an incident. The response process should be designed before the first incident, not during it. Three decisions need to be made in advance: what is the kill switch (how do you take the agent offline in under 60 seconds?), who is authorised to use it, and what is the standard for using it (what failure rate triggers an immediate rollback vs. a monitoring escalation)?

After an incident, the post-mortem should produce two things: an eval case that reproduces the failure, and a system change that prevents it. Incidents without corresponding eval additions are a debt that accumulates until the same failure mode reappears.

Architecture Patterns and a Worked Example

After surveying what frameworks to use, how to build tools and prompts, how to handle failures, how to test, and how to deploy, a natural question is: what does a well-architected production agent actually look like? Several patterns recur across domains and teams that have shipped agents at scale, and they compose cleanly — most production agents are a small combination of two or three of them. The section closes with a worked end-to-end example that stitches them together into one complete build.

The thin orchestrator pattern

The most robust production agents have a thin, deterministic orchestration layer and push all intelligence to the model. The orchestrator handles: routing tasks to the right agent configuration, managing conversation history within token budgets, dispatching tool calls returned by the model, implementing loop detection and cost ceilings, and emitting structured traces. It contains no business logic and makes no judgments about task content. Business logic lives in the tool implementations and the system prompt, where it can be tested and updated independently of the orchestration infrastructure.

The specialist-generalist split

For broad domains, a single agent with a large tool set often underperforms a network of specialist agents routed by a lightweight generalist. The generalist agent receives the task, identifies the domain (coding, research, data analysis, customer support), and delegates to a specialist agent configured with a domain-appropriate tool set and system prompt. Each specialist can be developed, evaluated, and updated independently. The routing decision is often simple enough to make deterministically, without another model call: a regex or a classifier over the task description is frequently sufficient and avoids adding latency and cost to every task.

The human-in-the-loop checkpoint pattern

For agents that take irreversible or high-stakes actions, the most reliable pattern is explicit checkpointing: the agent runs until it is about to take a consequential action, then pauses and surfaces the proposed action to a human for approval. The pause is implemented at the orchestration layer, not in the model — it is a property of the tool dispatch logic, not a prompt instruction the model can override. The human sees a structured proposal (what the agent intends to do, why, and what the consequences are) and can approve, reject, or modify before execution proceeds. This pattern trades speed for safety, and the right balance point is task-dependent.

The plan-and-execute pattern

For tasks that benefit from up-front structuring — multi-step research, end-to-end code changes, complex workflows — a plan-and-execute architecture runs a first pass that produces an explicit plan, then a second pass that executes the plan step by step. The plan is a first-class artefact: it is stored in the state object, shown to the user (or a reviewer) if appropriate, and referred back to on every subsequent step. When execution diverges from the plan, the agent records the divergence rather than silently abandoning the plan.

Why this works: the plan anchors the trajectory. A plain ReAct loop can forget its initial intent after ten steps of tool use; a plan-and-execute agent cannot, because the plan remains in the context on every turn. The trade-off is upfront latency — you pay for a planning call before any real work starts — and the cost of plans that turn out to be wrong and need to be replanned mid-run. For tasks longer than about ten steps, the trade is almost always worth it; for short tasks, the overhead isn't.

The reflection / self-critique pattern

Before committing a final answer, the agent produces its proposed answer, then runs a critique step that reads the answer against a rubric and either approves it or returns revision notes. If the critique rejects, the agent revises and the loop repeats up to a small fixed limit. On well-tuned deployments this pattern reliably improves output quality by 10–25% on measurable rubrics, at the cost of one or two extra model calls per run.

Two design notes. First, the critic and the solver should be different — either different models, different prompts, or both — so that biases cancel rather than compound. A critic that is the same model with the same prompt will approve its own mistakes at a high rate. Second, the critic should check a specific list of properties ("does the answer cite its sources? is the classification one of the allowed values? does any sentence state a fact not supported by the retrieved context?"), not a vague overall quality rubric. Specific critics produce actionable revision notes; vague ones produce flattery.

The router pattern

For broad product surfaces, the simplest effective architecture is a router: an extremely small model or classifier reads the incoming request and dispatches to one of a small number of specialised agents. Each specialist has its own system prompt, tool set, and eval suite; each can be developed and updated independently. The router itself is often not a model call at all — a regex, a classifier, or a rules engine over the request — which keeps routing latency and cost to a minimum.

Router architectures scale organisationally in a way that monolithic agents do not. Separate teams can own separate specialists. The blast radius of a bad change is confined to one specialist. And the router is the natural place to add governance: rate limiting, access control, user-specific policies. The downside is that the handoff boundaries between specialists become their own eval category — you need tests that verify the router picks the right specialist, not just that each specialist handles its own inputs correctly.

A worked end-to-end example: customer-support triage agent

To make the patterns concrete, this section walks through building a production-grade customer-support triage agent end to end, touching every topic in the chapter. The agent's job is to read an incoming support ticket, classify it, gather any missing context, and either resolve the ticket or route it to the right human team with a summary. This example draws directly on the tool schemas and system prompt sketched earlier.

Framework and architecture

The agent has a bounded workflow (classify → gather context → resolve or route), clear checkpoints, and needs state persistence across turns — the right shape for LangGraph's typed state graph. The state is a Pydantic model with fields for ticket_id, customer_id, classification, confidence, context_fetched, and recommendation. Each graph node reads and writes specific fields; the edges route based on classification confidence. This is the thin orchestrator pattern with a dash of plan-and-execute (the classification step acts as the plan), and a router at the edge when the agent decides which specialist team to route to.

Tools

Six tools, each designed for the single job the agent needs. Names carry specific verbs and nouns; descriptions include negative guidance:

• search_knowledge_base — query the internal KB; enum-restricted section; max 10 results.
• read_ticket_history — fetch the last N messages for a ticket; read-only.
• fetch_account — fetch customer account details; read-only.
• classify_ticket — structured output tool (returns classification + confidence).
• post_internal_note — post to the internal-only notes field; not customer-visible.
• route_to_team — route the ticket to one of the specialist teams; terminal action.

Note what's missing: there is no reply_to_customer, no refund_order, and no change_billing. Those actions are intentionally architectural dead ends — the agent's scope is triage, and the tool set enforces that boundary more reliably than any prompt ever could. This is the prompt-injection mitigation promoted to architecture: even a fully compromised agent can only post internal notes and route.

System prompt

The prompt follows the template from Section 03 — role, capabilities, process, constraints, output format — with XML tags and a specific anti-goal list distilled from the first two weeks of shadow-mode running. Critical directives ("never post anything customer-visible") are repeated in both the system prompt and the description of the relevant tool, because redundancy against prompt injection is cheap.

Memory

Turn-scoped memory is the LangGraph state object; session-scoped memory is the same object persisted via checkpointing; cross-session memory is a small vector store of resolved tickets, indexed by the problem description, with a TTL of 90 days. The agent has a recall_similar_tickets tool that surfaces 3–5 past tickets it can learn from. Procedural memory — triage policies, escalation rules — lives in versioned skill files loaded by the framework at startup.

Failure mitigations

A loop detector caps any single tool at 4 consecutive identical calls. A context manager compresses old tool results past a 60k-token threshold while preserving the state object and the last 8 turns verbatim. Tool schemas are validated before dispatch; unknown tool names return a did_you_mean suggestion. Untrusted content (ticket body, retrieved KB articles) is wrapped in <untrusted_content> delimiters. Every run has a hard token ceiling of 120k and a cost ceiling of $0.50.

Evals

The eval suite has four tiers: unit evals for each tool (especially schema conformance), integration evals on the classification node (does the model call classify_ticket with an allowed value?), end-to-end evals on 120 historical tickets with human-labelled outcomes, and a locked regression suite of 40 cases covering every failure mode discovered since launch. Scoring combines deterministic checks (classification in allowed set, no customer-visible action taken) with an LLM-as-judge that scores rationale quality, calibrated against 200 human-labelled trajectories.

Deployment

Releases go through dev → staging → shadow → canary (5%) → full, with the regression suite required to pass before each promotion. The model is pinned to a specific dated version; the system prompt is versioned in semver and emitted in every trace. Prompt caching is enabled on the stable prefix (system prompt + tool schemas + skill files), which brings input cost per turn down by roughly 80% on cached paths.

Monitoring

Traces are captured via Langfuse with the standard schema. The dashboard tracks classification accuracy against labelled samples, scope violation rate (hard zero), loop detection rate, cost per ticket, and user feedback rate from the support team. Every thumbs-down on an agent-authored internal note is automatically tagged for triage review; confirmed failures are promoted to eval cases within 48 hours.

What this shows

Nothing in this build is exotic. Every piece is a named pattern from earlier in the chapter. What makes it a production agent rather than a demo is the accumulation of small, disciplined choices: pinned model, versioned prompt, validated tool dispatch, explicit state object, bounded tool set, layered evals, shadow-mode rollout, and a feedback loop that turns complaints into regression tests. None of them are hard individually. The discipline is refusing to ship until all of them are in place.

Building agents that work reliably is a discipline, not a trick. The conceptual architecture — perception, reasoning, action — is the easy part. The hard part is every decision downstream of that: which framework fits your control flow, how to write tool schemas the model will use correctly, what to do when the trajectory goes wrong, how to build confidence through layered evals, how to deploy without surprises, and how to know — without guessing — whether the agent is working in the real world. That discipline, accumulated through building, breaking, and fixing real systems, is what this handbook has tried to make concrete.