Production Engineering

11.2 Idempotency keys

Retries and side effects are enemies. If the model called send_refund and the response was lost, retrying charges the customer twice. The fix is an idempotency key derived from the logical operation, not from the attempt.

For your own tools you get nothing for free: generate a key per logical operation and write it to a ledger table before executing, so a replay finds the record and returns the prior result.

11.3 Timeouts and deadlines

A tutorial has no timeout, so a hung TCP connection blocks a worker forever. Set connect/read timeouts, then propagate a deadline across the whole agent loop - a per-call timeout of 30s across an eight-step loop is a four-minute request.

11.4 Circuit breakers

When a provider is genuinely down, retrying is just a slower way to fail. Martin Fowler's circuit breaker trips after N consecutive failures, fails fast for a cooldown, then probes.

11.5 Rate limits and token budgets

Two separate concerns, routinely conflated:

• Provider limits - read the headroom from response headers and slow down before you get 429s.
• Your budget - a per-tenant token cap that exists for cost control and abuse prevention, and has nothing to do with the provider.

11.6 Cost accounting per request

If you cannot attribute spend to a trace and a tenant, you cannot find the query pattern that is costing you $4,000 a month.

11.7 Structured logging and OpenTelemetry GenAI spans

print() does not survive contact with production. There is a standard for this now - the OpenTelemetry GenAI semantic conventions - so use its attribute names and every tool downstream understands your traces for free.

Better still, auto-instrument (OpenAIInstrumentor().instrument() from OpenLLMetry / OpenInference) so every call - including ones buried inside a framework - emits a span.

Platforms that consume this: Langfuse (OSS, self-hostable, OTel-native), LangSmith, Arize Phoenix. All give nested traces, per-generation cost, and eval hooks.

Recall Chapter 9.8's lesson while designing your log schema: format failures and correctness failures are different signals. Count them separately or you will not be able to tell a broken JSON schema from a wrong answer.

11.8 Evals and CI regression gates

This is the single biggest cultural difference between a script and a product: prompts are code, so they get tests. Every PR that touches a prompt, a model id, or a tool schema runs a golden dataset and fails the build on regression.

# eval.yaml - `promptfoo eval --config eval.yaml` in CI
prompts: ["prompts/support_agent.txt"]
providers: ["openai:gpt-4o"]
tests:
  - vars: { question: "Refund policy?" }
    assert:
      - type: llm-rubric
        value: "Mentions the 30-day window and does not invent a policy"
      - type: latency
        threshold: 3000

Note the tolerance band. Model output is stochastic; a gate demanding exact equality will flake until someone disables it. Chapter 8.10 drew this line already - deterministic parts get assert ==, fuzzy parts get scored thresholds.

11.9 Prompt versioning

Inline string literals cannot be rolled back. Version prompts, pin a version per deployment, and record the version on the span so you can answer "what changed?" three weeks later.

11.10 Guardrails and the lethal trifecta

Simon Willison's framing is the clearest security model in the field:

private data + untrusted content + external communication = exfiltration.

Any agent with all three legs can be talked into leaking, and no system-prompt instruction reliably prevents it - the injected text and your instructions are the same kind of tokens.

The mitigation is architectural: break one leg.

   private data  ──┐
   untrusted text ─┼──▶ EXFILTRATION RISK
   can send out  ──┘

   break a leg:  no live fetch while reading private docs
                 allowlist outbound destinations
                 human approval before anything leaves the building

This connects straight back to Chapter 10.10's MCP tool poisoning: a poisoned tool description is the "untrusted content" leg arriving through a channel you thought was configuration.

Also worth wiring: OWASP LLM Top 10 (LLM01 is prompt injection), NeMo Guardrails, Guardrails AI for PII/toxicity validators on output.

Sandboxing tool execution

Look again at Chapter 8's calculator:

That empty-builtins trick is a teaching shortcut, not a sandbox - it is bypassable, and it is the kind of line that ends up in a postmortem. In production, model-generated code runs in an ephemeral microVM.

For shell tools: subprocess.run(..., timeout=T, cwd=jail_dir), resource.setrlimit, read-only rootfs, no network namespace, explicit binary allowlist - and never shell=True with a model-authored string.

Secrets

Keys never appear in prompts, tool schemas, or logs. Fetch from a vault at runtime and scrub on the way out.

11.11 Graceful degradation and fallback routing

Chapter 7's abstraction pays off here in cash. One provider down should mean a slower answer, not an outage.

11.12 Streaming, cancellation and concurrency

Streaming is a latency perception fix. Cancellation is a billing fix: when the user closes the tab, the upstream request must actually be closed, or you keep paying for tokens nobody reads.

Bound concurrency rather than asyncio.gather-ing a thousand calls at once:

11.13 Caching, and the layout rule it implies

The design implication is the same rule as Chapter 9.7: static content first (system prompt, tool definitions, few-shot examples, retrieved documents), variable user turn last. A stable prefix is money. Separately, application-level dedup - hash (model, messages, params) → Redis with a TTL, busted on prompt-version change - catches identical requests (GPTCache does this).

11.14 Typed config and locked dependencies

Failing at import time is a feature. A missing key should kill the deploy, not the request.

uv lock            # resolved + hashed versions
uv sync --frozen   # CI installs exactly the lock, fails if drifted

Chapter 2 explained why manifests exist; this is the production consequence. SDK minor bumps have changed default timeouts, retry counts, and response shapes. Pin, or inherit someone else's Tuesday.

11.15 12-Factor Agents

The humanlayer/12-factor-agents list is the closest thing to a canon. The five factors that matter most for everything above:

Factor 12 is the quiet one that unlocks the rest. Idempotency, cancellation and resume are all straightforward if the agent is a pure function over state, and all nearly impossible if it is a long-lived object holding a socket.

11.16 The whole delta, on one page

Every row is a decision you can make today, in the Chapter 8 project, for less effort than it takes to add another tool. That is the actual argument for production discipline: it is not a later phase, it is a set of small habits that compound - and the projects that skip them do not fail loudly, they just quietly cost ten times more and nobody can say why.

11.17 Live Streaming Reasoning (<think>), In-Process Telemetry & Multi-Agent Teams

Three patterns complete the production architecture built in the reference implementation:

1. Streaming <think> State Machine Parser

When working with reasoning models (DeepSeek R1, Qwen 2.5, Claude thinking), thought processes arrive inside <think>...</think> tags. Because network chunks split tags across boundaries (e.g. chunk 1 ends with <th, chunk 2 starts with ink>), a regex cannot run on live streams.

A robust parser uses a state machine with a partial-tag buffer:

2. In-Process Telemetry & Performance Tracking

Measure generation speed ($\text{tokens/sec} = \frac{\text{completion_tokens}}{\text{latency_ms} / 1000}$) and roundtrip latency for every call, appending structured JSONL events to .logs/raw_events.jsonl:

3. Multi-Agent Team Orchestration

Decompose complex goals using three core orchestration topologies:

1. Sequential Chains: Output of Agent A becomes input context for Agent B.
2. Parallel Fan-out + Lead Synthesizer: Concurrent execution via ThreadPoolExecutor where specialist agents (Researcher, Coder, Reviewer) execute subtasks in parallel, and a Synthesizer LLM consolidates their findings into the final response.
3. Dynamic Workflow Architect: An architect agent designs custom agent personas, system prompts, and subtasks on the fly based on the user's high-level goal.

Check yourself

• Why is retrying a 400 strictly worse than failing immediately? Which status codes are retryable, and why exactly those?
• Full jitter versus plain exponential backoff: describe the failure that jitter prevents.
• Your agent retried a tool call that had already sent an email. Design the idempotency key and say precisely when it must be written relative to the send.
• A per-call timeout of 30s in a loop capped at 8 steps: what is the real worst-case request duration, and how does a deadline object fix it?
• Name the three legs of the lethal trifecta and pick the cheapest one to break for an agent that reads internal documents and can browse the web.
• Why is eval(expr, {"__builtins__": {}}, {}) not a sandbox? What would you replace it with?
• Given Anthropic's cache economics (write 1.25×, read 0.1×), how many reads must a cached prefix get before it pays for itself?
• Explain 12-factor Factor 12 ("stateless reducer") and how it makes cancellation and idempotency easier to implement.