Memory & Context Engineering

9.1 The problem: your messages list is a leak

Recall the loop from Chapter 6:

Nothing ever leaves that list. Run twenty turns with a tool that returns a 4 KB JSON blob each time and you have quietly built an application that:

1. sends ~80 KB of stale tool output on every single request, and pays for it every time;
2. gets worse at answering, not better, as the list grows;
3. eventually hard-fails with a context-length error, mid-conversation, in front of a user.

Point 2 is the counterintuitive one, and it has a name.

Context rot

Anthropic's Effective context engineering for AI agents (Sep 2025) states the mechanism plainly. Transformer attention is O(n²) - every token attends to every other token, so each token you add spends from a finite "attention budget." Worse, models are trained on data dominated by shorter sequences, so they have proportionally fewer parameters specialised in long-range, whole-context dependencies. Length-extension tricks like position-encoding interpolation buy you a bigger number on the spec sheet at the cost of positional precision.

The result is context rot: needle-in-a-haystack recall accuracy degrades as token count grows. It is a gradient, not a cliff, and it is present in every model regardless of the advertised window size.

Common mistake: treating a 200K-token context window as 200K tokens of usable working memory. It is a hard limit, not a target. A 30K-token prompt that contains only relevant material routinely outperforms a 150K-token prompt containing the same material plus history.

So the discipline is not "fit everything in." It is:

Context engineering: curating and maintaining the smallest set of high-signal tokens that lets the model take the next correct action.

Anthropic draws the distinction from prompt engineering directly: prompt engineering is about the wording of one message; context engineering is about everything that lands in the window - system prompt, tool definitions, retrieved documents, MCP output, message history - and how that set is re-curated on every single turn of the loop.

   PROMPT ENGINEERING            CONTEXT ENGINEERING
   ┌───────────────┐             ┌──────────────────────────────────┐
   │ how do I word │             │ of everything I *could* put in   │
   │ this request? │             │ the window this turn, what       │
   └───────────────┘             │ earns its tokens?                │
        one-shot                 └──────────────────────────────────┘
                                      every turn, forever

9.2 A memory taxonomy that actually maps to code

Cognitive-science vocabulary gets thrown around loosely in agent frameworks. Here is the mapping that the systems below (MemGPT/Letta, Mem0, Zep) actually implement:

Your Chapter 6 Memory class conflates all four into one list. Every technique in this chapter is some version of pulling one tier out of the working set and giving it its own storage plus a retrieval path back in.

Note the shape: render() is called per turn and returns a derived payload. The stored state and the sent payload are no longer the same object. That single separation is the whole chapter in one line of design.

9.3 MemGPT / Letta: memory as an operating system

Source: Packer et al., MemGPT: Towards LLMs as Operating Systems, arXiv:2310.08560.

MemGPT's insight is an analogy carried all the way through to the implementation: an LLM with a fixed context window is a CPU with fixed physical RAM, so borrow virtual memory. The paper describes "hierarchical memory systems … that provide the appearance of large memory resources through data movement between fast and slow memory."

Two tiers:

• Main context - what is in the prompt. Fast, small, expensive.
• External context - recall storage (past conversation) and archival storage (documents). Slow, unbounded, cheap.

The part that makes it an agent technique rather than a caching technique: the model itself issues the paging calls, as ordinary function calls.

Compare this to Chapter 6's calculator. Structurally identical - a name, a schema, a Python function. The only difference is that the function mutates the agent's own memory instead of doing arithmetic. Memory management became a tool call. Once you see that, the design space opens up: eviction, consolidation, and recall are all just tools you can write.

MemGPT was evaluated on two domains that break naive agents: document analysis beyond the context length, and multi-session chat where the agent must "remember, reflect, and evolve." It was later productised as Letta (docs.letta.com), whose framing is stateful agents - agent state lives server-side and persists across calls, instead of the client resending the whole history each turn.

Common mistake: giving the model a remember(fact) tool and no search/evict counterpart. Memory that only grows is just your original list with extra steps.

Check yourself

• Why must archival_memory_search return a bounded number of results, and what happens to the attention budget if it returns fifty?
• Which tier from 9.2 does core_memory_append write into?

9.4 Mem0: extract, then reconcile

Sources: Chhikara et al., arXiv:2504.19413; github.com/mem0ai/mem0.

Mem0 runs a two-phase pipeline incrementally over each message pair (m_{t-1}, m_t):

Phase 1 - extraction. Build a prompt from a rolling asynchronous conversation summary S, the recent messages, and the new pair. An LLM returns a candidate fact set Ω = {ω₁ … ωₙ} - short, standalone statements.

Phase 2 - update. For each candidate, vector-search the top-s most similar existing memories, then hand candidate + neighbours back to the LLM through a tool-call interface whose only options are:

ADD     - genuinely new information
UPDATE  - refines/corrects an existing memory
DELETE  - contradicts an existing memory
NOOP    - already known, discard

The conflict resolution is done by the model, not by hand-written rules. This matters: "user moved from Berlin to Lisbon" must DELETE-or-UPDATE the Berlin fact, and no similarity threshold you tune by hand will reliably catch that.

A Mem0g variant stores memories as a directed labelled graph (entities as nodes, relations as edges) to support multi-hop questions.

Numbers, from the paper's LOCOMO benchmark - against a full-context baseline (stuff the entire history in):

Read that table twice. Selective memory beat "send everything" on quality, not just on cost. That is context rot showing up as a measurable business number.

The 2026 platform revision changed the algorithm again - single-pass ADD-only extraction (no overwrite; memories accumulate) with multi-signal retrieval fusing semantic + BM25 keyword + entity linking:

Benchmark        Old   New   Retrieved tokens   p50 latency
LoCoMo           71.4  92.5  7.0K               0.88s
LongMemEval      67.8  94.4  6.8K               1.09s
BEAM (1M)         --   64.1  6.7K               1.00s

The architectural lesson is bigger than the numbers: consolidation moved from write-time to read-time. Rather than deciding at ingestion which fact wins, store everything and let a strong ranker decide at query time. That is a recurring pattern - you will see it again in Chapter 10's retrieval fusion.

pip install mem0ai

9.5 Zep / Graphiti: memory that knows when

Sources: Rasmussen et al., arXiv:2501.13956; github.com/getzep/graphiti.

Vector stores answer "what is similar." They cannot answer "what is true now," and they certainly cannot answer "what was true in March." Graphiti - the temporal knowledge-graph engine under Zep - is built for exactly that gap.

The critical design decision: a new fact does not delete the old edge - it closes the old edge's validity window. Bi-temporal versioning. History is preserved, so "who was the account owner before the reassignment?" is answerable, and every derived fact traces back to the episode it came from.

   fact: (Alice) --works_at--> (Acme)     valid [2023-01 .. 2025-06)   ← invalidated, kept
   fact: (Alice) --works_at--> (Globex)   valid [2025-06 .. now)       ← current

Retrieval is hybrid - semantic similarity plus BM25 keyword plus graph traversal - and updates are incremental, with no full-graph recomputation on write.

Benchmarks from the Zep paper: 94.8% vs MemGPT's 93.4% on Deep Memory Retrieval, and on the harder, temporally-loaded LongMemEval up to +18.5% accuracy with −90% latency against baseline implementations, with the largest gains on cross-session synthesis.

Common mistake: modelling user facts as mutable rows you overwrite. The moment a user asks "wait, what did I tell you last month?", an overwriting store cannot answer and cannot even tell you that it can't.

9.6 Context engineering techniques you should ship

Straight from Anthropic's engineering guidance, with the implementation notes that matter.

System prompts: find the right altitude

Too low = brittle hardcoded if/else logic in prose, which shatters on the first unanticipated case. Too high = vague guidance that gives the model nothing to act on. Aim between, and structure it:

<background_information>
You operate on the internal billing database. Invoices are immutable once issued.
</background_information>

## Instructions

1. Resolve the customer before touching any invoice.
2. Never issue a credit note above $500 without the approval tool.

## Tool guidance

- `find_customer` - use first, always. Accepts email or account ID.
- `issue_credit` - mutating. Requires an approved `approval_id`.

## Output description

Reply with a one-paragraph summary, then a markdown table of affected invoices.

Minimal does not mean short. It means: everything needed, nothing more.

Tools: the interface is the product

Anthropic's test is worth memorising: if a human engineer cannot tell which tool to use, neither can the agent. Tools should be self-contained, non-overlapping, and token-efficient in their return values - a tool that dumps a 6 KB JSON payload for a yes/no question is spending your attention budget on your behalf.

Just-in-time retrieval instead of pre-loading

The emerging pattern: don't embed and pre-load everything. Keep lightweight references - file paths, saved queries, IDs, links - and let the agent load the payload at runtime with a tool. Humans work this way: nobody memorises the filesystem, they ls and grep.

Claude Code is the worked example. It writes targeted queries and uses head, tail, grep, and glob to inspect enormous repositories without ever loading whole files into the window. And it's explicitly hybrid: CLAUDE.md is pre-loaded up front because it's small, static and always relevant, while file exploration is just-in-time because an index would go stale.

The trade-off is named honestly in the source: runtime exploration is slower than precomputed retrieval, and needs good tools or the agent burns context chasing dead ends.

Compaction

Take a conversation nearing the window limit, summarise it, and reinitialise a new window with the summary.

Claude Code's implementation preserves architectural decisions, unresolved bugs, and implementation details, discards redundant tool output and chatter, and carries forward the five most recently accessed files.

Tuning method, in order: maximise recall first (capture everything relevant, even if bloated), then iterate for precision (strip the superfluous). Doing it in the other order silently drops information you never learn you lost.

The lightest-touch form is tool-result clearing: once a tool call is deep in history, clear its raw output - the agent already extracted what it needed and will never re-read the blob. This is shipped as a platform feature, and it is also about six lines of code you can write today.

Structured note-taking

Let the agent write persistent notes - NOTES.md, a to-do list - and pull them back in later. "Persistent memory with minimal overhead."

The cited example is memorable: Claude playing Pokémon maintains tallies across thousands of game steps ("for the last 1,234 steps I've been training on Route 1, Pikachu has gained 8 levels toward the target of 10") with no explicit memory scaffolding, and after a context reset it re-reads its own notes and resumes multi-hour sequences.

Sub-agent context isolation

An orchestrator holds the high-level plan. Specialised sub-agents each get a clean context window, explore extensively - tens of thousands of tokens - and return a condensed 1,000–2,000 token summary.

   LEAD AGENT  (plan, ~8K tokens, stays clean)
      ├─▶ sub-agent: search codebase   [40K tokens burned] ──▶ 1.5K summary ──┐
      ├─▶ sub-agent: read the RFCs     [30K tokens burned] ──▶ 1.2K summary ──┤
      └─▶ sub-agent: check the tests   [25K tokens burned] ──▶ 0.9K summary ──┘
                                                                    ▼
                                              lead synthesises from 3.6K, not 95K

Anthropic's multi-agent research system post reports substantial improvement over single-agent setups on complex research tasks using exactly this separation of concerns.

Which one, when

And the standing recommendation from the source, which is easy to forget while building elaborate machinery: do the simplest thing that works. Models keep getting better at operating with less scaffolding.

9.7 KV-cache-friendly prompt design

Inference servers (vLLM, SGLang) and hosted prompt caching all reuse the attention KV cache for a shared prefix. The rule that follows is mechanical:

1. Keep the system prompt and tool definitions byte-identical across turns.
2. Grow the conversation append-only.
3. Put volatile, just-in-time-retrieved content after the stable prefix, never interleaved before it.

One stray timestamp early in the prompt forces a full re-prefill on every request, wiping out the latency and cost gains of every technique in 9.6. This is one of the highest effort-to-payoff-ratio fixes in the entire book.

9.8 The Hermes lineage: reasoning and tool-calling as a format

Sources: Hermes 4 Technical Report, arXiv:2508.18255; github.com/NousResearch/Hermes-Function-Calling.

Chapter 7 abstracted providers behind one chat() contract. Hermes is worth studying because it shows what that contract looks like when the model is fully open and the entire protocol lives in a chat template you can read.

Hermes 4 is a family of hybrid reasoning models - one checkpoint, open weights - that "combine structured, multi-turn reasoning with broad instruction-following ability." Post-training used roughly 5M samples / 19B tokens: 3.5M reasoning + 1.6M non-reasoning, where reasoning samples average 5× more tokens and thinking traces reach 16K tokens.

The synthetic data came from DataForge, a graph-based generator: each datapoint is a random walk through a DAG of PDDL-style struct → struct nodes whose preconditions and postconditions define the edges. Hold that thought - Chapter 10 uses the same DAG idea for retrieval pipelines.

One model, two modes - switched by the template

There is no separate "reasoning model." The mode is a flag in the chat template:

{%- set thinking_prompt = 'You are a deep thinking AI, you may use extremely long chains of
 thought ... You should enclose your thoughts and internal monologue inside <think> </think>
 tags, and then provide your solution or response to the problem.' %}
{%- set standard_prompt = 'You are Hermes, created by Nous Research.' %}
{%- if not thinking is defined %}{% set thinking = false %}{% endif %}
{%- if thinking %}{%- set system_prompt = thinking_prompt %}
{%- else %}{%- set system_prompt = standard_prompt %}{%- endif %}

Tool calls as XML-wrapped JSON

Hermes does not rely on the serving stack having native function-calling support. The protocol is in the prompt:

# Tools
You are a function calling AI model. You may call one or more functions to assist with the
user query. You are provided with function signatures within <tools></tools> XML tags:
<tools>
{tool_json_schema_1}
{tool_json_schema_2}
</tools>
For each function call, return a json object with function name and arguments within
<tool_call></tool_call> XML tags:
<tool_call>
{"name": "<function-name>", "arguments": <args-json-object>}
</tool_call>

The model emits:

<tool_call>
{"name": "get_stock_fundamentals", "arguments": {"symbol": "TSLA"}}
</tool_call>

and results are fed back as user-role messages wrapped in <tool_response>:

{%- elif message.role == "tool" %}
    {{- '\n<tool_response>\n' }}{{- message.content }}{{- '\n</tool_response>' }}

The reference functioncall.py pipeline is exactly the Chapter 6 loop with a different parser: build the system prompt from a YAML template embedding the tool schemas and a Pydantic FunctionCall schema → generate → regex-extract <tool_call> blocks → validate the JSON against Pydantic → execute → recurse up to --max_depth (default 5) until no more calls appear.

Format compliance is trained separately from correctness

This is the most transferable idea in the Hermes report. They built distinct RL environments whose rewards score shape, not truth:

• Reasoning Format env - binary 1.0/0.0 reward for correct placement of <think>/</think> at the start of generation, sampled across 150+ target output formats (\boxed{} for maths, etc.). Content correctness is not scored.
• Schema Adherence env - generate/edit raw JSON against dynamically compiled Pydantic models with deliberately injected validation errors (type mismatches, constraint violations, extra fields); reward = does the output instantiate the target model.
• Tool Use env - the environment intercepts the <tool_call>, validates field hierarchy and values against a ground-truth dataset, binary exact-match reward.

The lesson for your own systems: "did you use the schema correctly" and "was the answer right" are orthogonal signals, and you should measure them separately. Almost every "the agent is broken" bug report is really one of those two failing, and if your logs collapse them into one "failure" counter you cannot tell which.

What makes Hermes distinct

• Single checkpoint; reasoning toggled by system prompt alone, no separate model, no extra head - unlike shipping a reasoning model and an instruct model side by side.
• Tool calling is XML + JSON in the template, so it works on any server that honours a chat template (vLLM, SGLang, LM Studio from Chapter 5) without native function-calling support.
• Heavy, explicit investment in format-compliance training as a first-class objective.

Common mistake: hard-coding OpenAI's message.tool_calls field access into your loop, then discovering your local Hermes model returns tool calls as text inside content. This is precisely what Chapter 7's adapter layer is for - the adapter's job is to normalise <tool_call> XML and native tool_calls arrays into one internal shape.

Drop that class into providers.py next to the three from Chapter 8 and nothing in loop.py changes. That is the payoff of the abstraction, tested against a genuinely different protocol.

9.9 Practical In-Process Compaction: Caveman Compression & Window Trimming

Not every context-saving technique requires an external vector database or an extra LLM summarization call. When running interactive agents, two fast, zero-cost in-memory algorithms provide immediate token reductions:

1. Caveman / Telegraphic Compression

Natural language contains high token redundancy in articles (a, an, the), prepositions (in, on, at), and polite conversational fillers (please, could you kindly).

The Caveman compression algorithm strips these stopwords while strictly preserving code blocks (...), inline backticks (...), numbers, and arithmetic symbols:

When applied to older turns in Memory.compact(mode="caveman"), this typically reduces past message token consumption by 30% to 45% without degrading the LLM's understanding of prior instructions or tool outputs.

2. Rolling Window Trimming

The simplest baseline for unbounded conversations is a fixed-turn FIFO buffer: keep the system message anchored at index 0, and retain only the last $N$ turns:

Check yourself

• Explain context rot in terms of the attention budget, without using the phrase "the model gets confused." Why does a 30K high-signal prompt often beat a 150K one containing the same facts?
• Mem0 beat a full-context baseline on quality by 26%, not just on cost. What does that result tell you about the "just use a bigger context window" strategy?
• Graphiti closes an edge's validity window instead of deleting the edge. Name a user-facing question that this makes answerable and an overwriting store cannot answer at all.
• Why is core_memory_append implemented as a tool rather than as logic in the loop? What does the model gain by controlling it?
• Rewrite this cache-hostile system prompt to be cache-friendly: f"You are a support bot. Today is {today}. The user is on the {plan} plan."
• Your agent has a lead planner and three sub-agents. If each sub-agent burns 40K tokens and returns a 1.5K summary, how many tokens does the lead ever see? What would the single-agent version have cost?
• Hermes trains format compliance and answer correctness as separate reward signals. Design the two counters you would add to your own logging to keep that distinction visible in production.