Retrieval and memory for AI agents

A language model forgets everything between calls. Whatever an agent knows about the task in front of it, the user it works for or the mistakes it made yesterday has to be put back into its context window, every time. Retrieval and memory are the machinery that decides what goes back in. This post maps that machinery as of October 2026: why a larger context window does not remove the need for it, how retrieval works, how agents write and organise memory, what the benchmarks show and where they mislead, and how to choose an architecture for an agent of your own. Every number links to its source. The date above says when the page was last checked.

Why the context window is not enough

An agent’s context window is its only working memory. Many models accept 128K to 1M tokens (Modarressi et al., 2025), and it is tempting to put everything in. The evidence says models use long inputs unevenly.

Liu and colleagues found that performance is often highest when the relevant information sits at the beginning or end of the input, and drops significantly when it sits in the middle, even for models built for long contexts (Liu et al., 2023). RULER tested 17 long-context models: almost all scored nearly perfectly on the plain needle-in-a-haystack test, yet only half of the models that claim 32K tokens or more held satisfactory performance at 32K (Hsieh et al., 2024). NoLiMa removed the literal word overlap between question and answer. At 32K tokens, 11 of 13 models fell below half of their short-context score, and GPT-4o dropped from 99.3% to 69.7% (Modarressi et al., 2025).

Chroma tested 18 models and found that performance degrades as input length grows, even on simple tasks. A single distractor, meaning related text that does not answer the question, reduced performance, and four distractors reduced it further (Chroma, 2025). Anthropic’s engineering guidance draws the practical conclusion: models have an attention budget, and context is “a finite resource with diminishing marginal returns” (Anthropic, 2025).

Long context still changed the trade-off. In a 2024 comparison across three models, long context beat retrieval-augmented generation (RAG) on average when resources allowed, while RAG cost far less. The two gave identical predictions for over 60% of queries, and a router that sent each query to one or the other cut cost by 65% for Gemini-1.5-Pro and 39% for GPT-4o at comparable quality (Li et al., 2024). Anthropic’s advice from the same year: if a knowledge base is smaller than 200,000 tokens, about 500 pages, put all of it in the prompt and skip retrieval (Anthropic, 2024).

So the question is rarely retrieval or long context. It is what to load, when, and in what form.

A map of agent memory

Agent memory borrows its vocabulary from cognitive science. The CoALA framework describes four kinds (Sumers et al., 2023):

• Working memory: the agent’s current situation, including recent inputs, active goals and intermediate results. CoALA defines it as a data structure that persists across model calls; in most agents it maps onto the context window.
• Episodic memory: records of the agent’s past behaviour and experiences.
• Semantic memory: knowledge about the world and about the agent itself.
• Procedural memory: how to act. CoALA splits it into knowledge implicit in the model weights and explicit knowledge written in the agent’s code.

CoALA warns that letting an agent write to its own procedural memory is riskier than writing episodic or semantic memory, because it can introduce bugs or let the agent subvert its designers’ intentions (Sumers et al., 2023).

A 47-author survey from December 2025 argues that the long-term/short-term split no longer captures the field. It sorts memory by form (token-level, parametric, latent), by function (factual, experiential, working) and by dynamics: how memory is formed, evolved and retrieved (Hu et al., 2025). It also separates agent memory from RAG and from context engineering, which are often used interchangeably.

This post uses a simpler split that maps onto engineering decisions. The read path is retrieval: finding what is relevant in a store. The write path is memory management: deciding what enters the store, in what form, and when it changes or leaves.

The read path: retrieval

Lexical, dense and late interaction

Retrieval has two broad families. Lexical retrieval such as BM25 matches the words of the query against the words of the document. Dense retrieval encodes both into vectors and compares them. On open-domain question answering, Dense Passage Retrieval beat a strong BM25 system by 9 to 19 points in top-20 passage accuracy (Karpukhin et al., 2020). Approximate nearest-neighbour indexes such as HNSW make vector search fast at scale, with logarithmic complexity scaling (Malkov and Yashunin, 2016).

Dense retrieval does not win everywhere. BEIR evaluated ten retrieval systems on 18 datasets without in-domain training. BM25 was a strong baseline, re-ranking and late-interaction models achieved the best average results at high computational cost, and dense models often underperformed (Thakur et al., 2021).

A 2025 paper by researchers at Google DeepMind and Johns Hopkins University shows a structural limit. The number of distinct top-k result sets that a single-vector embedding can return is bounded by its dimension. On LIMIT, a dataset with deliberately simple queries, current embedding models struggled to reach 20% recall@100, while BM25 reached about 94%. When the corpus terms in a smaller version of LIMIT were replaced with synonyms, BM25 dropped by more than 89% and fell behind most of the neural models (Weller et al., 2025). Each family fails where the other is strong.

Between the two sit learned sparse models such as SPLADE, which learn term weights and expansions but keep the inverted index of lexical search (Formal et al., 2021), and late-interaction models such as ColBERT, which encode queries and documents as bags of token embeddings and compare them cheaply at query time. ColBERT was competitive with BERT-based rankers while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query (Khattab and Zaharia, 2020).

Fusion and re-ranking

Hybrid systems run several retrievers and merge the results. Reciprocal Rank Fusion scores each document by the sum of 1 divided by (k + rank) across the input rankings. The original paper found k = 60 near-optimal, and RRF almost always improved on the best individual ranking (Cormack et al., 2009). A second stage then re-ranks the top candidates with a cross-encoder that reads query and passage together. A 2019 BERT re-ranker topped the MS MARCO passage leaderboard with a 27% relative improvement in MRR@10 (Nogueira and Cho, 2019).

Chunks lose their context

Documents are split into chunks before indexing, and a chunk often makes no sense alone. Anthropic’s Contextual Retrieval prepends a short explanation of where each chunk sits in its document, usually 50 to 100 tokens written by a model, before embedding and before building the BM25 index. On Anthropic’s tests, measured as top-20 retrieval failure rate, contextual embeddings cut failures by 35%, adding contextual BM25 cut them by 49%, and adding a re-ranker cut them by 67%. With prompt caching, generating the context cost $1.02 per million document tokens (Anthropic, 2024). Late chunking reaches a similar goal inside the embedding model: it embeds the whole long text first and splits it into chunks only at the pooling step (Günther et al., 2024).

The query can also be rewritten. HyDE asks a model to write a hypothetical answer document and retrieves real documents that resemble it (Gao et al., 2022). Rewrite-Retrieve-Read trains a small model to rewrite queries for the retriever, using the reader model’s feedback as the reward (Ma et al., 2023).

Structure over chunks

Flat chunk retrieval fails on questions about a whole corpus, such as “What are the main themes in the dataset?” GraphRAG builds an entity knowledge graph with a model, pre-generates summaries for communities of related entities, and answers global questions from those summaries. On corpora around one million tokens it gave more comprehensive and diverse answers than conventional RAG (Edge et al., 2024). RAPTOR recursively clusters and summarises chunks into a tree and retrieves across levels of abstraction. Paired with GPT-4, it improved the best result on the QuALITY benchmark by 20 points of absolute accuracy (Sarthi et al., 2024).

HippoRAG combines a knowledge graph with Personalized PageRank. On multi-hop questions, its single-step retrieval matched or beat iterative retrieval while being 10 to 30 times cheaper and 6 to 13 times faster (Gutiérrez et al., 2024). Its successor reports a cost of structure: graph-augmented RAG methods improved sense-making and associative questions but fell considerably below standard RAG on basic factual recall. HippoRAG 2 was built to close that gap (Gutiérrez et al., 2025).

From pipelines to agentic retrieval

The original RAG model paired a generator with a dense index of Wikipedia: parametric memory in the weights, non-parametric memory in the index (Lewis et al., 2020). Classic RAG retrieves a fixed number of passages, whether or not retrieval is needed (Asai et al., 2023).

Agents retrieve as an action. ReAct interleaved reasoning with calls to a simple Wikipedia API and reduced the hallucination and error propagation of chain-of-thought on HotpotQA and FEVER (Yao et al., 2022). Self-RAG trained a model to retrieve on demand and to critique the passages and its own output (Asai et al., 2023). Corrective RAG added a light evaluator that scores retrieved documents and, depending on its confidence, triggers other retrieval actions, including web search (Yan et al., 2024). Search-R1 trained models with reinforcement learning to issue their own search queries during reasoning and improved on RAG baselines by 41% for Qwen2.5-7B and 20% for Qwen2.5-3B (Jin et al., 2025).

Agentic search does not make the retriever irrelevant. BrowseComp-Plus fixed the document corpus so that retrievers could be compared. Search-R1 with BM25 reached 3.86%, GPT-5 with BM25 reached 55.9%, and GPT-5 with the Qwen3-Embedding-8B retriever reached 70.1% with fewer search calls (Chen et al., 2025). A better retriever made the same agent both more accurate and cheaper.

Grep, files and just-in-time context

Coding agents use a different pattern. Anthropic describes Claude Code as a hybrid: CLAUDE.md files are loaded into context up front, while glob and grep let the agent find files “just in time”. Agents built this way keep lightweight identifiers, such as file paths, stored queries and web links, and load data into context at runtime (Anthropic, 2025).

Semantic search still helps inside that loop. Cursor reported that adding semantic search to its agent’s grep-based tools raised answer accuracy by 12.5% on average in offline evaluations, between 6.5% and 23.5% depending on the model. In an online A/B test, code retention rose by 2.6% on codebases with 1,000 or more files. The post concludes that the combination of grep and semantic search gives the best outcomes (Cursor, 2025).

Files also work as memory. Letta stored LoCoMo conversation histories in plain files, gave a gpt-4o-mini agent file tools including grep and semantic search, and reported 74.0% on LoCoMo, above the 68.5% Mem0 had reported for its best graph variant (Letta, 2025). On LongMemEval-V2, a method that stores agent trajectories as files and lets a coding agent gather evidence reached 72.5% average accuracy, against 48.5% for the strongest RAG baseline. The authors note that coding-agent methods have high latency costs (Wu et al., 2026).

My read: “grep or vectors” is a false choice. The pattern that holds up is to give the agent several search tools over a store it can also read directly, and to let it iterate. The open question is cost: an agent that searches for ten turns is slower and more expensive than one retrieval call.

The write path: memory

Managing the working context

Long-running agents fill their context with tool output. Anthropic names three techniques for this (Anthropic, 2025):

• Compaction: summarise a conversation near the context limit and continue from the summary. One of the lightest forms is clearing old tool results.
• Structured note-taking: the agent writes notes, such as a NOTES.md file or a to-do list, outside the context window and reads them back later.
• Sub-agents: specialised agents work in clean context windows and return a condensed summary, often 1,000 to 2,000 tokens.

On an internal agentic-search evaluation, Anthropic reported that context editing, which clears stale tool calls and results, improved performance by 29%, and that combining it with a file-based memory tool improved performance by 39%. In a 100-turn web search evaluation, context editing cut token consumption by 84% (Anthropic, 2025). These are vendor numbers on an internal evaluation set.

MemGPT framed the same problem as an operating system. A fixed main context holds a FIFO queue of messages, the first entry of which is a recursive summary of evicted messages. When the queue nears its limit, the system warns the model of “memory pressure” so it can save important information. External context lives in two databases, recall storage for the message history and archival storage for arbitrary text, and the model moves data between tiers through function calls (Packer et al., 2023).

Episodic memory and reflection

Generative Agents kept a memory stream: a natural-language record of everything each simulated character observed. Retrieval scored each memory by recency, importance and relevance. Recency decayed exponentially with a factor of 0.995 per game hour since the memory was last retrieved, and importance was a 1-to-10 rating the model assigned when the memory was written. Periodic reflection synthesised memories into higher-level inferences (Park et al., 2023). MemoryBank added forgetting modelled on the Ebbinghaus curve: memories fade or strengthen based on elapsed time and significance (Zhong et al., 2023).

Reflexion stored something else: the agent’s own written reflections on task feedback, kept in an episodic buffer for the next attempt. With this, it reached 91% pass@1 on HumanEval against 80% for GPT-4 (Shinn et al., 2023).

Fact stores and graphs

Assistants that serve one user over months need semantic memory about that user. Mem0 extracts, consolidates and retrieves salient facts from conversations, with an optional graph variant. Its paper reports a 26% relative improvement over OpenAI’s ChatGPT memory on an LLM-as-judge metric, 91% lower p95 latency and over 90% lower token cost than a full-context approach (Chhikara et al., 2025). Zep builds a temporally aware knowledge graph that keeps historical relationships, and reports up to 18.5% higher accuracy on LongMemEval with 90% lower response latency than baseline implementations (Rasmussen et al., 2025). The section on evaluation returns to how far these vendor results can be trusted.

Other designs let the model organise memory itself. A-MEM writes each memory as a note with keywords, tags and links to related notes, in the manner of a Zettelkasten, and lets new memories update the descriptions of old ones (Xu et al., 2025). Memory-R1 trains a memory manager with reinforcement learning to choose ADD, UPDATE, DELETE or NOOP for each new fact, and reports gains over strong baselines with only 152 training question-answer pairs (Yan et al., 2025).

Procedural and experiential memory

Some memory stores ways of acting instead of facts. Voyager, a Minecraft agent, kept a growing library of skills as executable code and retrieved them for new tasks (Wang et al., 2023). Agent Workflow Memory induces reusable workflows from past web tasks and feeds the relevant ones back to the agent. It improved relative success rates by 24.6% on Mind2Web and 51.1% on WebArena (Wang et al., 2024).

Dynamic Cheatsheet keeps a self-curated memory of strategies and code snippets across questions. GPT-4o’s success rate on Game of 24 went from 10% to 99% once it found and reused a Python solution (Suzgun et al., 2025). ReasoningBank distils strategies from both successful and failed attempts, judged by the agent itself, and outperformed memories built from raw trajectories or successful routines alone on web browsing and software engineering benchmarks (Ouyang et al., 2025).

Rewriting a memory repeatedly can destroy it. The authors of Agentic Context Engineering describe “context collapse”, where iterative rewriting erodes detail, and brevity bias, where summaries drop domain insight. Their remedy is a playbook updated incrementally rather than rewritten, which gained 10.6% on agent benchmarks (Zhang et al., 2025).

Memory work outside the request

Memory updates do not have to happen while the user waits. Sleep-time compute lets a model process a context before queries arrive. On stateful versions of GSM-Symbolic and AIME, it reduced the test-time compute needed for the same accuracy by about 5 times, and sharing the work across related queries cut the average cost per query by 2.5 times (Lin et al., 2025).

At the far end, memory moves into the model. Titans adds a neural long-term memory module that learns to memorise at test time, and scales to context windows larger than 2 million tokens (Behrouz et al., 2024). Memory outside the weights has one practical advantage: people can read, edit and delete it.

Memory as a product feature

Memory is now an API feature. Anthropic’s memory tool lets Claude create, read, update and delete files in a /memories directory that persists between sessions. It runs client-side: the model requests file operations and the developer’s application executes them against its own storage. When the tool is enabled, the API adds a system instruction to check the memory directory first and to assume the context may be reset at any moment (Anthropic, memory tool documentation). The documentation pairs memory with compaction: compaction keeps the active context small, and memory keeps what must survive the summary.

How memory is evaluated

The first generation: conversational recall

Most published memory numbers come from question answering over long chat histories.

LongMemEval found that commercial chat assistants and long-context models lost about 30% accuracy when information had to be remembered across sustained interactions (Wu et al., 2024). On BEAM, even models with 1M-token windows struggled as dialogues grew longer, with and without retrieval (Tavakoli et al., 2025).

Contested numbers

Mem0, Zep and Letta all report LoCoMo scores, and their numbers are disputed. Mem0’s paper compared against Zep among others (Chhikara et al., 2025). Zep replied that the comparison used a flawed implementation of its system: both speakers were assigned the user role, timestamps were appended to messages instead of using Zep’s timestamp field, and searches ran sequentially, which inflated latency. Zep reported a corrected score of 75.14% and argued that LoCoMo conversations are short enough for modern models to read in full, that it lacks knowledge-update questions, and that one question category had missing ground-truth answers (Zep, 2025). Letta’s file-based agent, described above, beat Mem0’s reported graph score, and Letta concluded that memory “is more about how agents manage context than the exact retrieval mechanism used” (Letta, 2025).

My read: every party in that dispute sells a memory product, and none of the numbers comes from an independent evaluation. Treat vendor LoCoMo scores as marketing until someone without a stake reproduces them.

The second generation: memory in use

The 2026 benchmarks move from “can you recall it?” to “does memory help you act?”. MemoryArena found that agents with near-saturated scores on long-context memory benchmarks such as LoCoMo performed poorly once memory had to guide later actions, with near-zero success in two environments (He et al., 2026). DolphinBench, from Mem0’s research team, argues that question-answer benchmarks give the game away, because the question itself signals that a fact must be retrieved. It verifies each task by running an agent with and without the relevant history, keeps only tasks that succeed with it and fail without it, and requires every submission to report cost and latency (Rathi et al., 2026). StreamMemBench tested eight memory systems and found that they often failed to use evidence they had stored, or to turn feedback into reliable behaviour on follow-up tasks (Liu et al., 2026).

Time matters too. A single-author preprint from July 2026 plants facts with validity intervals before generating any text, then tests memory systems at three and nine weeks of history. Rankings inverted with tenure: a budgeted memory that led at three weeks fell from 96% to 72% recall of evicted content at nine weeks, while a provenance-typed graph rose to 90%. A baseline that rendered the full history tied or beat the best memory system at the short horizon, but showed no judge-independent advantage at nine weeks, at about twice the read cost. Questions whose underlying fact had been written poorly were answered wrongly 24% of the time, against 2% when the fact was written cleanly (Spencer, 2026). It is one paper, version 1, and not yet replicated, but the design point stands: test memory at the history length you expect in production.

Open problems

What to write, and how to keep it true

Retrieval cannot recover what was never written well. In the tenure study, write-stage quality correlated strongly with downstream quality (Spencer, 2026). Summaries drift: ACE documents context collapse under iterative rewriting (Zhang et al., 2025), and the SSGM framework names semantic drift through repeated summarisation as a core risk of evolving memory (Lam et al., 2026).

Change over time

Facts change: users move jobs, decisions get reversed. LongMemEval tests knowledge updates explicitly (Wu et al., 2024), and EverMemBench found that temporal reasoning fails without explicit version semantics beyond timestamps (Hu et al., 2026). Zep’s temporal graph is one response (Rasmussen et al., 2025).

Retrieving what nobody asked for

As the DolphinBench authors put it, question-answer benchmarks hand the agent a question that points at the memory it needs (Rathi et al., 2026). Real work often does not. EverMemBench found memory awareness bottlenecked by retrieval, with similarity-based methods missing implicitly relevant information, and multi-hop accuracy of at most 27% across speakers and groups: 26.5% for the best model reading the full history and 18.9% for the best memory system. Given oracle evidence, two of the three models recovered to 88–98%, so the failure is in retrieval and attribution (Hu et al., 2026). NoLiMa shows the same weakness inside the context window when the question and the answer share no words (Modarressi et al., 2025).

Poisoned memory

A memory store is an input channel, and it can be attacked. PoisonedRAG reached a 90% attack success rate by injecting five malicious texts per target question into a knowledge base of millions (Zou et al., 2024). AgentPoison backdoored agent memory and knowledge bases with an average success rate above 80% at a poison rate below 0.1%, with under 1% impact on benign performance (Chen et al., 2024). MINJA needs no access to the store at all: an attacker plants malicious records through ordinary queries, and the agent later retrieves them for another user’s request (Dong et al., 2025). File-based memory adds classic risks. Anthropic’s documentation warns that a path such as /memories/../../secrets.env can escape the memory directory unless every path is validated (Anthropic, memory tool documentation).

Privacy and forgetting

Memory that persists can also leak. SSGM describes how sensitive context can be solidified into long-term storage (Lam et al., 2026). Anthropic’s documentation advises stripping sensitive data before writing, capping file sizes, and deleting memory files that have not been accessed for a long time (Anthropic, memory tool documentation). MemoryBank is an early example of forgetting by design (Zhong et al., 2023).

Evaluation

The memory-product scores discussed above are self-reported and disputed, and the newer benchmarks are months old. The 2025 survey describes a fragmented field whose works differ substantially in implementation and evaluation protocol (Hu et al., 2025). Until results are reproduced independently, a memory system has to be judged on your own tasks.

How to choose and apply

Start with the smallest thing that works

When you build it

1. Measure retrieval on its own. Collect real queries, label the passages that answer them, and track recall@k before you tune prompts. A better retriever made GPT-5 both more accurate and cheaper on BrowseComp-Plus (Chen et al., 2025).
2. Use more than one retriever. Lexical and dense methods fail on different queries (Weller et al., 2025), and fusion is cheap (Cormack et al., 2009).
3. Keep the context small on purpose. Load identifiers, not documents, and let the agent fetch what it needs (Anthropic, 2025).
4. Write memory deliberately. Store facts with a timestamp, a source and, where you can, a validity period. Prefer incremental edits over full rewrites (Zhang et al., 2025).
5. Keep memory readable and editable by people. Plain files or simple records make bad memories easy to find and remove.
6. Treat every memory write as untrusted input. Scope memory per user, validate paths, record provenance, and test injection through ordinary conversation in the style of MINJA (Dong et al., 2025).
7. Expire what is not used, and make deletion work end to end (Anthropic, memory tool documentation).
8. Evaluate memory by task outcome with and without history, at the history length you expect, and report cost and latency next to accuracy (Rathi et al., 2026; Spencer, 2026).

My read: the systems that hold up in 2026 look less like a memory product bolted onto a model and more like a disciplined harness around one: a few search tools, a small set of well-written files, aggressive pruning of the context, and a test suite that measures outcomes over time. A specialised memory store earns its place when that test suite says so.