AI Agent Memory: Why Agents Forget and How to Fix It

An AI agent forgets for one structural reason: the language model behind it is stateless. Every call starts from a blank slate, and anything outside the context window is gone. Agent memory is the architecture that fixes this — a small, always-in-context core plus an external store the agent writes to and retrieves from — and getting it right separates a demo that dazzles on Tuesday from a product that still knows your customer on Wednesday.

The model has no memory — the system has to

Large language models do not remember previous conversations. GPT-5, Claude and Gemini alike carry no state between API calls. What looks like memory inside a chat product is an engineering trick: the application replays earlier messages back into the prompt on every turn. That works right up until the conversation, the retrieved documents and the tool outputs stop fitting the window — or stop being read carefully.

And the window is smaller in practice than the spec sheet suggests. Chroma’s Context Rot research tested 18 frontier models and found accuracy falling non-uniformly as inputs grow — sometimes by 30 to 50 percent well before the advertised limit, even on simple retrieval. Counter-intuitively, coherent, well-structured input degraded attention more than shuffled text did. The practical reading: a two-million-token window is a marketing number, and useful, high-accuracy budgets land far lower.

So “just use a bigger context window” is not a memory strategy. Replaying every past message into every call costs tokens, adds latency, and — worse — buries the one fact that mattered under ten thousand that did not. Memory is the discipline of deciding what the model should see this turn, and where everything else lives in the meantime.

The four kinds of agent memory

Borrowing from cognitive science, the agent-memory field — LangChain’s LangMem framing is the clearest — splits an agent’s memory into four types:

• Working memory: the current context window — the messages, tool definitions and data in front of the model right now. Free, but wiped at the end of the session.
• Episodic memory: summaries of specific past events — “this customer complained about a late delivery in March.” The agent learns from what happened before.
• Semantic memory: durable facts and preferences — “this account is on the enterprise plan and prefers email over phone.” The stable knowledge that should always be available.
• Procedural memory: learned rules and behaviour — “always confirm the address before booking a service call.” The agent’s own operating instructions, refined over time.

Working memory is fleeting; the other three you build deliberately, store outside the model, and pull back in when relevant. They can be written on the hot path (the agent saves a memory mid-conversation) or by a background process that reads finished transcripts and extracts what is worth keeping — the quieter, cheaper option for most teams.

The dominant 2026 pattern: tiered memory

Almost every production system in 2026 converges on the same shape: a small always-in-context core, a larger external store behind retrieval, and an explicit policy for what to forget. The idea traces to MemGPT, the 2023 Berkeley paper that framed an LLM as an operating system — the context window is RAM, an external store is disk, and the model pages information in and out with tool calls. Its successor, Letta, turns that into a runtime where the agent edits its own core memory and archives the rest to a vector database.

Two newer moves make the pattern cheaper to run. Compaction summarises old turns into a compact note instead of carrying them verbatim. And context editing — Anthropic shipped a memory tool plus context editing on its developer platform — lets an agent clear stale tool results automatically as it nears the token ceiling, which Anthropic reports cuts token use sharply on long-running tasks. The underlying lesson is identical: decide what stays hot, and move everything else out of the window before it rots.

The tooling landscape, honestly

Four names cover most of what a Dutch SME will actually evaluate, and they are not interchangeable:

• Mem0 — a managed memory layer that is fast to bolt onto an existing agent, with broad framework support and a large open-source following. Best when you want personalization quickly.
• Zep — built on a temporal knowledge graph, so it tracks how facts change over time. The right choice when “what was true when” matters, at the cost of more infrastructure to run.
• Letta — not just a memory layer but a stateful agent runtime; best for long-running autonomous agents that manage their own memory like an operating system manages pages.
• LangMem — LangChain’s SDK, the natural fit if you already build agents on LangGraph and want memory that plugs into the graph directly.

A word of caution on the benchmark wars. Vendors publish eye-catching scores on LoCoMo and LongMemEval — the two standard conversational-memory tests — and they rarely agree, because scoring methods and question mixes differ between reports. Treat a leaderboard number as a starting hypothesis, not a purchase decision. The number that matters is accuracy on your questions, over your data, in your language.

Do you actually need external memory?

Not every agent needs a memory system, and the vendors selling one will not tell you that. If your agent answers one-shot questions, or each session is genuinely independent — a form-filler, a document summariser — you do not need persistent memory. You need good retrieval, which is a different thing. See RAG: RAG fetches knowledge from documents, memory remembers interactions with people. They often sit side by side on the same vector database, but they solve different problems, and conflating them is a common design error.

If your agent holds multi-turn conversations, serves returning customers, or runs long autonomous tasks, memory earns its keep — and the case is less about tokens than most people assume. Replaying a thirty-message history — call it 8,000 tokens — at Claude Sonnet 5’s introductory input price of roughly two euros per million tokens is under two cents a call; retrieving 500 tokens of relevant memory instead is a tenth of that. The saving is real but modest. The real prize is accuracy: a short, curated memory beats a long, noisy transcript, precisely because of context rot. You are not saving money so much as buying a sharper answer.

In the Netherlands, memory is personal data

Here is the part the framework blogs skip. The moment your agent remembers something about a named customer — a preference, a past complaint, a home address — that memory is personal data under the GDPR (AVG). An agent’s long-term store is a processing system, and the same rules apply to it as to any customer database. This is exactly the ground you have to get right when you train an agent on company data.

Practically, that means three commitments. You need a retention and forgetting policy — the right to erasure means a customer can ask you to delete what the agent remembers, so “forgetting” has to be a feature you can actually execute, not an accident. You need to know where the memory physically lives — an EU-hosted store keeps data residency simple, while a US managed service raises questions you will have to answer. And where the memory shapes decisions about people, the EU AI Act’s transparency and governance duties come into scope. Building memory without a forgetting policy is the most common mistake we see: it is easy to store and surprisingly hard to un-store once the store is a graph of cross-referenced facts.

What we’d tell a Dutch SME building its first memory-enabled agent

Start smaller than the vendors suggest. For a first agent — a customer-service assistant for a 20-to-50-person company, say — begin with session memory only: remember within a conversation, forget between them. Ship it, then watch where it actually hurts. Usually the pain is narrow. Customers hate repeating their order number; the agent forgets a preference it was just told. That tells you precisely which one or two facts deserve semantic memory — and that nothing else does yet.

Then add a thin persistent layer for those facts, with a forgetting policy from day one, and measure retrieval accuracy on your own real transcripts rather than a public benchmark. Keep the hot context small on purpose. This is unglamorous work, and it is the whole difference between an agent that feels like it knows your business and one that greets every returning customer as a stranger. Larger firms of 250 to 500 staff face the same architecture plus a governance layer — who can read the memory, how it is audited, how it is deleted. If you would rather not assemble this yourself, production-grade AI agent development — memory, guardrails and all — is what we build, and it is why we keep insisting that production beats demos. An agent that remembers is not a smarter model; it is a better-engineered system around an ordinary one.

Frequently asked questions