Agent memory: what to persist, what to forget, and where

Pull-quote: “If one tool call can answer it, do not remember it, look it up. A stored copy of live state is a stale copy waiting to happen.”

Agent memory is three different problems wearing one name, and conflating them is why an agent feels amnesiac on Monday and confidently wrong on Friday. One store holding your timezone, a seven-step release procedure, and last Tuesday’s failed run serves all three badly, because each has a different write frequency, retrieval key, and expiry.

This post is for engineers building or operating an agent harness whose agent forgets something obvious or repeats a fact that stopped being true weeks ago. It assumes the vocabulary from the harness architecture post: the loop, state, tools, memory, budgets, and verification.

By the end you will have a layered design, a write policy that decides what earns persistence, a read policy that fits your context budget, and a plan for staleness, deletion, and privacy.

What you will be able to do

• Separate semantic, procedural, and episodic memory, and give each the retrieval key it needs.
• Write a memory write policy with a gate, so the store gains precision instead of volume.
• Cap what memory contributes to a prompt with a fixed token allocation and a relevance floor.
• Detect a stale fact, supersede it without losing the audit trail, and resolve records that disagree.
• Implement decay, archival, and real deletion, including the cascade into derived summaries.
• Make a memory-dependent agent reproducible by treating the store as a test fixture.

What are the three layers of agent memory?

Agent memory has three layers because an agent needs three kinds of durable knowledge, and each is retrieved by a different key: semantic holds facts, procedural holds skills, episodic holds trajectories.

Semantic memory is the set of durable facts about the user, the domain, and the active projects, stored as assertions that stay true across sessions. Each record holds one claim with a subject, a source, and the date it was observed, such as “production deploys require two approvals from the platform team”.

Procedural memory is reusable how-to knowledge, meaning named skills the agent can follow again, including workflows it learned by noticing a task repeat. A skill is a document: preconditions, ordered steps, the tool each step uses, and how to tell the step worked. Hermes Agent keeps skills as Markdown under ~/.hermes/, written when it sees a workflow twice.

Episodic memory is the time-stamped record of what happened, meaning trajectories: the ordered sequence of observations, tool calls, results, and outcomes for one run. You read it when something broke and export it to build an evaluation set, rarely to fill a prompt.

Layer	Holds	Retrieval key	Write rate	Lifetime
Semantic	One assertion, with subject and source	Relevance to the task	Rare, gated	Until superseded
Procedural	Named procedures, loaded whole	Task type match	On repeat or correction	Until the procedure changes
Episodic	Trajectories, append only	Time and task identity	Every run	Retention window, then archive

What earns a place in memory, and who decides?

A fact earns persistence when it passes three tests: it will still be true later, a future task will need it, and it cannot be cheaply re-observed. Fail one and it belongs in the current context window or in a log.

The third test does most of the work. Current branch, ticket status, account balance: that is live state, and a stored copy of live state is a stale copy waiting to happen. If one tool call can answer it, do not remember it, look it up. Memory is for stated preferences, agreed conventions, corrections the user made, and constraints that live in nobody’s database.

Three mechanisms decide what gets written, trading recall against precision.

• Model-proposed. The model emits a candidate when it notices something durable. High recall, low precision, and it proposes its own inferences as facts.
• Rule-based. Deterministic extraction into slots you defined: preferred name, timezone, default repository. High precision, and it misses what you did not model.
• Human-confirmed. The agent proposes, the person accepts or edits. Highest precision, costs an interruption, so spend it where being wrong is hard to undo.

Model-proposed candidates through a rule-based gate is the workable default. A write is a tool call with a schema, not an append to a text file.

{
  "tool": "memory_write",
  "args": {
    "layer": "semantic",
    "subject": "project:checkout-service",
    "assertion": "Production deploys require two approvals from the platform team.",
    "provenance": "stated_by_user",
    "observed_at": "2026-08-11T14:22:05Z",
    "source_ref": "session:7f3a91#msg42",
    "review_after_days": 180,
    "supersedes": null
  }
}

The provenance field separates what the user said, what a system returned, and what the model inferred, and later conflicts are resolved on that distinction. An inference stored as an unmarked fact is the seed of a confident error, because nothing in the record says it was a guess.

Writing everything fails worse than writing nothing. Writing nothing gives an agent that asks the same question every session, which is annoying and obvious. Writing everything gives four near-duplicates disagreeing about one subject, where scoring cannot tell which is current, so the answer comes from whichever scored highest. That failure is invisible.

How much memory should you read back into the context window?

Give memory a fixed token allocation in the prompt, decided in advance, and never let it grow with the size of the store. Memory competes with system instructions, tool definitions, history, and retrieved evidence for the same scarce window, which is the discipline in treating the context window as a budget.

Pick a number and hold it. If memory gets 1,200 tokens of a 128,000 token window, it gets 1,200 tokens whether the store holds 30 records or 3,000. Four rules fill it.

1. A pinned core. A few records loaded every turn regardless of retrieval: identity, timezone, hard constraints, active project. Pinning has a hard cap.
2. Scored retrieval with a floor. Score on relevance, weight by recency and provenance, and return nothing when nothing clears the floor.
3. Whole skills, never fragments. Load a matched procedure complete, because the model improvises the missing steps and the improvisation looks like the rest.
4. Episodic by summary only. Load a short summary of the last relevant run, or nothing.

Render every admitted record with its provenance and observation date. That is the cheapest staleness defence available, because the model can then discount a June fact against a tool result from five seconds ago.

MEMORY (3 records, 412 of 1200 tokens)
- [semantic · stated by user · observed 2026-06-02] Deploys need two platform approvals.
- [semantic · from system · observed 2026-07-19] Default branch is main.
- [procedural · skill v4 · updated 2026-07-30] release-checkout-service, 7 steps.

You know the read policy works when your harness can print, for any single turn, the records it admitted and the score that admitted each one. Without that, memory is a black box inside your loop.

How do you know a stored fact has gone stale?

You detect staleness by re-checking assertions against a system of record at the moment they matter, because a store cannot notice that the world changed. Timestamps and expiry dates help you triage, they detect nothing.

Four mechanisms earn their keep. Timestamp every record with when the fact was observed rather than when it was written. Give a review window to facts with a known cadence, such as quarterly plans and price lists, and leave facts with no natural cadence with no expiry, because a wrong expiry either drops something durable or renews something dead. Run a conflict check on write by first retrieving records with the same subject. And verify at use for anything a tool can confirm in one round trip.

When two memories disagree, resolve in a fixed order. A fresh observation from a system of record beats any remembered value. A more recent record beats an older one. A user-stated fact beats a model-inferred one. A tie none of those settle goes to the person, because silent tie-breaking is how an agent develops opinions nobody gave it.

Supersede, do not overwrite. Write the new record with a supersedes pointer, mark the old one inactive with superseded_at, and keep it. Overwriting destroys your ability to replay what the agent believed on the day it made the decision you are now investigating, which is why stale memory is one of the harder production failures to diagnose after the fact.

Why is forgetting a feature rather than a bug?

Forgetting is a first-class operation because a store’s only real asset is precision, and precision survives only if records leave. A store that grows monotonically converges on a store where nothing can be found.

You need three distinct mechanisms. Decay penalises age and non-use, so records nothing has matched in months sink below the read floor long before anyone deletes them, and one relevant hit lifts them back. Archival moves records into cold storage, where they stop competing for the read allocation but stay available for audit. Deletion is irreversible removal, for records the user asks you to drop and facts that were simply wrong.

Deletion has a trap. Deleting a source record does not delete the summary that absorbed it, so if you compress episodes into semantic facts, deletion must cascade into every derived artefact or the data survives in something nobody thinks of as a copy. Under a privacy regime that grants a deletion right, real deletion covers the index, the cold store, the summaries, and the backups inside your retention window.

Give the user a way to list what the agent stored about them, correct a record, and delete one. The first time the agent says something odd, that view turns a mystery into a two-minute fix.

What must never be written to durable memory?

Credentials, tokens, API keys, full payment instruments, government identifiers, and the raw contents of documents shared for a single task must never reach the durable store. A prompt lives for one turn; a memory record lives until somebody deletes it, which makes the store the highest-risk surface in the harness.

Redact in the write path, not the read path. Redaction on read means the sensitive value is already on disk, in your backups, and in whatever replicated that volume. Reject a candidate containing a detected secret rather than masking part of it.

Per-tenant isolation has one rule that matters: the tenant key comes from the authenticated session and the memory service applies the filter server-side. If the tenant identifier arrives as an argument the model can set, or as a prompt instruction asking the retriever to stay in its lane, you have built a cross-tenant leak with extra steps. Prompt text is not an access control mechanism.

Two boundaries remain. Shared team memory needs its own scope and write path, because a fact one person stated privately should not become team knowledge through unscoped retrieval. And memory is untrusted input whenever untrusted text can reach it: a summary of attacker-controlled content in the semantic layer gives that attacker durable persistence inside your prompt, which is prompt injection with a much longer lifetime.

How does memory break your evaluation harness?

A memory-dependent agent is not reproducible unless memory state is part of the fixture, so treat the store as an input to the test rather than as part of the environment. Two runs of one task against a mutable shared store are two different experiments.

Three practices follow. Seed the store per case and reset between cases, so a pass cannot be explained by a record another test wrote. Assert on the write path, since a regression that stops persisting corrections is invisible to output-only grading. And seed a stale record that contradicts what a tool will report, then check that the agent prefers the fresh observation.

An agent that uses the right memory is table stakes; an agent that declines to use the wrong memory is one you can put in front of a customer. Grading whole runs is the subject of building an evaluation harness for agents.

Where this goes wrong

• Confidently stale. The agent asserts something that was true in March, because the observation date is stored but never rendered into the prompt. Render dates and provenance, add review windows, and route cheap-to-check facts to a tool.
• Memory crowding out evidence. Quality drops as the store grows and retrieved documents arrive truncated, because the memory allocation grew with the store. Fix the allocation and add a relevance floor.
• Contradiction by accretion. Output flips between runs because writes were appended without a conflict check, so scoring picks a different winner each time. Add the subject-level check on write and supersede the loser.
• Cross-tenant bleed. One user sees another user’s preference, because the tenant filter lived in prompt text or came from an argument the model could set. Move the filter server-side and test reading across the boundary.
• Skill drift. The agent follows a procedure that works around a bug fixed last quarter, because procedural memory was written on repetition and never reviewed. Version and date skills, and let a failed run mark the skill it used.

Common questions

Is agent memory the same thing as RAG?

No. Retrieval-augmented generation retrieves from a corpus you did not write, such as your documentation. Memory retrieves from a store the agent itself wrote, which gives it a write policy, a provenance model, and a deletion obligation a document corpus does not have. The machinery can be identical; the governance is not.

Do I need a vector database for agent memory?

Not at first. A few hundred semantic records are well served by a relational table with a full-text index, or by plain Markdown files a person can read. Move to embeddings when the store outgrows keyword recall, and keep the assertion text as the source of truth rather than the vector.

Should the model decide what to remember?

It should propose, not decide. Model-proposed candidates give you recall, a rule-based gate gives you precision, and human confirmation covers records whose consequences are hard to reverse. Letting the model write straight to durable storage is how stores fill with its own inferences.

Does a large context window remove the need for memory?

No. Context is per-request, paid for on every request, and gone when the request ends, while memory is durable and selective. A bigger window makes the read policy more forgiving rather than unnecessary, and you still would not pay to re-read a thousand records per question.

Next steps

1. Instrument the read path first. Log which records were admitted each turn, their scores, and the tokens they consumed, because you cannot design a write policy for reads you cannot see.
2. Continue the course with tool design for agents. Memory tells the agent what is true, tools are how it changes anything, and a tool contract needs the same precision as a memory record.
3. To see all three layers as plain local files you can open in a text editor, read how Hermes Agent remembers and how skills make it improve.
4. This write policy, with redaction before persistence and human confirmation on any record that affects a filing, runs behind ComplyGrid in federal compliance work, where the audit question is what the agent believed and when it learned it.