Agent Memory, What to Persist, What to Forget, and Where

Pull-quote: “An agent that remembers everything is not thorough. It is a hoarder, and it reasons like one.”

Memory is three systems, not one

Agent memory fails in production for one recurring reason: teams treat it as a single bucket. It is three systems with different jobs, lifetimes, and failure modes. Working memory answers what am I doing right now. Episodic memory answers what happened and when. Semantic memory answers what do we know to be true. Blur those boundaries and you get the two classic pathologies: transcripts replayed into the window until reasoning drowns, and unverified claims quietly promoted into permanent facts.

The names come from cognitive science, but the engineering is ordinary data work: a context window, an append-only log, and a versioned store of facts.

The write path decides what the agent becomes

   during the run                          across runs
   ──────────────                          ───────────
   WORKING MEMORY (context window)
     │
     │  salient events, decisions, outcomes
     ├────────────────────────► EPISODIC STORE
     │                          append-only, provenance, TTL
     │                                   │
     │                                   │  verified facts only
     │                                   ▼  (reviewed promotion)
     │                          SEMANTIC STORE
     │                          versioned facts and entities
     │                                   │
     ▲                                   │
     └──── retrieval, scored and ◄───────┘
           budgeted, provenance attached

The design question is not which vector database to buy. It is what crosses each arrow. From working to episodic: salient events, decisions, and outcomes, each written with provenance, a source, a timestamp, a confidence. From episodic to semantic: only what survives verification. We treat that promotion the way a data team treats a bronze-to-gold pipeline. Facts are reviewed in, not accumulated in, because a semantic store that accepts unreviewed claims is not memory, it is rumor with an index. The stores themselves can stay boring: a relational table with pgvector beside the rows covers episodic retrieval and semantic search for most systems until scale says otherwise.

Persist little, forget on purpose

1. Persist decisions and outcomes, not transcripts. The transcript is evidence for the log, not material for the next prompt.
2. Promote, do not accumulate. A fact enters semantic memory through verification or it does not enter.
3. Attach provenance to every record. A memory that cannot say where it came from cannot be trusted, corrected, or deleted properly.
4. Give episodic memory a TTL. Expiry is the default, and retention is the exception that needs a reason.
5. Make forgetting provable. In regulated deployments, what the agent remembers about whom is a compliance question, and deletion has to be demonstrable rather than aspirational.

The read path deserves the same discipline as the write path. Retrieval into working memory is a budgeted act: scored candidates, a relevance threshold, provenance attached to every item that enters the window. An agent that pulls twenty memories to use two has not become wiser, it has become slower and more distractible, and the retrieval log will not tell you which memory actually changed the decision unless you record that too.

In practice

Investigative workloads show the full architecture: each investigation gets a case file, an episodic evidence log where every entry carries its source, and a semantic layer of entities and relationships distilled only from reviewed evidence. A public-facing assistant warrants the deliberate opposite: session-scoped working memory and nothing else, because a public agent that quietly builds visitor profiles would be a liability wearing a feature’s name.

Closing

Memory architecture is a data problem wearing an AI costume. Separate the stores by the question each answers, write forward with provenance, promote through verification, and design forgetting as carefully as remembering. The agent reasons better precisely because you gave it less to hold.