Knowledge graphs as the memory layer agents need
Pull-quote: “The deliverable was never the graph. It is the right forty nodes, assembled while the question is still open.”
A knowledge graph becomes agent memory the moment you can name the subgraph a decision needs and assemble it in one bounded query. Before that, it is a well-governed store an agent has to search, which puts you back where you started: precise structure, probabilistic retrieval across all of it.
This post is for teams who built or bought a graph and found agent quality did not move. It covers what a graph gives you that a vector index cannot, what the ontology is for, and the three-step pattern that turns a warehouse of facts into a working set. It assumes you have seen retrieval-augmented generation and know what an embedding is.
Two terms get used interchangeably and mean different things. A knowledge graph stores typed entities and the typed, directed relationships between them, where relationships are data you query rather than joins you must know in advance. A context graph is the small subgraph assembled for one decision, holding durable facts and live state, discarded once the decision is made. The first is architecture. The second is what the agent reads.
What you will be able to do
- Say when a question needs a traversal and when similarity search is sufficient.
- Write an ontology that makes a traversal legal rather than lucky, and enforce it.
- Assemble a context subgraph in three steps: resolve the anchor, bound the traversal, hydrate live state.
- Keep a subgraph small with an edge allowlist, a hop cap, a node budget, and degree limits on hub nodes.
- Separate facts the organisation owns from facts about the run, and set freshness rules for each.
What is a knowledge graph, and how does it differ from a vector index?
The retrieval unit differs: a vector index returns text chunks ranked by similarity to your question, and a graph returns a traversal from a specific entity along declared relationships.
A property graph models nodes with labels and properties, connected by directed, typed relationships that also carry properties. A vector index stores text as embeddings, numeric vectors positioned so similar passages sit near each other, and returns the nearest. Both are retrieval, for different question shapes.
| Question shape | What the answer needs | The right store |
|---|---|---|
| “What does the maintenance policy say about part replacement?” | One passage, found by meaning | Vector index |
| “Which plants would stop if this supplier failed?” | Several hops from a named entity | Graph traversal |
| “What changed about this contract in the last review?” | A specific document, found by identifier | Lexical search |
| “Summarise the themes across 4,000 incident reports” | Structure over the whole corpus | Graph with community summaries |
The second row is the one vector search cannot reach. No chunk says “these four plants depend on this supplier”, because that fact exists only as a path across records never written down together. Similarity search returns text that exists. A traversal computes an answer that does not.
Why does a bigger, more correct graph make an agent perform worse?
Because handing the whole graph to an agent replaces the structure you built with a similarity search over it, and the context window pays for every node you send.
Two mechanisms compound. The first is retrieval: an agent given tens of thousands of nodes has to decide which matter, and its only tool is the probabilistic ranking you built the graph to avoid. The second is attention. Lost in the Middle showed accuracy following a U-shaped curve against where relevant information sits, highest at the start or the end and degrading sharply in between. In that multi-document setup, GPT-3.5-Turbo scored worse with 20 or 30 documents in the middle position than with no documents at all, and performance saturated well before retriever recall did.
So the failure is not a missing graph or a badly modelled one. It is an unfiltered one. Forty nodes an agent can hold beats four thousand it has to search, and the context budget says the same thing from the cost side.
What does the ontology underneath the graph actually buy you?
It makes traversal legal rather than lucky, by declaring which entity types exist, which relationships may connect them, in which direction, and with what cardinality.
A taxonomy is a hierarchy answering what kind of thing something is: a laptop is a computer is a product. An ontology declares what may connect to what: a supplier supplies a part, a part is contained in an assembly, a machine uses a part, a machine is located in a plant. Taxonomy classifies. Ontology permits.
Three operational consequences follow:
- Traversals become writable in advance. If the only path from supplier to plant runs through part and machine, that path is a query you write once per question class rather than something the agent discovers per request.
- Bad edges get rejected at write time. A constraint language such as SHACL, or a schema check in your loader, stops a pipeline creating a relationship the model disallows. Without it, the graph acquires edges nobody intended that every traversal must defend against.
- The agent can be told what it may traverse. An edge-type allowlist per question class is only expressible if edge types are declared and stable.
Vendors are converging on this. Databricks Genie Ontology, in preview since Data + AI Summit 2026, builds a knowledge graph of business terms, entities, and metrics from tables, queries, dashboards, and connected applications, then uses a ranking algorithm called OntoRank to extract candidate snippets, rank the authoritative ones, apply the asking user’s permissions, and inject the survivors into the agent loop. Note the order: rank, filter, then inject. Governing those definitions is a separate discipline, covered in ontology governance and the semantic layer.
When should the subgraph be assembled, at index time or at question time?
At question time, because the useful part of the graph is defined by the decision rather than the domain, and much of it is live state that would be wrong if cached.
Three steps, in this order:
- Resolve the anchor. Map the entities named in the question to node identifiers. Teams skip this step, and it determines everything downstream: a traversal from the wrong node returns a confident, well-formed, entirely wrong answer. Use canonical names, an alias table, and a lexical match, and treat an ambiguous anchor as a question to ask, not a tie to break.
- Traverse under declared bounds. Follow only the edge types this question class allows, to a fixed hop limit, with permission predicates applied inside the traversal rather than after it.
- Hydrate live state. The graph knows a supplier supplies a part. It should not be the system of record for how many are in stock this morning. Read volatile values from the operational source at assembly time and stamp each with its read time.
In Cypher, steps one and two look like this. The relationship quantifier is the current GQL-conformant syntax; the older *0..2 form still parses in Neo4j but is not.
// 1 · anchor, resolved before any traversal begins
MATCH (s:Supplier {id: $supplierId})-[:SUPPLIES]->(p:Part)
// 2 · bounded traversal: three named edge types, at most two hops of assembly
MATCH (p)-[:CONTAINS]-{0,2}(a:Part)<-[:USES]-(m:Machine)-[:LOCATED_IN]->(pl:Plant)
WHERE pl.id IN $authorisedPlants
RETURN s.name, a.name, m.id, pl.name
LIMIT 200The assembly works when the returned subgraph fits in a few thousand tokens and someone on your team can read it and agree it is the right context. If it needs a scroll bar, the bounds are wrong.
How do you keep a traversal small enough to reason over?
With four bounds declared per question class rather than per query: an edge-type allowlist, a hop cap, a node budget with an ordering rule, and a degree limit on hub nodes.
The edge allowlist does most of the work. A supply-risk question traverses SUPPLIES, CONTAINS, USES, and LOCATED_IN and nothing else, so an unrelated MENTIONED_IN edge added by a document pipeline cannot widen the result. A hop cap of two or three follows, because the third hop is where relevance ends and combinatorics begin.
The node budget needs an ordering rule. A bare LIMIT 200 truncates arbitrarily, so the node that mattered may be the one dropped. Order by something the domain cares about, such as recency, criticality, or a precomputed centrality score, then cap.
Hub nodes deserve their own control. A country, a currency, or a standard fastener may connect to hundreds of thousands of nodes, and any traversal passing through one explodes. Either exclude those types from the allowlist for that question class, or cap the edges you follow out of a single node. This is what turns a fast query into a timeout after someone loads a new reference dataset.
How do graph retrieval and vector retrieval work together?
They work in one of two orders, chosen by whether the question names an entity you can resolve.
Graph first, then vector. The question names something specific, so you resolve the anchor, assemble the subgraph, collect the documents attached to those nodes, and rank within that smaller set. Precision comes from the traversal; the ranker breaks ties. Use this whenever an entity is nameable, because it removes the failure class where a similar-sounding passage about a different supplier wins on cosine distance.
Vector first, then graph. The question is exploratory and names no resolvable entity, so you search text, extract the entities in the top passages, and expand from there. Use this for questions starting with “why” or “what patterns”.
Microsoft GraphRAG builds the graph from text rather than from a database. Its indexing pipeline splits documents into text units, 1,200 tokens by default, extracts entities and relationships with a model, runs hierarchical Leiden community detection over the result, and writes a summary report per community. Its query engine offers local search for entity-anchored questions, global search that maps and reduces across every community report, and DRIFT search, which starts locally and widens. Global search is useful and expensive, since it touches every report, so reach for it when the question is about the corpus rather than a thing in it.
What belongs in the graph, and what belongs in the agent’s own memory?
The graph holds facts the organisation owns and other systems depend on. Agent memory holds facts about the work: what this user prefers, what this run has already tried, which approach failed last time.
The test is ownership. If a fact would still need to be true with this agent switched off tomorrow, it belongs in the graph, under the change control the rest of your data has. If it exists only because of a conversation, it belongs in the agent’s memory layers, which agent memory architecture covers in detail. Mixing them gives you the worst outcome available: a model’s guess about a supplier relationship, written into the enterprise graph, indistinguishable from a fact a steward approved.
Freshness cuts across both. Split every property into durable and live. Durable properties change on a human timescale and are safe to read from the graph. Live properties are read from the operational source at assembly time, never cached, and stamped with a read time so a downstream step can see the value is nine minutes old. An agent acting on a seat count from last Tuesday is not a retrieval failure. It is a design decision nobody made.
Where this goes wrong
The graph exists and the ontology does not. The symptom is every retrieval query hardcoding its own path, no two written the same way, because entities and edges were loaded without declaring which combinations are permitted. Write the types down, enforce them at write time, and traversals become reusable.
Entity resolution is treated as a search problem. The symptom is confident answers about the wrong subject at a low but persistent rate, because the anchor came from nearest-neighbour similarity over names, which cheerfully returns the other company called Acme. Anchor on identifiers and curated aliases, and escalate ties.
Traversal bounds live in the query, not the configuration. The symptom is one question class melting down after an unrelated pipeline adds an edge type, because bounds are whatever the query author wrote. Declare the allowlist, hop cap, and node budget per question class, in configuration you can audit.
Nobody owns the edges. The symptom is a graph accurate at launch and quietly wrong six months later, with no alert, because the load pipeline has no freshness contract. Give every relationship type an owner, a source, and a staleness threshold that raises an error, not a warning.
Common questions
Do I need a graph database, or can I model this in the warehouse?
You can model a graph in relational tables, and for two-hop questions over a stable schema that is often the better choice, because the data is already governed there. Reach for a graph database when path length is variable, when you need shortest-path or centrality operations, or when relationship types change faster than schema migrations allow.
How large should an assembled subgraph be?
Small enough that a person on your team can read it and agree it is the right context. In practice that means tens of nodes rather than thousands, and a few thousand tokens rather than tens of thousands. If you cannot state a node budget for a question class, the class is not defined tightly enough.
Does GraphRAG replace vector search?
No. It builds a graph from unstructured text and adds community summaries on top, and it still embeds text units and entity descriptions for retrieval. Treat the graph as the thing that decides what is in scope and the vector index as the thing that ranks within that scope. Global search over community reports answers a question shape neither handles alone.
Who should own the enterprise graph?
The same people who own the semantic definitions it encodes, usually a data governance function rather than the team building the agent. An agent team that owns the graph will optimise it for one application, and the next application will build a second graph. That is where the investment stops compounding.
Next steps
Take one decision your agent makes badly and write down the subgraph it should have had: the nodes, the edge types, the hop count, and which properties must be live. That artefact tells you whether the problem is a missing graph, a missing ontology, or a missing bound, and the three have different fixes.
Then read AI systems fail quietly at the retrieval layer, on measuring whether the context you assembled was the right context, since a graph gives no more feedback about a miss than a vector index does. More engineering writing sits in the Signals index.
Assembling entity context at question time, bounded per question class rather than per query, is the pattern behind Aquil in geopolitical intelligence work, where relationships between actors matter more than any single document about them.