Why RAG needs a World Model too
Fei-Fei Li’s recent piece on world models made me revisit a question that we often skip when talking about agents.
Not whether an agent sees enough.
But whether it has a world inside the system that can change, persist, and be acted on.
At first glance, world models sound like a physical AI topic. Robots need to understand rooms. Autonomous driving systems need to understand roads. Video models need to generate scenes that evolve over time.
But the part I find most interesting is not limited to physical space.
Knowledge-work agents have the same problem. Their world is not a 3D room. Their world is a document space.
A picture, a video frame, or a paragraph retrieved from a document is an observation. It can be detailed, relevant, and accurate. But an observation is not the world itself.
A model can generate a video that looks like a room without knowing what objects exist in the room, how they relate to each other, or what happens if one object moves.
In the same way, an agent can retrieve a paragraph that looks relevant without knowing which document it came from, which version it belongs to, what section it sits in, or which table supports it.
That is why I find world models useful as a lens, not as a buzzword.
They force us to ask three simple questions:
- What does the system see?
- What state does it believe the world is in?
- What can it do next?
In some discussions, these map to renderer, simulator, and planner. The names matter less than the separation.
The first layer presents the world to the model: a frame, a paragraph, a snippet, a summary.
The second layer maintains a state that is more stable than a single observation: what objects exist, how they relate, what constraints hold, and what might happen after an action.
The third layer chooses what to do next based on that state.
In physical AI, this is obvious. A robot cannot just recognize “cup” and “table.” It needs to know that the cup is on the table, that the table has an edge, and that moving an arm may cause a collision.
But enterprise agents face the same structure in a different medium.
Their world is made of PDFs, decks, spreadsheets, contracts, reports, screenshots, product docs, meeting notes, comments, versions, citations, permissions, and links.
This world has its own geography.
Sections are terrain. Page numbers are coordinates. Tables and images are objects. Citations are paths. Versions are time. Permissions are boundaries. Comments and revisions are traces of past actions.
If an agent treats all of that as flat text, the problem starts before reasoning begins.
The world has already been compressed into snippets.
This is where traditional RAG starts to show its limits.
A common RAG pipeline looks like this:
document -> chunk -> embedding/search -> top-k context -> answer
This pipeline is useful. For FAQs, definitions, and finding a specific passage, it is often the simplest and most reliable path.
The problem is not that RAG is useless. The problem is that it often reduces document work into text retrieval.
For one-shot question answering, that may be enough.
For long-running work, it breaks down.
An agent can quote the right paragraph but not know whether it belongs to the current version. It can summarize all comments but not know which ones have already been resolved. It can move a number from a report into a memo but not know which table it came from, whether the number has changed, or whether it invalidates a later conclusion.
This is not always a reading failure. Often, the model did read the relevant text.
The failure is deeper: the agent received an observation, not a state.
Context is temporary material placed into a prompt.
State is the persistent current condition of the task: documents, sections, evidence, versions, citations, unresolved decisions, and what has already changed.
Context can be truncated, compressed, or forgotten.
State should persist across actions.
That is why I think document agents need a document world state.
Not because documents are physical worlds.
But because agents need something stable to return to, inspect, update, compare, and cite.
A minimal document world state might look like this:
Document -> Section -> Chunk -> Asset -> Metadata -> Source path -> Graph link -> Retrieval trace
The point is not prettier storage.
The point is to give the agent objects it can act on.
A Document tells the system where evidence belongs. A Section preserves hierarchy and scope. A Chunk is not just content, but content with position, type, path, and source. An Asset keeps tables, images, pages, and slides from disappearing from the pipeline. Metadata preserves page numbers, filenames, summaries, and other signals. Graph links capture relationships across sections, documents, and assets. Retrieval traces make it possible to inspect where an answer came from.
Once those objects exist, the agent’s action space changes.
With only top-k snippets, the agent can mostly read and answer.
With document state, it can search, open a document, expand a section, inspect a table or image, compare chunks, follow a graph edge, cite a source path, and revise its query with a reason.
This is also why flat chunking loses more than boundary precision.
It turns a document from a structured object into isolated slips of paper.
You can improve ranking over those slips of paper, but the agent still cannot move through the document world.
Tree-like chunking matters because it changes the shape of the retrieval object. A chunk is no longer just content. It is a node in a document tree. It knows which document it belongs to, which section it sits under, whether it connects to a table or image, where the source page is, and how it relates to nearby nodes.
This is a more basic question than “how do we optimize RAG?”
It asks: what kind of object is the agent actually consuming?
This is where Knowhere fits inside Noire.
Its job is not simply “PDF to text.”
If Noire wants agents to handle complex documents, it cannot just parse files into text and pass that text to a model. It needs a document state layer.
Knowhere turns messy documents into a persistent, navigable state:
dirty documents -> parse -> hierarchy / sections -> chunks / assets / metadata -> graph -> agentic retrieval -> cited evidence
The important part is not the PDF parser itself. Parsing into Markdown is only the entrance.
What defines Knowhere is what happens after parsing: document trees, section paths, chunk metadata, table and image links, lightweight graph relationships, retrieval tables, and navigation interfaces that an agent can use.
In engineering terms, the system is turning one-time file processing into durable document state publication.
The API creates jobs. Workers parse files. Parsers normalize different formats into intermediate output. Chunk converters structure that output. The publication step writes Documents, Sections, Chunks, graph nodes, and graph edges into the system.
After that, retrieval is no longer searching through temporary text. It is searching through state objects.
This also draws a boundary around Knowhere.
Knowhere is not the model. It is not the full agent harness. It does not replace sandboxing, permissions, planning, task queues, or execution loops.
It also does not replace vector search, BM25, Milvus, or GraphRAG.
Those systems can still be part of the stack. Vector search handles similarity. BM25 handles lexical matching. GraphRAG handles entity relationships and graph reasoning.
Knowhere sits earlier in the pipeline. It asks whether the document entered the system with structure, paths, assets, sources, and relationships intact.
This changes how retrieval should be evaluated.
Recall@k still matters, but it is not enough.
For complex document work, we also need to ask:
- Can the answer return to the original document, section, page, and asset?
- Can the agent expand the surrounding context after a result is found.
- Does the evidence include the necessary assumptions, exceptions, and table relationships?
- Are the same entities and metrics consistent across steps?
- When multiple documents conflict, does the system expose the conflict, or hide it?
These are not as simple as “did the answer look good?”
But they are closer to what production agents need.
The next layer is finer-grained semantic relationships: same entity, same metric, same event, supporting evidence, conflicting evidence, time alignment, version alignment, and source path.
Those relationships require extraction, normalization, evidence typing, timeline alignment, and conflict detection.
Once they become stable graph relationships, Knowhere can move from “documents are navigable” toward “evidence is comparable, verifiable, and traceable.”
It is important not to overstate the claim.
A document world state is not a physical world model. It does not model geometry, physics, contact, materials, or robot actions.
It is a domain-specific state representation for document work.
Traditional RAG is also not obsolete. For short QA, simple retrieval, and low-risk lookups, flat retrieval is often enough.
The problem appears when tasks become longer, documents become messier, and evidence needs to be checked across versions, sources, tables, and decisions.
A larger context window helps, but it is not state management.
It lets the model see more at once. It does not automatically recover document hierarchy, asset relationships, version differences, or citation paths.
GraphRAG is also not a complete answer by itself. Graphs are valuable, but graph quality depends on ingestion quality: parsing, asset preservation, entity normalization, evidence extraction, and version management.
If the document world is damaged before it enters the graph, the graph only stores damaged information in a different shape.
So the core argument is simple:
Agent = Model + Harness is still useful.
But for complex knowledge work, the harness needs an explicit World State layer.
Physical AI needs physical world state: scenes, objects, constraints, and action consequences.
Knowledge-work AI needs document world state: documents, sections, chunks, assets, sources, versions, citations, relationships, and traces.
Without that layer, agents remain reactive. They see observations, respond to them, and then start over again on the next step.
With that layer, agents can return to the same world, inspect it, compare evidence, update their understanding, and take more reliable actions.
Knowhere’s role in Noire is to provide that document world state.
Not more context for its own sake.
Not another wrapper around RAG.
But a persistent, inspectable, actionable map of the documents and evidence the agent is operating on.
The next generation of document agents will not just read more.
They will know what they are reading, where it lives, what it depends on, what changed, and what action is safe to take next.
The missing piece is not more context.
It is actionable knowledge state.
Knowhere:https://knowhereto.ai/?utm_source=medium
Originally published on Medium.
