How to Build RAG in Harness Engineering
Agent Harness Engineering is not about waiting for stronger models. It is about engineering everything around the model: tools, context, memory, retrieval, state, and feedback loops.
Over the past two years, most conversations around AI applications have centered on model capability: which model reasons more reliably, which one writes better code, which one hallucinates less. That discussion matters. But once we deploy an agent in a real work environment, it becomes obvious that the model is only one part of the system.
O’Reilly’s article, _Agent Harness Engineering_, offers a useful framing:
Agent = Model + Harness
The harness is the scaffolding around the model. A raw model does not automatically become a working agent. It starts supporting long-running tasks only when it is embedded in a system that gives it state, tools, feedback, and constraints.
The Hidden Problem with Traditional RAG: Memory Breaks at Ingestion
When we look at knowledge-intensive agent systems today, this scaffolding often fails during document ingestion.
This problem is common in RAG. When the answer is bad, where did it actually fail? Was it the model, the vector database, or the way the document entered the system in the first place?
Most teams respond by upgrading to a more expensive model, rewriting the prompt, or tuning temperature. But the real failure may have happened much earlier: the right evidence was never retrieved, or the document had already lost structure during ingestion, sliced into fragments and disconnected from its source.
A more precise way to diagnose RAG failures is to split them into three layers. A recall failure means retrieval did not find the right evidence. A faithfulness failure means the model invented details that do not exist in the source. A relevance failure means the answer is grounded but does not fully address the user’s intent.
In this system, the document parsing layer plays a critical role. Tools such as Unstructured and LangChain’s text splitters should not be treated merely as parsers or RAG chunking utilities. If we view them only through the lens of parsing or chunking, they are reduced to a narrow category: turning PDFs, Office files, Markdown, and images into text blocks.
Once we adopt this flattened-fragment approach, low recall becomes a hard systemic problem to solve.
Low Recall Is Often Structural Loss, Not a Single-Point Failure
In today’s RAG engineering practice, one pattern is becoming increasingly clear: when the result is bad, we should not casually blame the model or the vector database. Low recall is usually not caused by one broken module. More often, it is the combined result of how the document was processed, how the query was expressed, how retrieval was executed, and how results were evaluated.
Among these factors, the first limit on recall is usually set by how the document enters the system.
Many RAG pipelines still rely on fixed-length chunking, splitting documents into 512- or 1,024-token blocks. This is easy to implement, but it breaks down quickly. A complete answer may be split across two chunks. Sliding windows can reduce boundary effects, but they also create duplicate content, causing similar chunks to crowd the top-k results. Chunking by heading, paragraph, or section is closer to how humans read, but only if the document structure is reliable.
Real-world PDFs, slide decks, spreadsheets, scanned files, internal wikis, and ticket histories are rarely clean enough for that. So the chunking problem is not really a question of “how long should each chunk be?” It is a question of whether the system understands the document structure at all.
Real documents are not flat text. They have titles, chapters, sections, paragraphs, tables, images, footnotes, citations, context relationships, and a reading path that moves from overview to detail. Once that structure is flattened into a pile of disconnected chunks, the agent no longer receives a document. It receives fragments that have lost their hierarchy and source.
It may know that one paragraph is semantically similar to the question, but not which chapter it belongs to, what the previous section explained, whether the next paragraph adds a constraint, or whether the nearby table is actually the key evidence. Many RAG failures begin at exactly this step.
This also means retrieval cannot rely on a single signal. For example, a user asks for the power-consumption specifications of product XR-2048, but the system retrieves documentation for XR-1024. At a generic semantic level, both are close to “power consumption specs.” From a business perspective, the model number is wrong, so the answer is wrong.
General-purpose embeddings are good at semantic similarity. They can pull “refund process” close to “return request.” But they are much less stable with exact terms such as product models, API names, error codes, contract IDs, and internal abbreviations.
BM25 helps capture exact lexical matches. A reranker can rescore relevance across a candidate set. Query rewriting can translate a user’s casual phrasing into the terminology used in the documents. Each of these solves a different blind spot in the retrieval chain.
But from the perspective of Agent Harness Engineering, the key point is not the list of optimization techniques. The deeper point is this: a reliable agent system needs more than one semantically similar passage. It needs a set of retrieval signals that can be combined.
Semantics, keywords, section paths, document hierarchy, metadata, table positions, image-text relationships, and citation sources all affect whether the agent can ultimately find the right evidence.
So RAG recall optimization is not a single tuning problem. It is a systems engineering problem. It requires document parsing, structured chunking, multi-route retrieval, reranking, query rewriting, evaluation sets, monitoring, and feedback loops to work together.
In other words, RAG evaluation and optimization are themselves part of Agent Harness Engineering. The pipelines, tools, state, and feedback mechanisms outside the model determine whether the agent can obtain correct, complete, and citable context.
Knowhere: Structured, Navigable Memory for Agents
To solve the structural loss and single-signal retrieval problems above, the harness needs a new layer of infrastructure. This is why Knowhere, our recently open-sourced GitHub repository ontos-ai/knowhere, belongs in the conversation around Agent Harness Engineering.
If Agent = Model + Harness, then Knowhere is not the model, and it is not the whole harness. It sits inside the harness as the document memory, search, and context layer.
The problem it addresses is simple to state, but hard to solve in practice: how do large volumes of unstructured and semi-structured enterprise data, from messy spreadsheets and casually edited Excel files to poorly formatted Word documents, enter an agent’s long-term memory and become reliably searchable, navigable, and citable when needed?
Knowhere’s value does not start at the generation stage, the planner stage, or the review stage that people often talk about. It starts further upstream. In this system, a robust document-parsing layer for complex files becomes a document memory and retrieval substrate: a layer that turns complex documents into structured memory that agents can navigate, search, reuse, and cite reliably.
This is just as important as how the model reasons, how the agent plans tasks, or how hooks are used to review agent work.
In knowledge-intensive agents for finance, legal, and enterprise workflows, memory, search, and context engineering are core parts of the harness. A model does not automatically know about a newly uploaded document. It does not naturally understand the section structure of a PDF, the relationships within a table, the meaning of an image, or the source path behind a quoted answer. All of that has to be supplied to the agent workflow through an external memory system.
But the key is not to stuff the entire document into the model. Context windows are limited, and as context gets longer, models are more likely to suffer from attention dilution and weaker reasoning quality.
A useful harness does not blindly push everything into context. It decides what should be injected directly, what should stay in external memory and be retrieved on demand, what can be summarized, and which answers must include original evidence and source paths.
So the RAG problem should not only be framed as “did the retriever return text chunks?” A better question is whether the harness can organize external knowledge into a form the agent can actually use.
Knowhere is built for this layer: durable document memory and retrieval infrastructure for the agent harness. It is not doing one-time document-to-text conversion. It is not simply cutting documents into chunks and handing them to a vector database. Its larger role is to preserve as much structure and context as possible when documents enter the system, turning raw documents into multidimensional signals that downstream retrieval can use.
Technical Core: From Flat Chunks to Tree-like Chunking
But that is only the positioning. The real question is how Knowhere turns complex documents from a pile of flat chunks into a memory substrate that agents can navigate, compare, and cite.
If flat fragments do not work, then the shape of documents inside agent memory should return to the shape of the documents themselves. Knowhere approaches this through its tree-like chunking algorithm.
When a complex document enters the system, Knowhere’s output chunks are not just the result of changing chunk size. They are not just chunks with a few extra metadata fields. Conceptually, the document is treated as a structure with hierarchy, paths, and relationships.
The first layer is the tree inside a single document. A chunk is no longer just a piece of text cut out of the file. It becomes a memory node with position, a parent node, path, neighboring context, and source information. It knows which section it belongs to, which chapter that section belongs to, where the related tables and images are, whether nearby context should be read together, and whether the final evidence can be traced back to the original document.
This means that when an agent calls a retrieval tool inside the harness, it is not facing a pile of isolated top-k semantically similar snippets. It is working with document memory that can still be followed through sections, paths, relationships, and evidence sources.
In the end, the core of agentic RAG is not retrieval before generation. It is retrieval as agent action. Retrieval becomes an essential part of the harness itself.
That is the layer Knowhere is built for.
Originally published on Medium.
