The Four Eras of OCR: From Manual Features to Multimodal Large Models, A History of Text…

OCR (Optical Character Recognition) might be one of the oldest problems in the field of computer vision. From the mechanical device in 1929 to the end-to-end in 2026, it has traversed nearly a century.

But what is truly interesting is that the four upgrades in this field almost perfectly map the evolutionary trajectory of the entire AI technology stack.

Next, I will take you through the four eras of OCR, and the technological motivations behind each leap

OCR 0.5 Era: Rules and Feature Matching (1929–2012)

This era spans the longest, and the machine is like a highly myopic, rote-learning old man. Its working logic is very simple: a human first tells it what a character looks like, and it takes a caliper to compare them on paper one by one.

In 1929, Austrian engineer Gustav Tauschek applied for the first OCR patent in human history — a mechanical device that uses template matching to recognize characters.

This device is very simple: use light to illuminate the paper, black ink will absorb light, while white paper will reflect light, and then utilize a disk engraved with character cutouts, by rotating this special disk to match, when the phototube behind receives reflected light, it means it is not this character, and when no light passes through the cutout, it means the match is successful, and then output the corresponding character through a mechanical device.

By the 1950s and 60s, OCR began to be commercialized. Companies like IBM and RCA launched OCR devices capable of recognizing specific fonts, mainly used for bank checks and postal codes. In order to allow machines to better recognize fonts, two fonts were even born: OCR-A and OCR-B.

In 1974, American inventor Ray Kurzweil encountered a blind person on a flight, and this blind person told him that visual impairment did not bring him travel obstacles, but the biggest obstacle was reading, so he did something that changed the history of OCR: founded Kurzweil Computer Products, and created the first omni-font OCR system, which is an OCR system capable of handling almost any printed font, rather than just specific standard fonts.

In order to realize the omni-font OCR system, Ray Kurzweil abandoned traditional template matching, and instead extracted character features to achieve text recognition, this algorithm will first analyze the structural features of characters, such as the direction of line segments, where the line segments intersect, etc., for example, the character A is defined as two diagonal lines intersecting at the top, connected by a horizontal line in the middle, while the lowercase b is defined as a long vertical line on the left, connected to a closed loop on the bottom right. Through this method of dismantling features, the machine acquired the ability to recognize all fonts.

Afterwards, it is still necessary to match the features to the corresponding text, Kurzweil’s team wrote a complex rule engine, the system will input the extracted features into the rule base for probability scoring and logical matching, and when these feature combinations satisfy the rule conditions of a letter, the system will output the corresponding ASCII character.

At the same time, in order to solve the problem that some font features are almost identical, such as uppercase O and number 0, lowercase l and number 1, Kurzweil’s system also introduced a post-processing mechanism based on vocabulary and linguistic rules, and the system will make logical judgments based on context: if this character appears in the word App_e, then it must be the letter l; if it appears in the formula 199_, then it is highly likely a number.

Next, Ray Kurzweil invented the CCD flatbed scanner and the TTS text-to-speech synthesizer, and combined them with OCR to create the famous Kurzweil Reading Machine.

Later, HP Labs started developing Tesseract in 1985, and open-sourced it in 2005, and handed it over to Google for maintenance in 2006. The development of Tesseract has gone through two main stages, the classic architecture, corresponding to versions 1.0–3.x, and the modern architecture, corresponding to versions 4.0 and later.

Since version 4.0 and later introduced deep learning, which is beyond the scope of this era, we will only discuss the classic architecture.

The recognition process of Tesseract is like an experienced grading teacher marking handwritten exam papers:

First, it converts the picture into black and white colors, and cuts out every blob of black ink, and every blob usually corresponds to a character. Then, it arranges these ink blobs into lines of text according to their positions, and then judges where the gaps between letters are and where the spaces between words are based on spacing.

The most ingenious part is its way of handling joined and broken characters: if two letters are joined together (for example, rn looks like m), it will try to break them apart from the thinnest point; if a letter is broken in half, it will try to piece the fragments back together.

And the secret to Tesseract’s extremely high accuracy lies in its two-pass reading method: the first pass first recognizes those clear and easy-to-read characters, and then learns the font style of this document from them; the second pass then uses the learned font knowledge to go back and recognize those blurry characters that were not recognized in the first pass. Just like when humans read, first figure out the author’s handwriting based on clear handwriting, and then guess those messy characters.

Finally, it will also use the built-in dictionary and linguistic rules to perform error correction (for example, q is usually followed by u), and output the final result.

However, all the above methods have a fatal weakness: severely relying on geometry, topology and exhaustive rules.

Almost all the above OCR requires writing specific rules, or recognizing specific features, however, this is simply a disaster in multiple languages, because for English, extracting features and writing rules for 26 letters, and a small amount of punctuation marks is feasible, however for languages such as Chinese, Japanese, Arabic, it is almost unrealistic, because they have complex character structures, a massive number of characters, and will deform according to the position, and the above methods also severely rely on the input, if problems such as unclear characters or deformation occur, these rules and features will fail, leading to a sharp drop in recognition rate.

This problem plagued the OCR community for decades — until the arrival of deep learning.

OCR 1.0 Era: Deep Learning Debuts (2012–2017)

In 2012, a deep learning model named AlexNet emerged out of nowhere, made a name for itself in the world’s largest image recognition competition, and dropped the error rate from 26% to 15% in one breath. It proved one thing to the whole world: as long as enough data is fed to the machine, and let it figure out the rules on its own, the effect is much better than humans writing rules manually. Since then, the field of computer vision comprehensively shifted to deep learning, and major tech giants (Google, Microsoft, Baidu, etc.) entered the market one after another, and the AI arms race officially began.

In 2015, Shi et al. published CRNN, and this paper almost redefined the way of text recognition.

It cleverly strung three types of neural networks together, like an efficient assembly line:

CNN (Convolutional Neural Network) — acts as eyes, responsible for extracting the visual features of text from pictures, such as the shape and texture of strokes

BiLSTM (Bidirectional Long Short-Term Memory Network)— acts as a brain, reading the entire line of text twice from left to right, and then from right to left, to understand the contextual relationship between characters. Just like when humans read a sentence, they will combine the context to judge what the blurry character in the middle is

CTC (Connectionist Temporal Classification)— acts as a translator, solving a tricky problem: the width of each character in the picture is different, and the signal output by the model varies in length and is full of repetitions. CTC can automatically organize these messy signals into final text (for example, automatically organizing H-ee-l-ll — o into Hello). In this way, the model does not need to know the exact position of each character, it only needs to know what the entire line of text is

Until today, the core model used at the bottom level of the text recognition APIs of some tech companies you see on the market, or the most famous open-source artifacts (such as PaddleOCR), to recognize text, is still the CRNN architecture or its variants.

This era formed a classic two-stage pipeline: detection then recognition. First, use a detection model to find the text area, crop it out, and then send it into the recognition model. This paradigm has landed widely in the industry, and has become the foundational architecture for various scanning Apps, receipt recognition, and license plate recognition.

But the two-stage pipeline has its own problems:

1. Detection and recognition are two independent steps. If the first step makes a mistake, the second step can only follow the mistake — just like the previous process on the assembly line cut crookedly, there is no way to correct it later
2. When encountering curved, dense or obscured text, it will only use rectangular boxes to select, and frequently includes irrelevant content in the same box
3. For complex documents (tables, formulas, mixed image and text layout), just finding where the text is is far from enough, it also requires understanding the layout structure of the entire page

A deeper problem is: this architecture will become forgetful when processing long texts. Just like a person reading a very long sentence, reading to the back and forgetting what was said in the front.

With the birth of the Transformer architecture in 2017, a new architecture is brewing.

OCR 1.5 Era: Transformer-Driven Refinement (2017–2023)

In 2017, Vaswani et al. published Attention Is All You Need, and the Transformer architecture swept NLP. But its impact on OCR was not accomplished in one stroke — from infiltrating to dominating, it went through about three years.

The transformation of Transformer on OCR happened in two dimensions: detection is more refined, and recognition is more powerful. Besides this, this era also gave birth to a brand new field: Document Understanding (Document AI).

Evolution on the Detection End: Bidding Farewell to Rigid Rectangles, Moving Towards Pixel-Level Polygons

In the 1.0 era, most detection models continued the line of thought of object detection, drawing a rectangular box to box out the text. But in reality, the text on seals is circular, and the text on product packaging is distorted. If a rectangular box is used, it will box in a large amount of irrelevant background and even other text.

To solve this problem, detection models turned to the image matting route — no longer drawing boxes, but like PS matting, accurately cutting out the pixels of the text area. In this direction, DBNet, jointly proposed by Huazhong University of Science and Technology and Megvii in 2020, is an insurmountable peak, and it almost ruled the detection end in the industry (and is also the core detection algorithm of PaddleOCR v2/v3).

Its core breakthrough can be explained using a metaphor:

Previous models were like a worker using a fixed standard to judge whether it is text here — for example, brightness lower than 50% counts as text. But the differences in light and color of different pictures are large, and a one-size-fits-all standard frequently makes mistakes. The genius of DBNet lies in: it let AI learn to adapt to local conditions by itself — using one standard for bright areas, and another standard for dark corners, and dynamically adjusting the judgment scale for every small area.

The result is: no matter how the text is curved, how densely it is arranged, DBNet can draw exact outlines sticking to the characters extremely fast and accurately, completely solving the detection puzzle under complex layouts.

Evolution on the Recognition End: Throwing Away CNN and RNN, Wholesale Transformer-ization

Since detection has become accurate, how should recognition break through the bottleneck of forgetfulness when reading long sentences? The answer is to directly change to a smarter brain.

In 2021, Microsoft proposed TrOCR, and this paper was like a revolution, completely flipping over the table ruled by CRNN for many years. It declared: the previous three-piece suite (CNN + RNN + CTC) are all no longer needed!

The architecture of TrOCR can be understood as two collaborating roles:

The person looking at the picture (Visual Encoder): it cuts the text line picture into small squares (such as 16×16 pixels), and then simultaneously examines the relationships between all squares — unlike before where it could only see locally, now it can survey the global view at a single glance.

The person writing words (Text Decoder): this part is essentially a language model, similar to the predecessor of ChatGPT. It receives the visual information transmitted by the person looking at the picture, and then writes out the recognition result word by word just like writing an essay.

Key advantage: traditional models easily guess randomly when they cannot see letters clearly (for example, recognizing clear as dear), because it almost does not understand language. While TrOCR comes with powerful language knowledge, and can deduce blurry words based on context — just like humans even if they see a messy character, can also guess it out based on the meaning of the entire sentence. And the forgetfulness problem of ultra-long texts is also thoroughly solved.

From Recognizing Characters to Understanding Documents: The Awakening of Multimodality

At the end of the 1.0 era we mentioned, just cutting out characters is not enough, we need to understand the tables and key-value pairs (Key-Value) on invoices, resumes, and reports.

In 2020, Microsoft Research Asia published LayoutLM, officially lifting the curtain on document understanding.

Its innovation can be understood like this: when traditional AI reads text, it is like being blindfolded and listening to people read aloud — it only knows that the two words Name and Zhang San are next to each other front and back, but does not know that Name is printed on the left, and Zhang San is written on the right, and the two are actually a label-content relationship.

The thing LayoutLM does, is taking off the blindfold of AI. It not only let the model know what each word is, but also let the model know which position each word is at on the page, and what the surroundings of this word look like.

So, the model suddenly opened its eyes to see the world. It discovered that, the bold words on the left on the invoice are usually the table header, and the aligned words on the right are usually the content. With LayoutLM, OCR no longer just spits out a pile of text, but can directly convert a complex report into structured data.

But the 1.5 era is still not perfect:

Even though DBNet solved curved text, TrOCR solved long text recognition, and LayoutLM solved layout understanding, this is still an era of jigsaw puzzles.

Currently, document understanding requires a multi-step pipeline: DBNet for detection, TrOCR for recognition (after cropping), and LayoutLM for structured extraction. The entire system is highly bloated. It suffers from slow inference and exorbitant maintenance costs, alongside the persistent issues of siloed modules and error accumulation.

It wasn’t until the end of 2023, along with the shocking debut of GPT-4V and the explosion of multimodal large models, that the OCR field finally realized: actually, we do not even need the two concepts of detection and recognition.

An ultimate revolution named Pixels as Text (Pixels-to-Text) — the OCR 2.0 Large Model Era, officially descended.

OCR 2.0 Era: The Dimensional Strike and End-to-End of Large Language Models (2023–Present)

In 2023, with the shocking release of GPT-4V, the entire AI community suddenly woke up: since multimodal large language models (MLLM) already possess incredible abilities to look at pictures and talk, then why do we still painstakingly go cut pictures, recognize characters, and arrange layouts?

The essential difference between OCR 2.0 and the 1.5 era, lies in the complete demise of the pipeline (Pipeline).

In the 1.5 era, OCR is an extremely bloated relay race. First use one model to box out text positions, then use another model to recognize characters, and finally use a third model to understand layout. This is not only slow, but what is more fatal is the domino effect — as long as the first step is boxed a little crooked, every subsequent step will follow the mistake.

In the 2.0 era, OCR has become an all-around player. No detection boxes needed, no independent recognition needed, no post-processing needed. Give the model a picture, and it directly outputs perfectly typeset Markdown, LaTeX with formulas, or structured JSON. One model, in one step, solves all problems.

During this process, a classic technical architecture that is highly representative, and can even be said to be a synthesizer, was born: GOT-OCR 2.0 (General OCR Theory, General OCR Theory).

It emerged out of nowhere in 2024, using an extremely elegant architecture and a parameter volume of less than 1B (one billion), showing the world the ultimate form of OCR 2.0. If you want to understand how OCR 2.0 operates, seeing through GOT-OCR 2.0 is enough.

Classic Technology Disassembly: The Great Unification Magic of GOT-OCR 2.0

GOT-OCR 2.0 is able to take one look at a picture, and directly output typesetting code, because it completed three revolutionary innovations on the bottom-level architecture:

Visual Encoder: From Global Zoom to Dynamic Resolution (Dynamic Resolution)

Previous AI had a fatal weakness when looking at pictures: it could only look at small photos of a fixed size (such as 224×224 pixels). If you gave it a 4K high-definition report, it would forcibly shrink the picture into a small picture, and as a result the text would all be blurry.

GOT-OCR introduces a dynamic resolution mechanism, much like how humans read a large newspaper: first scanning globally to grasp the overall layout, and then zooming in on local details. It divides a large image into smaller patches to process them individually, and finally stitches them back together in its ‘brain’. This way, whether it’s a small business card or a massive engineering drawing, it can accurately capture every word.

Text Decoder: The Dimensional Strike of Large Language Models (LLM)

Although TrOCR in the 1.5 era also used Transformer, its Decoder is just a very weak language model, and can only barely piece together words.

But GOT-OCR directly takes real large language models (such as Qwen-0.5B and other LLMs) to serve as brains. What does this mean? This means the model is not only recognizing characters, but even more so understanding the world. When it sees that there is a crossed-out typo in the picture, traditional OCR will foolishly recognize the typo out too; but with the common sense blessing of the large model, it knows that is invalid information, and will automatically ignore it. When encountering extremely blurry medical prescriptions, the large model can even accurately infer that unclear medicine name based on the pharmacological logic of context.

Interactive Paradigm: Everything can be Prompted

In the 1.0 and 1.5 eras, OCR was considered a one-way task: input a picture, and the machine spits out all text, regardless of whether you need it or not.

GOT-OCR 2.0 turned OCR into conversational interaction. You can issue various extremely refined instructions (Prompt) to it:

1. Recognize the text in the red box in the top right corner of the picture. (Regional OCR)
2. Turn this paper screenshot full of mathematical formulas and charts, directly into LaTeX source code. (Formatted OCR)
3. What are the notes on this musical staff? (Pan-OCR surpassing text)

Even molecular formulas, geometric figures, sheet music, in its eyes are just a certain kind of language. As long as the large model has seen the rules of this language, it can perfectly translate visual pixels into corresponding code.

Summary: The End and Rebirth of OCR

From using CNN to cut characters one by one in the 1.0 era, to using Transformer to understand layouts in the 1.5 era, and then to MLLM ruling everything in the current 2.0 era, OCR has walked an incredibly long but charming evolution road.

The advent of OCR 2.0 has, in a sense, heralded the demise of the traditional OCR engineer. This is because text recognition is no longer a standalone sub-field of computer vision; it has been completely subsumed and integrated into the multimodal perception capabilities of Artificial General Intelligence (AGI).

The AI of the future, no longer needs a specialized organ to read text, because when it opens its eyes and looks at the pixels of this world, it has already understood everything.

Standing at OCR 2.0, looking back

Putting the four eras together, you will discover a clear main thread: from humans defining rules to models learning rules, from step-by-step resolution to end-to-end resolution, from recognizing text to understanding documents.

OCR 0.5 Era : Adopted template matching + manual rules, able to convert scanned pictures into pure text. The biggest bottleneck is when encountering character adhesion and inaccurate cutting, the recognition rate is extremely low.

OCR 1.0 Era : Introduced deep learning (CNN+RNN) to recognize single lines of text. The primary bottleneck was its highly fragmented pipeline, where text detection and recognition operated as completely isolated modules.

OCR 1.5 Era : Adopted Transformer architecture, supports full-page document converted to text. The biggest bottleneck is it still requires complex multi-step pipelines, relying on splicing multiple models.

OCR 2.0 Era : Based on multimodal large models, able to directly convert any picture into structured text. The biggest bottleneck is currently slow reasoning speed, expensive computing power, and high cost.

Every transition, is not simply replacing old technology with new technology, but redefining the problem itself:

OCR 0.5 to 1.0: From how to cut characters open one by one to recognize to how to let the machine learn to recognize entire lines of characters by itself

OCR 1.0 to 1.5: From how to let the machine recognize more accurately to how to let the machine bring knowledge reserves to recognize characters

OCR 1.5 to 2.0: From how to recognize text to how to understand the entire document

Unsolved Problems of OCR 2.0

Of course, OCR 2.0 is not the end point, and the challenges it faces are equally huge:

1. Resolution dilemma: The picture after scanning a page of A4 document is very large, letting AI look at such a large picture requires consuming huge computing power — just like letting a person use a magnifying glass to pixel-by-pixel examine an entire wall
2. Hallucination problem: Large models sometimes brain-supplement text that does not exist in the picture at all — this is unacceptable in scenarios requiring extremely high accuracy such as finance, law, medical
3. Speed problem: Large models write out results character by character, facing a document of several thousand words, the speed is far inferior to traditional schemes
4. Controllability: Traditional schemes are like manual transmission cars, every link can be precisely adjusted; large models are more like autonomous driving systems, the whole is very strong, but when problems occur it is difficult to locate and fine-tune

So within the next 2–3 years, OCR 2.0 and OCR 1.5 will coexist for a long time. Scenarios requiring extremely high precision and extremely fast speed (such as financial receipts, ID cards, license plate recognition), traditional pipeline schemes are still the better choice. And facing scenarios of complex document understanding, multi-language, flexible typesetting output, large model schemes will gradually become mainstream.

Ultimately, the boundaries of OCR are melting. When a model can both recognize text, understand tables, and also parse formulas and charts, the word OCR itself might already not be enough to use.

From OCR to PDF Document Parsing: The True Battlefield

If OCR is recognizing text, then document parsing is understanding documents. And among all document formats, PDF is the hardest one.

Why are PDFs the toughest nut to crack in document parsing?

The design philosophy of PDF is what you see is what you get — it faithfully preserved the visual appearance of the document, but thoroughly discarded the semantic structure. What does the inside of a PDF file look like?

Text is not stored according to human reading sequence, but arranged according to rendering sequence. A seemingly continuous paragraph, at the bottom level might be dozens of scattered text blocks

PDFs lack structural table tags. Those visually neat rows and columns are actually nothing more than a collection of precisely positioned text and lines.

Formulas do not even need to be mentioned. Formulas rendered out by LaTeX, turn into a pile of special characters with precise coordinates in PDF

Pictures might be embedded at any position on the page, mixed and typeset with text

Multi-column typesetting, footnotes, headers and footers, cross-page tables — every single one is an independent parsing puzzle

MinerU: The synthesizer of traditional pipeline

In the PDF parsing field of 2024–2025, MinerU (Shanghai Artificial Intelligence Laboratory, open-sourced in 2024) is the representative work of the traditional route. Its line of thought is like a team with clear division of labor:

One model is specifically responsible for layout analysis — recognizing where text is, where pictures are, where tables are

One model is specifically responsible for text recognition

One model is specifically responsible for table recognition

One model is specifically responsible for formula recognition

Finally, a set of rules aggregates and integrates all the results, generating a structured output.

MinerU performs excellently on standard documents such as academic papers, technical reports. But its essence is still a pipeline of multiple experts collaborating — each expert only manages their own piece, spliced together through complex rules.

This means:

1. If any expert has a problem, the result of the entire pipeline will become worse — just like someone on the team dropping the chain, the overall output will drop
2. When encountering non-standard documents (engineering drawings, scanned copies, handwritten notes), each model easily becomes unaccustomed to the climate, and the effect sharply deteriorates
3. High maintenance costs — each model has to be trained independently, upgraded independently, deployed independently

Pure Vision Route: One model solves all problems

In 2025, a radical idea began to gain true traction in the industry: why don’t we directly treat PDF pages as images, let a visual language model look at this page, and then directly output structured text?

This is the pure vision route (Pure Vision Approach).

PDF turns into Page rendered as image turns into VLM (Visual Language Model) turns into Structured Markdown/JSON

The entire process is compressed into one step. No need for layout analysis, no need for regional classification, no need for independent text recognizer, table recognizer, formula recognizer. A sufficiently powerful visual language model, directly looks at the page image, and outputs any formatted text you want.

This sounds too simplified, but the practice of 2025–2026 proved that, for more and more document types, the pure vision route has already reached or even exceeded the quality of traditional pipelines, while substantially lowering system complexity.

It is worth noting that, even MinerU itself is also embracing this trend. In the latest version, MinerU’s API already supports the model_version: vlm parameter — this means its backend is migrating from traditional multi-model pipeline towards end-to-end schemes based on visual language models.

Knowhere: Document memory infrastructure built for AI Agent

As a personal experiencer of this era, we (Knowhere team) in May 2026 comprehensively open-sourced the Knowhere core technology stack (adopting Apache 2.0 protocol). We always believe that, the memory layer of AI Agent should not be a closed black box, but ought to serve as open-source infrastructure shared by the entire AI developer ecosystem.

Full-stack open source and privatized deployment

Currently, the Knowhere open-source project has already been completely published on GitHub, covering the full set of technology stack from data access, hierarchical parsing to Agent retrieval:

Core framework (Knowhere Core): Ontos-AI/knowhere https://github.com/Ontos-AI/knowhere — — Provides complete document parsing routing, tree-shaped hierarchical reconstruction, multimodal alignment as well as Cross-document Graph memory construction algorithms.

One-click privatized deployment (Knowhere Self-Hosted): Ontos-AI/knowhere-self-hosted https://github.com/Ontos-AI/knowhere-self-hosted — — Supports developers to quickly pull up the entire set of end-to-end services via Docker locally or in enterprise intranets with one click, ensuring sensitive data does not exit the intranet.

Visualized console (Knowhere Dashboard) : Ontos-AI/knowhere-dashboard https://github.com/Ontos-AI/knowhere-dashboard — — Convenient for developers to manage document knowledge bases, monitor parsing tasks and perform visualized retrieval debugging.

Through thorough open-sourcing, Knowhere endowed enterprises and developers with the ability to completely control their own document memory data, bidding farewell to the commercial binding and privacy concerns of cloud service providers.

Who are we? What are we doing?

Knowhere is the memory layer connecting dirty documents (complex, unstructured real-world documents) and AI Agent.

Many developers handle documents very roughly when constructing AI applications — just like randomly tearing a book into page-by-page fragments, and then letting AI flip through the fragment pile via keywords to find answers. This way will thoroughly shatter the original logical structure of the document, leading to AI forever getting information fragments taken out of context.

We do not think this is the correct path leading towards document intelligence. Just as we say on the homepage: we are not forging the next MinerU parser, we are constructing document memory infrastructure that can be efficiently consumed by Agent.

Our work is divided into two core steps: the first step, parsing documents into memories that AI can understand; the second step, letting AI intelligently retrieve these memories like a researcher.

How did we realize it?

First step: Parse and Memory Construction (Parse and Build Memory)

We established an intelligent triage desk — no matter whether what you throw over is PDF, Word, picture or Markdown, the system will automatically allocate it to the most suitable parser to process.

But this is merely the foundation. Our most core technology is the original tree-shaped algorithm. Making an analogy: traditional schemes tear a book into fragments, while we build a complete directory tree for this book — retaining the hierarchical relationships of chapters, sections, paragraphs, letting every block of content know which chapter, which section it belongs to. At the same time, the system will automatically recognize pictures and tables in the document, use AI to generate summaries for them, and ultimately turn the entire document into a knowledge map that AI can consult at any time.

Second step: Agent-style retrieval (Agentic Retrieval)

This is our biggest difference from traditional schemes. Under Knowhere’s architecture, AI is not blindly flipping through in the fragment pile, but methodically reading documents like a human researcher consulting materials:

1. Discovery: The system leverages keywords and semantic understanding to quickly locate the most relevant documents and sections — much like flipping through a table of contents to find the right chapter.
2. Navigation: Along the directory tree and knowledge relationship network of the document, going deep layer by layer to the most core paragraphs — just like clicking in level by level along the directory, finding the most crucial page.
3. Citation: Every time answering can accurately point out which document’s which chapter which section this information comes from, guaranteeing the answer is well-documented and traceable.

Why can we do better?

AI constructed above the Knowhere memory layer, has displayed strength far exceeding traditional schemes in real business scenarios:

1. More accurate: The accuracy rate of the first answer improved by 36%, and the ability to find correct information improved by 11%.
2. Improved Reliability: Through feedback-driven optimization, business accuracy reaches 79%, whereas directly feeding raw documents to large language models (LLMs) frequently hits a performance ceiling of 53%.
3. More money-saving: Because AI can accurately navigate in the knowledge map, avoiding reading massive irrelevant content, computing consumption and waiting time are substantially reduced.

Additionally, Knowhere is not picky on models — no matter if it is DeepSeek, Qwen-VL, or OpenAI, Zhipu, etc., they can all seamlessly connect.

Landing PageMemory V2: First creating granularity adaptive routing

In order to let Agent more perfectly balance the efficiency of complete understanding and local retrieval, Knowhere is currently comprehensively landing the PageMemory V2 scheme. The core of this scheme lies in introducing the DOC_PROFILE granularity router, acting as a brain to adaptively output the most fitting memory granularity judgment (Verdict) according to the document’s classification (such as scientific papers, financial reports, etc.), TOC structure as well as complex page proportion:

1. whole_doc whole memory: For simple documents with few pages, no TOC conduct whole-article full-amount memory, not doing dismantling, avoiding destroying Agent’s understanding towards macroscopic context.
2. page page-by-page memory: For complex documents conduct page-by-page persistence and deep memory, combining full-page image, image summary with original text to construct association.
3. shard_page sharded memory: Targeting super large documents, first shard and then conduct page-by-page memory, reasonably evenly sharing computing power.

Through this first created granularity adaptive mechanism, cooperating with pluggable Profiler interface (open source defaults to supporting rules and general VLM, commercial version supports proprietary fine-tuned models with higher precision and cost-effectiveness ratio), Knowhere realized key breakthroughs in the consumption efficiency and cost control of multimodal document memory.

Standing here today in 2026, we firmly believe that unstructured data should no longer be an obstacle to AI systems; instead, it should become an endlessly regenerating memory for them. We, at Knowhere, are actively pioneering this future, moving from merely recognizing text to constructing navigable document memories.

Looking back from 2026, nearly a century of OCR history — from Gustav Tauschek’s mechanical template-matching device in 1929 to modern VLMs directly analyzing documents to generate structured knowledge — points to a simple yet profound conclusion:

We never needed to merely recognize text; what we have always needed is to understand documents.

As multimodal models drive the convergence of OCR and document intelligence, an evolutionary path extending from basic document parsing to Agent-consumable knowledge infrastructure is becoming increasingly clear. This path points toward a grand vision: in the era of AI Agents, unstructured data is no more an obstacle to AI systems, but the very foundation of their memory.

Perhaps this is the true destination of OCR’s century-long evolution — not simply teaching machines to read text, but enabling them to truly retain and comprehend the knowledge within documents.

---

Originally published on Medium.