NeoMME shrank visual document embeddings 255x while retaining over 95% of retrieval quality
Visual document search has an awkward economics problem. Embedding a rendered page preserves its layout, tables, charts, type hierarchy, and images. It also creates a much larger index than ordinary text retrieval.
NeoMME, an open family of 260 million and 800 million parameter multimodal encoders, makes that tradeoff unusually concrete. Its authors measured an average of about 1.5 MB per page for a late-interaction index on ViDoRe v3. After hierarchical token pooling and asymmetric quantization, the figure fell to 6 kB per page, 255x smaller, while retaining more than 95% of baseline nDCG@10.
Compression is not free here. The useful result is a quality-versus-storage curve that an engineering team can test against its own budget.
Why one page becomes thousands of vectors
Conventional text retrieval begins by extracting and chunking text. That is convenient, but a table can be flattened, a caption separated from its chart, and page structure discarded. Visual retrieval indexes screenshots of document pages instead.
NeoMME-Retriever produces two representations in one forward pass. The dense head averages hidden states into a single normalized vector. The late-interaction head projects every text token or image patch into a 128-dimensional normalized vector, allowing individual query tokens to match different regions of a page.
The detail has a price. A 2048 by 2048 page produces about 4,200 vectors and roughly 2.1 MB of float32 data. The average across the authors' ViDoRe v3 evaluation was about 1.5 MB per document. By simple arithmetic, one million pages at that average would consume roughly 1.5 TB before database overhead, replicas, or metadata. That is not a measured deployment bill. It does show why index compression can matter more than shaving a few parameters from the model.
Two ways to cut the index
Hierarchical token pooling clusters similar document vectors on the same page and replaces each cluster with its mean. The index stores fewer vectors, and late-interaction scoring has fewer items to process.
Asymmetric quantization reduces stored document embeddings to int8 or binary values. Query embeddings can remain at higher precision because they are generated only when a search runs. Memory is cut on the large, persistent side of the system while the small, temporary side keeps more detail.
The authors published two useful operating points. With a pooling factor of 10 and int8 queries and documents, average storage fell from about 1.5 MB to 39 kB per page, a 39x reduction, while retaining more than 99% of baseline nDCG@10. The aggressive setup used a pooling factor of 8, int8 queries, and binary documents. It used 6 kB per page, 255x less storage, and retained more than 95%.
Those percentages describe retained quality, not absolute benchmark scores. They also come from the NeoMME authors' own ViDoRe v3 experiments. A team should reproduce the curve on its own pages and questions before making 6 kB the default.
What the smaller model changes
NeoMME sends text tokens and raw image patches through one bidirectional Transformer. It does not connect a separately pretrained vision tower to a causal language model. Images are divided into non-overlapping 32 by 32 patches, projected by a small MLP, and passed through the same encoder path as text.
Both model sizes support a 16,384-token context, which the authors say is enough for up to two standard 3840 by 2160 4K UHD images. Most layers use symmetric sliding-window attention; every sixth layer and the final layer use global attention. Training started from scratch with a masked discrete-diffusion text-denoising objective and mixed multilingual text, code, mathematics, natural images, and document images. Each model processed about 524 billion packed input tokens, including 290 billion text-only tokens.
On ViDoRe v3, the authors report 0.523 nDCG@10 for NeoMME-260M and 0.556 for NeoMME-800M. The 260M score led the evaluated models strictly below 800M parameters. It did not lead the broader table: Vultron Flash, listed at 850M parameters, scored 0.565. In a matched 2048 by 2048 test on one NVIDIA L40S, the paper reports about 51 encoded pages per second for NeoMME-260M versus 26 for ColModernVBERT.
These results have not been independently reproduced. ViDoRe v3 covers eight domains: HR, English finance, industry, pharma, computer science, energy, physics, and French finance. Handwriting, noisy scans, forms, less represented languages, or a different document mix could change the ranking.
A test that can survive contact with production
Begin with the retrieval target. Assemble real questions and judge which pages answer them. Include examples where layout carries meaning and examples where extracted text should be enough.
Compare three configurations:
- Dense retrieval only, with an approximate nearest-neighbor index.
- Dense retrieval followed by late-interaction reranking over a shortlist.
- Late-interaction retrieval at several pooling and quantization settings.
Track retrieval quality, index bytes per page, indexing throughput, query latency, and peak memory. Record document resolution and corpus size. A score or pages-per-second figure without those details does not travel well between systems.
The second configuration is often the practical middle ground. NeoMME-Retriever emits dense and late-interaction representations together, so the dense vector can select candidates before the detailed score reranks them. It follows the same discipline as managing an agent context budget: keep costly detail where it can change the answer, not everywhere by default.
Reproducibility matters as much as the final score. BLOGish recently looked at how AgentArena replays coding-agent runs locally. A useful NeoMME evaluation should likewise preserve page images, queries, relevance judgments, resolution, compression settings, and the exact checkpoint.
Limits to account for
The 260M and 800M checkpoints are available under Apache 2.0, and NeoMME support is in Transformers. For document search, use a retrieval checkpoint. The backbone model card notes that the base encoder still needs fine-tuning or a task-specific head for downstream work.
The model cards also say comprehensive safety, bias, and privacy evaluation has not been completed. Visual RAG does not remove document-governance obligations. Page images can expose personal data, signatures, and details that an OCR pipeline may previously have filtered. Access controls and deletion need to cover source images, embeddings, caches, and generated answers.
NeoMME does not prove that visual retrieval now fits every archive. It supplies open checkpoints and measured knobs for choosing where storage and retrieval quality should meet. The useful next step is to draw that curve with your own documents, not copy the authors' endpoint.
