# NeoMME: A New Class of Compact Bidirectional Encoders for Multimodal Retrieval
## The Problem with Repurposed Models
Most visual document retrieval systems deployed in production today are built by taking a generative vision-language model and stripping it down to serve as an encoder. While this approach has been dominant — seen in models like ColPali and its successors — it comes with a fundamental inefficiency. These repurposed architectures still carry a separately pretrained vision tower and a causal language decoder, neither of which are ever used during the actual retrieval task. The decoder never generates a single token, and the vision tower was never designed for this purpose. The result is significant parameter bloat and wasted compute for a task that fundamentally only needs high-quality representations.
A team at H Company has addressed this inefficiency head-on with **NeoMME**, a family of compact bidirectional encoders designed from the ground up for document retrieval. By eliminating both the pretrained vision tower and the causal decoder entirely, NeoMME achieves impressive retrieval performance at a fraction of the size of existing approaches.
—
## Architecture: One Tower, Two Modalities
The core innovation of NeoMME is architectural simplicity. A single Transformer processes both multilingual text tokens and raw image patches through the same layers, starting from completely random initialization. There is no separate vision encoder, no text encoder, and no generative head.
**Text Processing**
Text tokens enter through an ALBERT-style factorized embedding layer, where a 256-dimensional lookup table projects inputs up to the full model width. This design keeps the text pathway lightweight while maintaining expressiveness.
**Image Processing**
Images are divided into non-overlapping 32×32 pixel patches, and each patch is projected into the model’s embedding space by a 2-layer MLP trained from scratch. There is no patch-merging module and no pretrained image encoder like SigLIP2 — raw pixels go straight into the shared Transformer.
**Shared Backbone**
Both model variants — 260 million parameters and 800 million parameters — share the same architectural design. The Transformer stack spans up to 16,384 tokens, which is sufficient to accommodate two standard 3,840×2,160 4K UHD images after patching. Key design choices include:
– Symmetric sliding-window attention across most layers, with global attention on every sixth layer and the final layer
– Grouped-query attention for efficiency
– Query-key normalization and gated attention mechanisms
– 2D rotary position embeddings for spatial awareness
– Squared-ReLU activation in the feed-forward layers
The tokenizer is a whitespace-unconstrained BPE vocabulary with 131,072 entries, trained from scratch. Across 14 target languages in the FLORES-200 devtest benchmark, it produces 44.4% fewer tokens than ModernBERT, improving efficiency for multilingual workloads.
Exact parameter counts are **262,937,906** and **793,715,032** for the two variants respectively.
—
## Training: Masked Diffusion as a Foundation
NeoMME is pretrained using a discrete masked diffusion objective applied to text, optionally conditioned on visible image patches. This training paradigm is notable for its simplicity and effectiveness.
For text-only segments, the corruption rate is drawn uniformly from 0 to 1. For multimodal segments, the corruption rate is drawn from 0.30 to 1.0. This design choice is deliberate — by never allowing multimodal examples to have zero masking, the model is forced to actually read and understand the page content rather than relying on language-only cues. It removes the “language shortcut” that can plague multimodal models during pretraining.
A cross-modal ablation probe confirms the effectiveness of this approach. At 90% masking, visible page patches improve masked-token accuracy by **38.4 points** for the 260M model and **40.5 points** for the 800M model. Each training run processes approximately 524 billion packed input tokens, of which roughly 290 billion are text-only, distributed across 16 and 32 H100 accelerators respectively.
—
## Retrieval Performance
The retrieval fine-tuned variant, **NeoMME-Retriever**, adds two jointly trained heads on the shared backbone. A mean-pooled dense head uses Matryoshka widths for flexible dimensionality, while a late-interaction head projects every token and patch embedding to 128 dimensions. Both heads produce outputs from a single forward pass, making deployment straightforward.
### ViDoRe Benchmark Results
On the ViDoRe v3 benchmark, the 260M model achieves **0.523 nDCG@10**, while the 800M model reaches **0.556 nDCG@10**. The 260M result is within 0.002 of ColQwen2.5-v0.2, a 3.75 billion parameter model, and surpasses the best other sub-300M model by 26.1 points. The 800M model lands 0.9 points behind the similarly sized Vultron Retriever Flash.
On earlier ViDoRe versions, the models also perform competitively:
– **ViDoRe v1**: 0.860 / 0.522 nDCG@5 (260M / 800M)
– **ViDoRe v2**: 0.874 / 0.559 nDCG@5 (260M / 800M)
### Text-Only Retrieval
On the BEIR-15 text retrieval benchmark, late interaction reaches 0.4881 and 0.5126 for the two variants respectively. This trails the 0.5722 achieved by LateOn at 149M parameters. The authors attribute the gap partly to supervision scale — NeoMME trained on roughly 430K text query examples compared to roughly 660 million contrastive examples for mLateOn. This remains an area where further pretraining data or scaled supervision could yield meaningful improvements.
—
## Storage and Throughput
Late-interaction indexes are inherently storage-intensive. A single 2048×2048 page produces 4,162 vectors, consuming approximately 1.5 MB per document in float32 format. NeoMME addresses this through two complementary compression strategies:
1. **Hierarchical token pooling** with a factor of 10, combined with int8 quantization for both queries and documents, reduces storage to **39.0 kB per page** — a **39.4× compression** that retains **99.16%** of baseline nDCG@10.
2. **Pooling at factor 8** with int8 queries and binary documents further compresses to **6.0 kB per page** — a remarkable **255.5× reduction** while retaining **95.19%** of baseline quality.
Indexing throughput is equally impressive. At matched 2048×2048 input resolution on a single NVIDIA L40S, NeoMME-260M encodes **51.3 pages per second**, compared to ColModernVBERT’s 26.0 pages per second — nearly a 2× improvement. Encoding a query on a CPU-only host takes just **78.3 ms**.
—
## Deployment and Licensing
Every NeoMME checkpoint is released under the **Apache 2.0 license**, ensuring unrestricted use in both research and commercial settings. Day-zero support is available in Hugging Face Transformers, and all model collections are accessible for immediate download. This combination of permissive licensing, framework integration, and competitive performance makes NeoMME a practical choice for teams looking to build or upgrade multimodal retrieval pipelines.
—
## FAQ
### What makes NeoMME different from ColPali and similar models?
NeoMME eliminates the pretrained vision tower and causal decoder that ColPali-style models carry forward. Instead, it uses a single bidirectional Transformer that processes both text tokens and raw image patches through shared layers, trained entirely from scratch. This removes unused parameters and unused compute, resulting in a leaner and more efficient architecture.
### How small is the smallest NeoMME model?
The smallest variant has 260 million parameters — roughly 14.4 times smaller than ColQwen2.5-v0.2, which it matches on ViDoRe v3 performance.
### What is the maximum context length?
Both variants support a 16,384-token context window, enough to encode two standard 3,840×2,160 4K UHD images after patching.
### How is the model trained?
NeoMME uses discrete masked diffusion pretraining over text, optionally conditioned on visible image patches. Multimodal examples use a corruption rate between 30% and 100%, which prevents the model from relying on language-only signals.
### What compression options are available for the index?
Two configurations are recommended: hierarchical token pooling with int8 quantization (39.0 kB per page, 99.16% quality retention) and pooling with binary documents (6.0 kB per page, 95.19% quality retention).
### Is NeoMME available for commercial use?
Yes. All checkpoints are released under the Apache 2.0 license, which permits commercial use, modification, and distribution.
### How does NeoMME handle multilingual documents?
The model uses a whitespace-unconstrained BPE tokenizer with a 131,072-entry vocabulary trained from scratch. It was evaluated across 14 target languages in FLORES-200 and produces 44.4% fewer tokens than ModernBERT on that benchmark.
### What are the known weaknesses?
Text-only retrieval and transfer to natural images (as opposed to document layouts) remain areas where NeoMME underperforms relative to larger specialized models. The text retrieval gap on BEIR-15 is partly attributed to the smaller scale of text-only supervision compared to contrastive training pipelines.
—
## Conclusion
NeoMME represents a compelling case for purpose-built multimodal retrieval architectures. By designing a single bidirectional Transformer that handles both text and image patches — without carrying the baggage of a pretrained vision tower or a generative decoder — H Company has demonstrated that compact models can rival or exceed much larger alternatives on document retrieval benchmarks.
The practical implications are significant. With sub-second query encoding on CPU hardware, over 50 pages indexed per second on a single GPU, and index sizes that can be compressed by over two orders of magnitude, NeoMME is built for real-world deployment from day one. The Apache 2.0 license and Hugging Face integration further lower the barrier to adoption.
While challenges remain — particularly in text-only retrieval and cross-domain generalization — the NeoMME family establishes a clear blueprint for efficient multimodal encoding that could influence the next generation of retrieval systems. As the field moves away from repurposed generative models and toward architectures designed for their actual task, the efficiency gains demonstrated here point toward a future where multimodal retrieval is faster, smaller, and more accessible.
Thank you for reading
