# Liquid AI’s LFM2.5 Open-Weight Encoders: What You Need to Know
Liquid AI has released two open-weight bidirectional encoders, **LFM2.5-Encoder-230M** and **LFM2.5-Encoder-350M**, designed to bring efficient, long-context understanding to a wide range of applications. Both models are built on the LFM2 hybrid backbone and carry an 8,192-token context window, making them suitable for tasks that require processing long documents, contracts, or records in a single pass.
## From Decoder to Encoder: The Conversion Process
The encoders are not trained from scratch but are converted from existing LFM2.5 decoder backbones through three key changes:
1. **Bidirectional Attention Mask** – The causal mask is replaced so every token attends to both left and right context.
2. **Non-Causal Convolutions** – Short convolutions are made symmetric with center padding, allowing each token to mix information from neighbors on both sides.
3. **Higher Mask Rate Training** – A 30% masked language modeling objective is used, which is denser than BERT’s 15%, following evidence that this improves performance at scale.
Training happens in two stages:
– **Stage 1:** General language competence is developed using short-context MLM on a large web corpus at 1,024 tokens.
– **Stage 2:** Context is extended to 8,192 tokens on a full data mix, strengthening factual, legal, and multilingual capabilities.
The models share a hidden size of 1024, a vocabulary of 65,536 tokens, and support 15 languages under the LFM Open License v1.0.
## Benchmark Results and Performance
Liquid AI evaluated 14 models across 17 tasks from GLUE, SuperGLUE, and multilingual benchmarks. The results highlight strong efficiency and competitive accuracy:
– **LFM2.5-Encoder-350M** ranks **4th** with a 17-task mean score of **81.02**, outperforming models like ModernBERT-base and XLM-R base, while being significantly smaller than the top performers.
– **LFM2.5-Encoder-230M** ranks **6th** with a score of **79.29**, beating larger models such as ModernBERT-base and all EuroBERT variants tested.
– Both encoders outperform Liquid AI’s own retriever models, reinforcing the value of using a dedicated encoder rather than adapting retrievers for general-purpose tasks.
In terms of speed, the 230M encoder delivers a **~3.7× CPU speed-up** compared to ModernBERT-base when processing 8,192 tokens, completing a forward pass in about **28 seconds** versus over **90 seconds** for the base model.
## Use Cases and Deployment Environments
The encoders are particularly well-suited for environments where GPU access is limited or latency must be minimized:
– **Edge and embedded devices** such as automotive compute units or industrial controllers.
– **On-premise and regulated settings** in finance, healthcare, and legal domains, where documents are long and sensitive data cannot leave local infrastructure.
– **Cost-sensitive pipelines** where a lightweight encoder acts as an efficient first-pass filter before invoking larger models.
Thanks to their 8K context, these encoders can handle a full contract or patient record in a single forward pass, making them practical for real-world document understanding tasks.
## Getting Started
Both encoders are available through Hugging Face Transformers and can be loaded with `trust_remote_code=True`. They are distributed as masked language models, meaning they produce general-purpose representations and must be fine-tuned for downstream tasks such as:
– Text classification
– Named entity recognition
– Semantic similarity
– Retrieval and reranking
– NLI and question answering
Fine-tuning follows standard BERT-style procedures, with recommended learning rates between 1e-5 and 5e-5, warmup ratios around 0.1, and weight decay of 0.1.
## Key Takeaways
– Liquid AI has released two open-weight bidirectional encoders with long context support.
– They convert existing decoders into encoders using bidirectional attention, non-causal convolutions, and higher MLM mask rates.
– The 350M model ranks among the top performers in benchmark evaluations, while the 230M model offers strong efficiency and accuracy trade-offs.
– On CPU, these encoders significantly outperform comparable models at long context lengths.
– They are designed for deployment in edge, on-premise, and cost-sensitive scenarios where long documents must be processed efficiently.
For more details, including interactive benchmarks and demo applications, you can refer to the [Liquid AI blog](https://liquid.ai) and the corresponding [Hugging Face model cards](https://huggingface.co/LiquidAI).
—
## Frequently Asked Questions (FAQ)
### What are LFM2.5-Encoder-230M and LFM2.5-Encoder-350M?
They are open-weight, bidirectional text encoders released by Liquid AI. Both are based on the LFM2 hybrid architecture and support an 8,192-token context window. They are designed for efficient long-document understanding and are available under the LFM Open License v1.0.
### How are these encoders different from standard BERT-style models?
The encoders are converted from LFM2.5 decoders using bidirectional attention, non-causal convolutions, and a denser masking strategy (30% MLM). The LFM2 architecture is tailored for slower cost growth as input length increases, making it more efficient for long-context tasks.
### What performance can I expect from these encoders?
In benchmark evaluations, LFM2.5-Encoder-350M scores 81.02 (4th place among 14 models), while LFM2.5-Encoder-230M scores 79.29 (6th place). Both outperform several larger models and are particularly strong in multilingual and long-context settings.
### What hardware are these models optimized for?
These encoders are designed to run efficiently on CPU, making them suitable for edge devices, on-premise systems, and environments without GPU access. They also support GPU deployment, where they show competitive performance for inputs longer than 2K tokens.
### How can I use these encoders in my project?
You can load them via Hugging Face Transformers using `trust_remote_code=True`. After loading, you can attach task-specific heads for classification, NER, retrieval, or semantic similarity, and fine-tune them on your dataset following standard BERT-style training procedures.
—
## Conclusion
Liquid AI’s LFM2.5 encoders represent a significant step forward in efficient, long-context understanding. By converting existing decoders into bidirectional encoders and optimizing for longer inputs, they deliver competitive accuracy with lower computational cost. Whether deployed on CPU for latency-sensitive environments or GPU for high-throughput scenarios, these models provide a flexible and powerful foundation for a wide range of real-world NLP tasks.
