## Article: Soofi S 30B‑A3B — A New Open Multilingual Foundation Model for German and English
A German research consortium has published the pretraining report for **Soofi S 30B‑A3B**, an open base model designed for German and English. The model was trained end‑to‑end on Deutsche Telekom’s Industrial AI Cloud in Munich, and preview weights are available on Hugging Face. Among fully open base models tested, Soofi S achieves the highest aggregate English and German scores.
—
### What is Soofi S 30B‑A3B?
Soofi S is a **Mixture‑of‑Experts (MoE) hybrid Mamba‑Transformer** foundation model with approximately **31.6B total parameters**, activating around **3.2B parameters per token**. As a base model, it has **no instruction tuning, alignment, or safety tuning**. The consortium is coordinated by the KI Bundesverband, funded by the German Federal Ministry for Economic Affairs and Energy, and includes partners such as Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum.
—
### How the Architecture Works
The model uses a **52‑layer stack**, comprising:
– **23 Mamba‑2 sequence‑mixing layers**
– **23 MoE layers**
– **6 Grouped‑Query Attention (GQA) layers**, which retain the KV cache
Each MoE layer contains **128 routed experts**, selects **6 experts per token**, and includes **2 shared experts**. The architecture follows the **Nemotron 3 Nano reference design**, which the team chose for better deployability on stacks like vLLM, serving efficiency, and scientific control. Other notable design choices include a model dimension of 2688, squared ReLU activation, RMSNorm, and no positional embeddings.
—
### Efficiency Through Sparsity
Thanks to MoE sparsity, only **10.1% of parameters are computed per forward pass** (3.2B out of 31.6B). This reduces compute per token while keeping storage requirements at the full 30B level, since all experts must remain resident in memory.
—
### KV Cache Optimization
Only the **6 GQA layers** maintain a key‑value cache, dramatically reducing memory usage compared to fully attention‑based models. Depending on context length, Soofi S can require **8–9× less KV cache** than a dense 14–24B model while maintaining strong throughput.
—
### Training Curriculum
The pretraining followed a **Warmup–Stable–Decay learning‑rate schedule** across three phases:
1. **Diverse pretraining** (~20T tokens) with balanced English and German data
2. **High‑quality annealing** (~6.6T tokens), emphasizing German and high‑quality web data
3. **Long‑context extension** (~0.1T tokens) at up to 1M tokens sequence length
Overall, the model consumed **approximately 26.68T tokens**.
—
### Benchmark Performance
In the pretraining report, Soofi S 30B‑A3B was evaluated using the **lm‑evaluation‑harness** with consistent prompts and few‑shot settings. Results show strong performance on:
– English and German aggregate benchmarks
– HumanEval and MBPP‑DE for coding
– GSM8K and GPQA‑Diamond for reasoning
– Industry‑specific benchmarks such as GLP‑DE
In many tasks, Soofi S ranks among the top open models, often outperforming larger dense models in efficiency‑critical settings.
—
## FAQ
**Q: Is Soofi S 30B‑A3B an instruction‑tuned model?**
A: No. Soofi S is a base model with no instruction tuning, alignment, or safety tuning.
**Q: How many parameters are actually computed during inference?**
A: Approximately 3.2B parameters are active per token, thanks to MoE sparsity.
**Q: What is the context length supported?**
A: The model supports context lengths of up to **1 million tokens**.
**Q: Where can I access the model weights?**
A: Preview weights are available on **Hugging Face**.
**Q: Which languages is the model trained on?**
A: The model is trained for **German and English**, with significant portions of the data dedicated to both languages.
**Q: How does the KV cache compare to dense models?**
A: Only 6 GQA layers use KV cache, reducing cache size by **8–9×** compared to fully attention‑based dense models at similar scale.
**Q: What is the total pretraining token count?**
A: The curriculum consumed approximately **26.68 trillion tokens**.
—
## Conclusion
Soofi S 30B‑A3B represents a significant step for open multilingual models in German and English. By combining a sparse MoE architecture with the efficiency of Mamba‑2 and GQA, the model achieves strong benchmark performance while reducing compute and memory costs during inference. With open weights and a transparent training report, Soofi S provides a valuable foundation for research, localization, and production deployments in German‑English NLP tasks.
