# AMD Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM
## Overview
AMD has released **Instella-MoE-16B-A3B**, a fully open Mixture-of-Experts (MoE) language model that represents a significant milestone in open-source AI development. The model was trained from scratch on AMD’s Instinct MI300X and MI325X GPUs, featuring 16 billion total parameters with only 2.8 billion active parameters per token. This technical achievement demonstrates AMD’s commitment to open AI research and provides the community with a powerful foundation for experimentation and research.
## Technical Architecture
### Model Specifications
Instella-MoE-16B-A3B employs a sophisticated architecture that balances efficiency with performance:
– **Total Parameters**: 16 billion
– **Active Parameters**: 2.8 billion per token
– **Layers**: 27 decoder layers
– **Hidden Size**: 2048
– **Attention Heads**: 16 heads
– **Vocabulary**: 128,896 tokens
– **MoE Configuration**: 2 shared experts + 6 routed experts from 64 total experts
### Key Innovations
The model incorporates two groundbreaking structural choices:
1. **Gated Multi-head Latent Attention (Gated MLA)**: This innovation adds a lightweight learned output gate to Multi-head Latent Attention. A dedicated linear projection derives an input-conditioned gate that is applied multiplicatively before the output projection, effectively muting weak attention heads and improving efficiency.
2. **FarSkip-Collective Connectivity**: This approach passes outdated and partial activations into the MoE and attention layers, enabling overlapping of expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to 39.2% reduction in time to first token when serving with expert parallelism.
## Training Methodology
### Data and Training Pipeline
The training process was comprehensive and multi-stage:
– **Pre-training**: 7.1 trillion tokens from diverse open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360
– **Mid-training**: Dolma3 Dolmino 100B across three data variants, merged by weight averaging
– **Long-context Stage**: Extended context window from 4K to 64K using YaRM, increased RoPE theta, and document masking
– **Post-training**: Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO), and reinforcement learning (RL) using the Miles framework
### Research Infrastructure
The model was trained entirely on AMD’s hardware ecosystem:
– AMD Instinct MI300X and MI325X GPUs
– ROCm software platform
– Primus framework
– Miles framework for RL training
## Performance Results
### Benchmark Performance
The base checkpoint demonstrates competitive performance among fully open models:
– **Average Score**: 76.7 (leading among fully open models)
– **Comparison**: Ahead of Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9)
– **Gap**: Trails Qwen3.5-4B-Base (79.5)
### Specialized Benchmarks
– **WinoGrande**: 86.5 (leading performance)
– **HumanEval+**: 65.7
– **Long-context HELMET**: 41.5
– **Long-context RULER**: 79.4
– **IFEval**: Improved from 77.08 (SFT) to 83.70 (post-training)
### Training Efficiency
The architectural innovations delivered significant efficiency gains:
– **Pre-training Speedup**: 12.7% improvement
– **Time to First Token**: Up to 39.2% reduction with expert parallelism
## Deployment Considerations
### Licensing and Usage
**Important Distinction**:
– **Weights**: Subject to ResearchRAIL license (academic and research purposes only)
– **Training Code**: MIT licensed (more reusable for commercial applications)
### Target Users
**Ideal For**:
– AI research labs
– University research groups
– Enterprise R&D teams with data-center GPU capacity
– Semiconductor and cloud infrastructure companies
– AI tooling vendors
**Not Suitable For**:
– Lean startups seeking hosted commercial endpoints
– Commercial deployment without proper licensing
### Technical Requirements
– **Memory**: 16GB high-memory accelerator (BF16 weights require ~32GB)
– **Serving**: Compatible with SGLang inference code
– **Applications**: MoE recipe reproduction, expert-parallel serving studies, 64K long-context evaluation, RL post-training experiments
## Key Takeaways
1. **Scale with Efficiency**: 16B total parameters with only 2.8B active per token through expert routing
2. **Performance Leadership**: Strong results among fully open models, particularly in specialized benchmarks
3. **Innovation Focus**: Gated MLA and FarSkip-Collective demonstrate AMD’s architectural innovation
4. **Open Research Commitment**: Comprehensive release of weights, data, configs, and code
5. **Hardware Integration**: End-to-end optimization from architecture to training infrastructure
6. **Research-First**: Licensed for academic use while training code remains commercially reusable
## Conclusion
AMD’s Instella-MoE-16B-A3B represents a significant contribution to the open-source AI ecosystem. By combining a sophisticated MoE architecture with innovative connectivity solutions, AMD has delivered a model that achieves competitive performance while maintaining complete openness for research purposes. The 12.7% training speedup and 39.2% reduction in time to first token demonstrate that architectural innovation can rival scaling approaches.
While the ResearchRAIL license limits commercial deployment, the MIT-licensed training code provides researchers and developers with a valuable foundation for experimentation. The model’s strong performance on long-context benchmarks and specialized tasks makes it particularly valuable for research into efficient serving mechanisms and MoE optimization.
As AMD continues to advance its AI hardware and software ecosystem, models like Instella-MoE-16B-A3B demonstrate the company’s commitment to an open, competitive AI landscape where innovation in architecture and efficiency can thrive alongside scale.
—
**Sources**: ROCm blog · Hugging Face collection · GitHub · July 24, 2026
**Author**: Asif Razzaq, CEO of Marktechpost Media Inc.
*Note: This article incorporates interactive visualizations and technical demonstrations originally published in the source ROCm blog post for enhanced reader understanding.*
