## Overview
Moonshot AI has unveiled **Kimi K3**, billed as the world’s first open 3 trillion-parameter (3T-class) model. Built on a sparse Mixture-of-Experts (MoE) architecture, Kimi K3 introduces key innovations such as Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), alongside Stable LatentMoE routing. The model is designed for long-horizon coding, knowledge work, and advanced reasoning, targeting use cases that demand extended context and efficient scaling.
—
## What is Kimi K3?
Kimi K3 represents a major step forward in open large language models. As a sparse MoE, it leverages two architectural advances:
– **Kimi Delta Attention (KDA):** A hybrid linear attention mechanism that significantly speeds up decoding in long contexts.
– **Attention Residuals (AttnRes):** A mechanism that selectively retrieves representations across model depth, improving training efficiency.
With 2.8 trillion parameters and a native 1-million-token context window, Kimi K3 is positioned as a strong open alternative for complex coding and reasoning tasks. While it does not yet match the top proprietary models in absolute performance, it sets new benchmarks among open models in scaling and efficiency.
—
## The Architecture Underneath
### Kimi Delta Attention (KDA)
KDA is a hybrid linear attention mechanism that decouples query-key interactions from softmax computation, enabling faster and more memory-efficient decoding. Moonshot reports up to **6.3× faster decoding** at million-token contexts compared to traditional attention.
### Attention Residuals (AttnRes)
AttnRes changes how information flows across model depth by selectively retrieving representations rather than accumulating them uniformly. This design delivers approximately **25% higher training efficiency** with less than 2% additional computational cost.
### Stable LatentMoE and Sparsity
K3 uses **Stable LatentMoE**, activating around **16 of 896 experts per token**. Key techniques include:
– **Quantile Balancing:** Routes tokens based on router-score quantiles, eliminating heuristic balancing.
– **Per-Head Muon:** Optimizes attention heads independently for better representation learning.
– **SiTU and Gated MLA:** Improve activation control and attention selectivity.
These techniques, combined with refined training and data pipelines, yield roughly **2.5× better scaling efficiency** than Kimi K2.
—
## Serving and Efficiency
Kimi K3 is trained with **quantization-aware training** from the supervised fine-tuning stage onward, using **MXFP4 weights** and **MXFP8 activations** to ensure broad hardware compatibility. Moonshot recommends supernode configurations with **64 or more accelerators** and has contributed a **prefix caching implementation** to the **vLLM** project to address KDA-specific caching challenges.
—
## Interactive Capabilities
The model supports a wide range of benchmarks and real-world tasks, including coding, agentic workflows, and vision-language understanding. While detailed benchmark results are available from Moonshot’s own evaluations, the architecture emphasizes efficient inference, long-context performance, and open accessibility.
—
## FAQs
### What is Kimi K3?
Kimi K3 is an open 3 trillion-parameter model from Moonshot AI, designed for long-context and complex reasoning tasks. It uses a sparse Mixture-of-Experts architecture with novel attention mechanisms to deliver strong performance and efficiency.
### What makes KDA and AttnRes different?
KDA is a hybrid linear attention mechanism that speeds up decoding in long contexts, while AttnRes improves how information flows across model depth, enhancing training efficiency without heavy overhead.
### How efficient is MoE routing in K3?
K3 activates only 16 of 896 experts per token using Stable LatentMoE. Quantile Balancing ensures even load distribution, reducing imbalance and improving routing efficiency.
### What hardware is recommended for serving K3?
Moonshot recommends supernode setups with **64 or more accelerators**. The model uses MXFP4/MXFP8 precision to maximize compatibility across different hardware platforms.
### How does K3 compare to proprietary models?
While proprietary models like Claude Fable 5 and GPT 5.6 Sol still lead in raw performance, K3 sets new standards among open models in scaling, efficiency, and long-context capability.
—
## Conclusion
Moonshot AI’s Kimi K3 represents a significant milestone for open large language models. By combining architectural innovations like KDA and AttnRes with efficient MoE routing and quantization-aware training, K3 delivers strong performance and scalability for long-horizon coding and knowledge-intensive tasks. As an open model, it lowers barriers to advanced AI capabilities and pushes the frontier of what open architectures can achieve.
