## Thinking Machines Lab Releases Inkling: A 975B-Parameter Mixture-of-Experts Foundation Model for Customization
Thinking Machines Lab has released **Inkling**, its first model trained from scratch and designed as a base for customization. The model is offered with open weights and supports fine-tuning on Tinker, positioning it as a flexible starting point for downstream applications and research.
### What is Inkling?
Inkling is a **Mixture-of-Experts (MoE) transformer** architecture with **975 billion total parameters** and **41 billion active parameters per forward pass**. It supports a **context window of up to one million tokens** and was pretrained on **45 trillion tokens** spanning text, images, audio, and video. The model takes **text, images, and audio** as input and produces **UTF-8 text** as output.
To provide a more compact alternative, the lab also previewed **Inkling-Small**, a 276B-parameter MoE with 12B active parameters. According to the team, this smaller variant matches or exceeds the performance of its larger counterpart on many benchmarks, with its weights expected to be released after testing concludes.
### Inside the Architecture
Inkling’s architecture is built around a **66-layer decoder-only transformer** with a **sparse MoE feed-forward backbone**. Each layer contains **256 routed experts**, with **6 experts activated per token**, plus **2 shared experts** that are active on every token. A **sigmoid-based router** handles expert selection using an auxiliary-loss-free load-balancing bias, and routed and shared expert scores are normalized jointly before being used to weight their combined outputs. This MoE design is largely inspired by **DeepSeek-V3**.
Key architectural features include:
– **Interleaved attention**: Sliding-window and global attention layers are arranged in a **5:1 ratio**
– **8 KV heads** for attention
– **Relative positional embeddings** instead of RoPE, which the lab reports offers better length extrapolation
– **Short convolutions** applied after key and value projections, as well as on residual branches
– **Encoder-free multimodal input handling**: images are processed as **40×40 pixel patches**, and audio as **dMel spectrograms**, both projected through a lightweight embedding layer before joint processing with text
Training relied on **Muon** for large matrix weights and **Adam** for other parameters, executed on **NVIDIA GB300 NVL72 systems**. Post-training included **SFT on synthetic data**, including data generated by **Kimi K2.5**, followed by **asynchronous RL** scaled to over **30 million rollouts**, which the team says improved performance log-linearly and shaped the model’s control surface.
—
### Interactive Visualization: How Inkling Works
To help users understand how Inkling operates, Thinking Machines Lab embedded a self-contained interactive module. It illustrates:
1. **MoE Routing**
– 256 experts with 6 routed and 2 shared per token
– Visual representation of expert activation and load balancing
2. **Attention Mix**
– 66-layer stack with sliding-window and global attention
– Dynamic token-by-token attention paths
3. **Multimodal Fusion**
– Text, image, and audio streams projected into a shared hidden space
4. **Effort Dial**
– A user-controlled slider that adjusts reasoning effort
– Real-time visualization of token efficiency versus latency
The module is built as a sandboxed iframe and runs directly in the browser without external dependencies.
—
### Frequently Asked Questions (FAQ)
**Q1: What makes Inkling different from other MoE models?**
Inkling is notable for being trained from scratch with fully open weights, a context window of one million tokens, and native multimodal input support. Unlike many models that rely on external encoders, Inkling handles images and audio directly within a decoder-only MoE framework.
**Q2: What is the purpose of Inkling-Small?**
Inkling-Small is a more compact version with 276B total parameters and 12B active parameters. It is designed to offer similar benchmark performance at lower resource cost, making experimentation and deployment more accessible.
**Q3: Can I fine-tune Inkling on my own data?**
Yes. The model weights are open and described as “fine-tunable on Tinker,” indicating that fine-tuning is a core use case.
**Q4: What hardware was used for training?**
Training was carried out on **NVIDIA GB300 NVL72 systems**, with Muon used for large matrix updates.
**Q5: How does Inkling handle multimodal inputs?**
Images are tokenized into 40×40 pixel patches, and audio is represented as dMel spectrograms. Both are projected into a shared embedding space and processed together with text by the decoder.
**Q6: What does the “effort dial” in the visualization represent?**
It demonstrates how increased reasoning effort trades off against generation speed and token efficiency, based on internal benchmarks.
—
### Conclusion
With Inkling, Thinking Machines Lab presents an open, multimodal, and highly customizable foundation model built around a modern MoE architecture. By combining a permissive approach to weights with strong context handling and native support for multiple input modalities, Inkling aims to serve as a flexible base for research, product development, and domain-specific adaptation. The accompanying interactive visualization offers an accessible way to explore the model’s design, while the FAQ clarifies its key features and intended use cases.
