# Two Labs, Same Blueprint: How GLM-5.3-Flash and Qwen3.8-Flash-Next Converged on a Shared Architecture
This week has been a landmark moment for open-weight large language models. Two major releases from Chinese AI labs arrived within 24 hours of each other, and what’s striking isn’t just the timing — it’s how remarkably similar their internal designs are. Despite building from scratch and working independently, Z.ai’s GLM-5.3-Flash and Alibaba’s Qwen3.8-Flash-Next arrived at nearly identical architectural decisions. This article explores the shared blueprint, the one area where they diverge, and the lab that pushed back entirely.
## The Releases That Sparked the Conversation
Z.ai introduced GLM-5.3-Flash, a 320-billion-parameter multimodal mixture-of-experts model with 18 billion parameters activated per forward pass. It supports a 1-million-token context window and was trained on a 30-trillion-token multimodal corpus. The model is MIT-licensed and available on Hugging Face, with API pricing set at $0.15 per million input tokens and $0.50 per million output tokens. Z.ai claims it matches or exceeds the previous generation at a fraction of the cost while approaching Claude Opus 4.8 on coding and agent tasks.
Alibaba’s Qwen team followed with Qwen3.8-Flash-Next, a 125-billion-parameter model that activates 6 billion parameters per token. It serves a native context of 262,144 tokens, extendable to 1 million through YaRN scaling. The architecture is positioned as an early public look at the Qwen4 design family, and the team reported that training required roughly one-ninth the compute budget of its predecessor, Qwen3.7-Plus.
## The Shared Recipe
### A 3:1 Ratio of Cheap to Expensive Layers
Both models organize their transformer layers into a 3:1 split between lightweight linear attention and full precision attention. GLM-5.3-Flash uses 34 linear layers and 11 sparse attention layers across 45 total layers. Qwen3.8-Flash-Next uses 36 Gated DeltaNet layers and 12 Qwen Sparse Attention layers across 48 total layers. The ratio is the same even though the absolute numbers differ slightly.
The linear layers act as cheap compressors. Rather than maintaining a key-value cache that grows with every incoming token, they compress the entire conversation history into a fixed-size recurrent state. This means compute cost per token stays constant regardless of how long the input gets. GLM implements this through Kimi Delta Attention, applying a per-channel decay mechanism, while Qwen uses its own Gated DeltaNet variant that operates at the per-head level. Different implementations of the same underlying principle.
The remaining quarter of layers handle precise retrieval. These are the layers where actual key-value caches persist, and where the second major convergence appears.
### The Indexer: Compress Four Times, Keep 2048 Tokens
Neither model allows its retrieval layers to attend across the full context window. Instead, both attach a small learned scoring network that evaluates chunks of history and retains only the most relevant ones. The parameters are almost identical.
GLM uses a 32-head lightning indexer with top-2048 token selection. To handle the cost at million-token scales, Z.ai added a pooling mechanism called IndexPool that compresses four indexer key vectors into one before scoring. Qwen’s approach operates at a finer granularity: its compressed indexer scores 4-token micro-blocks and keeps the top 512 blocks, which equals exactly 2048 tokens. Both compress context by 4x before scoring and both enforce a 2048-token attention budget.
The results are significant. Z.ai reports that the Flash architecture cuts attention compute by approximately 3x and reduces KV cache memory by 4.4x compared to the full GLM-5.3 model, while also halving active parameters and layer count.
### Four Gated Streams Replace One
Perhaps the most dramatic shared decision is the abandonment of the single residual stream that has been standard in transformers since 2017. Both models expand the residual pathway into four parallel branches, each governed by learned gates that control what information flows in and out.
GLM uses Manifold-Constrained Hyper-Connections, a design originating from DeepSeek, configured with exactly 4 branches. Qwen developed its own variant called Gated Residual, which uses a data-dependent read gate and a per-branch scalar write gate. The Qwen team ablated both approaches and found them equivalent in quality. Two separate labs, two different implementations, same conclusion: four gated streams outperform one.
### Muon Optimizer with Per-Component Splitting
Both models were trained using the Muon optimizer, which applies orthogonalization to projection matrices. The shared refinement here is that fused matrices are decomposed into their component transformations before Muon processes them. Qwen, for example, splits fused QKV, SwiGLU, and GDN projections so that Muon handles genuine two-dimensional linear maps while AdamW manages embeddings, routers, and low-rank parameters. Qwen also eliminated batch-size warmup entirely after finding it consumed 18.8% more optimizer steps with no measurable quality gain.
## Where They Split: Positional Encoding
The cleanest disagreement between the two models involves rotary position embeddings in the full-attention layers. GLM-5.3-Flash removes them entirely, setting the configuration parameter `qk_rope_head_dim` to zero. Position information instead flows through the recurrent linear layers. Qwen experimented with this approach and found identical pre-training performance. However, after post-training alignment, the NoPE variant frequently failed to stop generating text. Qwen therefore kept RoPE, and this result serves as an important warning for the broader community: pre-training loss curves can mask behavioral defects that only surface during downstream tuning.
## The Dissenter: MiniMax
Not everyone in the field agrees with this convergence. MiniMax conducted extensive ablations during its M2 development, testing both linear attention and sliding-window attention at scale. The results were troubling: severe deficits in multi-hop reasoning emerged, particularly beyond 32,000 tokens after supervised fine-tuning. M2 shipped with standard softmax attention in every layer. For M3, MiniMax adopted MiniMax Sparse Attention, which prunes softmax attention through block selection but includes no linear-attention layers whatsoever. The verdict from MiniMax’s scaled experiments is that linear attention, at least in their configuration, degrades complex reasoning.
This creates a fascinating fork in the road. Z.ai, Qwen, DeepSeek, and Moonshot’s Kimi are all betting that a 3:1 hybrid of linear and full attention preserves capability while dramatically improving efficiency. MiniMax’s data says otherwise for their stack, and their M3 architecture is the proof of concept.
## Key Takeaways
– Two labs independently converged on the same 3:1 linear-to-full attention ratio, validating the hybrid attention paradigm.
– Both compress context by 4x and cap sparse attention at 2048 tokens, following patterns pioneered by DeepSeek’s DSA.
– Both abandon the single residual stream in favor of four gated branches, with Qwen confirming equivalence to DeepSeek’s mHC design.
– The disagreement centers on positional encoding, with GLM dropping RoPE and Qwen retaining it after NoPE caused post-training failures.
– MiniMax stands as the field’s counterpoint, with ablations showing linear attention harms multi-hop reasoning at scale.
## Frequently Asked Questions
**What makes these two models different if their architectures are so similar?**
While the architectural patterns overlap significantly, the models differ in scale and specialization. GLM-5.3-Flash is a 320B-parameter multimodal MoE with 18B active parameters, while Qwen3.8-Flash-Next is a 125B model with 6B active parameters. GLM also natively supports multimodal input from the start, whereas Qwen3.8-Flash-Next is primarily a text model previewing its next-generation architecture.
**Why does the 3:1 ratio matter?**
The ratio balances cost and capability. Linear attention layers are computationally cheap — their state size stays fixed regardless of context length — while full-attention layers provide precise long-range retrieval. The 3:1 split means 75% of layers operate efficiently on long contexts, with 25% performing targeted retrieval when needed.
**What is the significance of the 2048-token attention budget?**
Capping sparse attention at 2048 tokens means the expensive retrieval layers never need to process more than that amount, regardless of whether the input is 1,000 tokens or 1,000,000 tokens. This creates a hard ceiling on compute and memory use that makes million-token context windows economically viable for serving.
**Why did Qwen keep RoPE while GLM dropped it?**
Both teams tested NoPE during pre-training and saw no difference in loss curves. However, after post-training, GLM’s NoPE variant did not exhibit the same failure to stop generating that Qwen observed. This suggests the interaction between position encoding and alignment techniques may be more nuanced than pre-training metrics alone can reveal.
**What does MiniMax’s dissent mean for the industry?**
MiniMax’s findings suggest that the benefits of linear attention may depend heavily on model architecture, training data, and downstream task distribution. Their M3’s return to pure softmax attention with sparse pruning offers an alternative path that other labs will need to evaluate against their own use cases.
## Looking Ahead
The simultaneous emergence of these two models from different labs points toward a broader pattern in AI development. Architecture components — linear attention layers, sparse indexers, multi-branch residual streams, Muon optimization — are being shared, adapted, and independently validated across the Chinese open-model ecosystem. This cross-pollination accelerates progress but also raises questions about diversity of approach. With MiniMax charting a different course and Qwen’s NoPE cautionary tale, the field is far from settled on any single blueprint.
The coming months will reveal whether the 3:1 hybrid becomes the dominant paradigm or whether alternative arrangements prove superior for specific workloads. For now, the convergence itself is the story: two labs, working in parallel, arrived at the same answer — and that agreement may matter more than the disagreement.
Thank you for reading
