## Article: AngelSpec — Tailored Speculative Decoding for Diverse Workloads
Tencent has open‑sourced **AngelSpec**, a **torch‑native** training framework for **speculative‑decoding draft (proposer) models**. The framework is designed around a key insight in production LLM serving: **“one universal drafter underperforms”** because real traffic mixes open‑ended conversation, code, and mathematical reasoning. Instead of searching for a single “best‑average” proposer, AngelSpec treats **workload heterogeneity as a first‑class design constraint**, and provides **specialized drafters** whose structure, training data, and verification depth are optimized for each regime.
—
### Why One Universal Drafter Underperforms
Speculative decoding is lossless: a lightweight drafter proposes multiple future tokens, and a target (verifier) model validates them in a single forward pass via rejection sampling. Acceleration therefore depends on (1) how many proposed tokens are accepted, and (2) how long the draft–verify round takes. These two quantities move in opposite directions across domains:
– **High‑entropy open‑ended conversation** favors **short, autoregressive Multi‑Token Prediction (MTP)** proposals, since many continuations are valid and long blocks are rejected quickly.
– **Code and mathematics** involve strong syntactic constraints and predictable patterns, enabling **longer, block‑parallel drafts** that amortize verification cost effectively.
AngelSpec ships **two complementary drafters**, rather than a single compromise:
1. **MTP drafter** — trained on diverse, conversation‑oriented data.
2. **Block‑diffusion (DFly) drafter** — strengthened with code‑ and math‑focused samples.
—
### The MTP Path: Training‑Time Test and Target‑Model Rollout
A key mismatch in traditional approaches is between training and inference:
– The original Hy3 MTP model is trained with a **single prediction layer** and no recurrent self‑conditioning.
– At inference, the draft block can be reused **recurrently**, but the training objective never prepared it for long self‑generated chains, causing error accumulation at deeper positions.
AngelSpec resolves this via **Training‑Time Test (TTT)** and **target‑model rollout**:
– **Shared‑parameter, multi‑depth scheme**: The model retains ( D ) logical prediction depths but reuses one physical MTP block.
– During training, the block is **autoregressively unrolled** for ( D ) steps, with each depth supervised independently.
– Depth ( k+1 ) receives the **arg‑max prediction from depth ( k )** instead of the ground truth, aligning training with inference.
– The **target model and its LM head are frozen**, and MTP inputs from the backbone are detached, so the drafter improves its proposal distribution without distorting verification.
– **Target‑model rollout** generates drafts using the frozen target, producing the exact token choices and hidden‑state trajectories seen at serving time.
**Results**:
– Mean acceptance at ( T=0 ) rises from **52.8% → 66.4%**.
– Mean accepted length grows from **2.58 → 2.99**.
– Deeper positions improve substantially:
– ( p_3 ) on GSM8K: **0.290 → 0.706**
– ( p_3 ) on HumanEval: **0.387 → 0.757**
—
### MTP vs Block‑Parallel DFlash: Domain‑Specific Strengths
| Drafter | Best Use Case | Key Strength |
|———|————–|————–|
| MTP | Open‑ended chat | Short, flexible proposals |
| DFlash | Code / Math | Long, predictable spans |
| DSpark | MT‑Bench | Strong reasoning traces |
| DFly (final) | Balanced | Highest average throughput |
AngelSpec explicitly leverages this diversity, delivering **state‑of‑the‑art throughput** while preserving per‑domain accuracy.
—
### D‑Cut: Production‑Grade Verification Budget Control
In production, AngelSpec introduces **D‑Cut**, a mechanism that dynamically adjusts the verification budget (fraction of tokens verified) to maximize throughput:
– Evaluated on **Hunyuan Live traffic** against Hy3‑295B‑A21B with 8× H200 GPUs.
– At **c64 concurrency**, D‑Cut achieves:
– **+15.7% throughput** vs DFly alone
– Mean accepted length drops only from **2.50 → 2.43**
– D‑Cut selects the verification ratio that maximizes estimated batch progress per unit latency.
—
### Frequently Asked Questions (FAQ)
**Q1: What problem does AngelSpec solve?**
AngelSpec solves the “one‑size‑fits‑all” limitation of speculative decoding by providing **domain‑aware drafters** and a **Training‑Time Test** framework that aligns training with real‑serving autoregressive reuse.
**Q2: How is AngelSpec different from Hy3 and EAGLE?**
– Like EAGLE, AngelSpec uses **Training‑Time Test** and **target‑model rollout**.
– Unlike EAGLE, it supports **multi‑depth MTP** with shared parameters and **separate block‑parallel drafters** for code/math workloads.
**Q3: Does AngelSpec increase latency?**
No. By aligning training with inference and freezing the target model, AngelSpec **reduces verification waste**, lowering end‑to‑end latency while increasing throughput.
**Q4: Can D‑Cut be used with other frameworks?**
Yes. D‑Cut is workload‑agnostic and can be integrated into any speculative‑decoding pipeline that supports variable verification budgets.
**Q5: What hardware was used for the reported results?**
Benchmarks were run on **Hy3** systems with **H200 GPUs** and **8× H100** configurations for throughput studies.
—
### Conclusion
AngelSpec represents a **paradigm shift** in speculative decoding: rather than optimizing a single universal drafter, it **co‑designs architecture, training data, and verification strategy around workload characteristics**. By combining **multi‑depth MTP Training‑Time Test**, **target‑model rollout**, and **dynamic verification budgeting (D‑Cut)**, AngelSpec achieves substantial gains in throughput and acceptance rates across conversational, coding, and mathematical workloads. This makes it a practical and scalable solution for real‑world LLM serving at Tencent and beyond.
