**LingBot-VA 2.0: Ant Group’s Embodied AI for Generalist Robot Manipulation**
Ant Group’s embodied AI unit, Robbyant, has introduced **LingBot-VA 2.0**, positioning itself as the first *embodied-native foundation model* for generalist robot manipulation. Rather than adapting digital-content video generators, the team designed and pretraining the entire stack with embodiment as the core objective. The result is a video–action foundation model that enables robots to understand and interact with the physical world.
—
### What is LingBot-VA 2.0?
Most existing video–action models rely on components originally built for digital content creation: a reconstruction-oriented VAE and a bidirectional video-diffusion backbone with an attached action module. This architecture leads to four key mismatches for robot control:
1. Pixel-reconstruction latents preserve appearance but lack physical structure.
2. Iterative denoising over video tokens is too slow for closed-loop control.
3. Generic video objectives do not teach how actions reshape the world.
4. Bidirectional backbones conflict with the strictly causal nature of robot control.
LingBot-VA 2.0 addresses these issues by pretraining a **causal DiT (DiTusion Transformer)** natively, instead of finetuning a video generator. The model learns a **semantic visual–action tokenizer** that unifies world states and actions in a single latent space, trained on unlabeled web video without task-specific supervision.
—
### Version 1: The Semantic Visual–Action Tokenizer
The first stage replaces the standard VAE with a tokenizer that adds **semantic alignment and action-aware objectives**:
– A frozen Perception Encoder teacher aligns visual latents.
– A latent-action objective models transitions between consecutive latents.
– An inverse dynamics model predicts latent actions; a forward dynamics model decodes them into transport maps and residuals.
This process embeds world states and actions into a shared latent space, allowing web-scale, unlabeled video to provide action-relevant supervision.
—
### Version 2: A Causal DiT with a Sparse MoE Video Stream
Building on the tokenizer, LingBot-VA 2.0 pretrains a **causal DiT** with a Mixture-of-Transformers (MoT) architecture:
– A video expert uses a **sparse MoE layer** (128 experts, top-8 routing, one shared expert).
– An action expert retains a dense feed-forward network (hidden dimension 768).
– Both streams share causal self-attention but follow separate parameter paths.
– The video backbone is roughly **13B parameters**, with about **1.9B active per token**.
– Total training parameters reach **~15.3B**, with **~2.5B activated per inference token**.
– Training employs a rectified-flow objective and a hybrid Muon + AdamW optimizer.
—
### Training Signal: Multi-Chunk Prediction and Multi-Objective Co-Training
Two design choices shape what the model learns:
– **Multi-Chunk Prediction (MCP)**: Three lightweight predictors anticipate future chunks, reducing myopic supervision and accelerating training by **2.3×**.
– **Five Co-Training Objectives**: T2I, T2V, TI2VA, ICL, and human–robot co-training are trained jointly in a coarse-to-fine schedule, preventing the model from overwriting earlier capabilities.
—
### Hierarchical Planning and Foresight Reasoning
To handle long-horizon tasks, LingBot-VA 2.0 uses a **VLM planner** (LoRA-finetuned with a frozen vision tower) that emits structured JSON instructions at ~2 Hz. The low-level policy reads these instructions at chunk boundaries, ensuring planner latency never blocks execution.
For real-time performance, **Foresight Reasoning** runs prediction and execution in parallel:
– While the robot executes chunk *aₜ*, the video expert imagines the resulting latent.
– The action expert proposes *aₜ₊₁*.
– Upon receiving the real observation, the latent is re-ground to correct drift.
This pipeline reduces control latency significantly.
—
### Performance
Evaluated on **RoboTwin 2.0** across 50 tasks (2,500 clean + 25,000 randomized demonstrations), LingBot-VA 2.0 achieves:
– **93.8% success on clean data**
– **93.4% success on randomized data**
– **93.6% average**
These results outperform prior methods, including X-VLA, π0.5, and Motus.
Speed optimizations deliver:
– **142 ms per chunk** end-to-end latency
– **225 Hz async control**
– **5× end-to-end speedup** versus a BF16 PyTorch baseline
Techniques such as consistency distillation, FP8 TensorRT compilation, long-horizon attention optimization, and runtime overhead reduction drive these gains.
—
### Version 1.0 vs. Version 2.0
| Dimension | LingBot-VA | LingBot-VA 2.0 |
|——————————-|———————-|————————————-|
| Tokenizer | Wan2.2 VAE | Semantic visual–action tokenizer |
| Backbone | Finetuned generator | Causal DiT, pretrained from scratch |
| Video FFN | Dense | Sparse MoE (128 experts, top-8) |
| Supervision | None | MCP + multi-objective co-training |
| Inference | Async execution | Foresight reasoning + re-grounding |
| Peak async control frequency | Not reported | 225 Hz |
The tokenizer change alone improves a 1.3B model’s success rate from 78.0% to 86.6%.
—
### Use Cases and Examples
– **Few-shot onboarding**: Adapts from 10–20 teleoperated demos per task.
– **Demonstration-conditioned control**: Human videos replace text instructions.
– **Cheap data scaling**: Human–robot co-training retargets hand poses, creating 65.4k episodes.
– **Reactive control**: Demonstrations in Air Hockey and conveyor settings show anticipatory behavior.
—
### Key Takeaways
– Pretrains a causal video–action DiT from scratch instead of adapting video generators.
– Semantic tokenizer aligns visual and action latents in one space.
– Sparse MoE video stream activates ~2.5B of ~15.3B parameters per token.
– Foresight reasoning overlaps prediction and execution with re-grounding.
– Achieves sub-150 ms latency and >200 Hz async control.
—
### Interactive Dynamic Explainer
A live demo illustrates Foresight Reasoning, the acceleration stack, and benchmark results in real time, showing model latency, control frequency, and success rates across methods.
—
**Conclusion**
LingBot-VA 2.0 represents a shift from repurposed generative models to **embodied-first foundation models** for robot manipulation. By combining a causal, action-aware architecture with sparse attention, multi-chunk prediction, and co-trained objectives, it delivers state-of-the-art performance both in simulation and on real hardware. This work establishes a clear path toward scalable, general-purpose robot intelligence grounded in interaction with the physical world.
