NVIDIA Unveils 4-Bit Pretraining with NVFP4: Scaling a 12B Hybrid Mamba-Transformer Across 10 Trillion Tokens

Training cutting-edge large language models (LLMs) using FP8 precision has become the norm, but dropping down to 4-bit floating point has been a persistent challenge. The issue is that narrower formats squeeze the dynamic range and magnify quantization errors over long sequences of tokens. A new study from NVIDIA introduces a pretraining approach built on NVFP4, a 4-bit microscaling format with native support in Blackwell Tensor Cores. The team validated this method by training a 12-billion-parameter hybrid Mamba-Transformer model on 10 trillion tokens. According to the researchers, this represents the longest publicly documented 4-bit precision training run so far. The finished model scores 62.58% on MMLU-Pro 5-shot, compared to 62.62% for the FP8 baseline, and is supported through NVIDIA’s Transformer Engine.

Understanding NVFP4

To appreciate why NVFP4 matters, it’s useful to look at how microscaling formats function. In a microscaling (MX) format, a group of consecutive low-precision values shares one scale factor, which maps the group back to a wider numerical range during matrix multiplication. MXFP4 uses blocks of 32 elements, with each element stored as E2M1 — that’s 1 sign bit, 2 exponent bits, and 1 mantissa bit — representing only the values ±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, and ±6. Block scale factors are stored in UE8M0, limiting them to powers of two.

NVFP4 introduces three key changes. First, the block size shrinks from 32 to 16 elements, reducing the dynamic range each scale factor must handle. Second, block scale factors use E4M3 instead of UE8M0, sacrificing some exponent range for greater mantissa precision so the per-block amax (absolute maximum value) can be mapped much closer to the FP4 maximum. Third, NVFP4 adds a second level of scaling: an FP32 per-tensor scale that remaps values so the E4M3 block scales themselves remain representable. The outcome is that at least 6.25% of values in each block — specifically the per-block amax — are captured at near-FP8 accuracy, while the rest use standard FP4.

On NVIDIA Blackwell hardware, FP4 GEMMs deliver 4× the throughput of BF16 on GB200 and 6× on GB300, translating to roughly 2× and 3× speedups over FP8. Memory usage for operands is cut roughly in half compared to FP8.

What Gets Quantized — and What Stays in Higher Precision

Only the GEMMs within linear (fully-connected) layers — specifically the forward pass (Fprop), gradient with respect to inputs (Dgrad), and gradient with respect to weights (Wgrad) — actually run in NVFP4. Embeddings, the output projection head, normalization layers, activation functions, and all attention components (including softmax and the query-key and attention score-value batched GEMMs) remain in BF16 or FP32. Model weights, weight gradients used for accumulation across microbatches and data-parallel replicas, and optimizer states are all maintained in FP32. Tensor parallel reductions are performed in BF16.

The Four-Part Training Methodology

Simply quantizing every linear-layer GEMM to NVFP4 using default settings (1×16 block scaling everywhere, round-to-nearest-even on every tensor, no transforms) causes training to diverge early. NVIDIA’s approach stabilizes training through four components, and ablation studies on the 12B model confirm that each one is essential.

Selective high precision: Linear layers in the first two and the last eight of the model’s 62 blocks (roughly 16% of all linear layers) are kept in BF16. Ablation experiments showed that the final blocks are the most sensitive because they need more dynamic range than FP4 can offer; keeping only the last four blocks in BF16 was also sufficient for stable convergence.

Random Hadamard Transforms (RHT): Outlier values in weight gradients are redistributed into an approximately Gaussian distribution by multiplying input tiles with a 16×16 Hadamard matrix combined with a random ±1 sign vector. Since the orthogonal transforms cancel out within the dot product, no mathematical correction is required in the GEMM. The d=16 size was determined through experimentation: d=4 harmed convergence, while d=128 produced similar results. RHT is applied only to the inputs of the weight-gradient (Wgrad) GEMM, and a single random sign

The same vector is used for all linear layers. At the 1.2B scale, randomization had no effect, but it produced a measurable improvement in the 12B run.

Two-dimensional (2D) block scaling for weights: Conventional NVFP4 applies scaling to 1×16 blocks along the dot-product axis. Since the backward pass transposes the weight matrix, the forward and backward passes end up working with different quantized versions of the weights, which violates the chain rule. NVIDIA addresses this by scaling weights in 16×16 blocks so that both passes share the same quantized representation. Activations and gradients continue to use 1×16 scaling, as they are less affected by this mismatch.

Stochastic rounding on gradients: Round-to-nearest-even creates a systematic bias when used on gradient tensors. Stochastic rounding instead rounds probabilistically, weighting the choice by how close the value is to each of the two nearest representable numbers, which eliminates that bias. The researchers explicitly state in their paper that stochastic rounding is harmful when applied to forward-pass tensors, so its use is limited to gradients only.

Results on the 12B Hybrid Mamba-Transformer

The 12B model is built on the Nemotron-Nano-12B-v2-Base architecture — 62 blocks (6 Self-Attention, 28 FFN, 28 Mamba-2), hidden dimension 5120, FFN dimension 20480 — trained with a Warmup-Stable-Decay schedule (constant learning rate for the first 80% of training, then decay over the remaining 20%), batch size 736, sequence length 8192. The FP8 reference baseline follows the DeepSeek-V3 approach: E4M3 elements, 128×128 weight blocks, 1×128 activation and gradient blocks, with the first block and last two blocks kept in BF16.

NVFP4 validation loss remains within 1% of the FP8 baseline during the stable phase and drifts to just over 1.5% during the decay phase. Downstream accuracy is similar across most benchmarks: MMLU 76.57% vs 77.36%, GSM8K CoT 92.27% vs 89.08%, MATH 81.48% vs 83.32%, AGIEval English CoT 70.31% vs 67.01%. Coding benchmarks show the widest gap — HumanEval+ 57.43% vs 59.93%, MBPP+ 55.91% vs 59.11% — which the team partly attributes to noise in the final-checkpoint evaluation. The team also describes a precision-switching strategy: switching the forward pass from NVFP4 to BF16 starting at 8.2T tokens (roughly 18% into the schedule) cut the relative loss error from 1.5% down to 0.5%.

NVFP4 vs MXFP4

On a separate 8B hybrid Mamba-Transformer trained on 1T tokens, NVFP4 achieved a relative loss error of roughly 1.5% compared to BF16, while MXFP4 hovered around 2.5%. To match the NVFP4 loss at 1T tokens, MXFP4 needed 1.36T tokens — a 36% increase in token consumption. The team traces this gap to NVFP4’s smaller block size and E4M3 scales, which retain more of the FP4 dynamic range than MXFP4’s power-of-two UE8M0 scales (which can sacrifice up to one entire binade and the ±4, ±6 values in the worst case).

Marktechpost’s Visual Explainer