← All posts

The case for FP4

We are at an inflection point for low-precision computation. With sparsity, FP4 reaches 20 PFLOPS per GPU on GB300 and 50 PFLOPS on Rubin—8× and 12.5× their dense BF16 peaks, respectively.1

Those numbers change where we need to spend our effort. Faster matrix multiplication makes quantisation, sparsification, scaling and data movement more expensive relative to the work they support. But we have exceptionally many levers to pull and explore to train and serve models in FP4:

  • Choose formats by layer. NVFP4 offers finer scaling, while MXFP4 offers more scale range. NVIDIA’s pretraining experiments found the final layers particularly sensitive to FP4, giving us a concrete reason to explore different formats across the network.
  • Change the scale. Our UE5M3 experiments kept the four-bit payload unchanged and demonstrated stable pretraining without Hadamard transforms in software emulation. Changing the scale can change the stabilisation recipe too, which makes alternatives such as UE4M4 worth investigating.
  • Change the payload. E2M1 offers more range, while E1M2 offers more evenly spaced values. MixFP4 explores choosing between them block by block, matching the representation to the local distribution of values.
  • Improve stability. Tensor scaling, stochastic rounding and Hadamard transforms give us several ways to manage quantisation error. Four Over Six shows that even choosing between two scaling targets—four and six—can improve accuracy without changing the four-bit payload.
  • Add sparsification. Structured sparsity, such as 2:4 or V:N:M, reduces the work left for tensor cores. Sparse transformer pretraining shows that pruning costs and accuracy recovery must be addressed alongside kernel speed; combining these lessons with FP4 gives us another direction to explore.

I believe there exists a world where we no longer need to train, tune or serve LLMs in BF16, and every layer can run as A4W4.2

We should care about this because serving a model on 4 GPUs instead of 16 at the same speed would cut GPU costs per token by a factor of 4, at the same hourly rate per GPU.

The path to full A4W4 has proven perilous: we face quantisation and stabilisation overheads, transcendentals that can't keep up with FP4 tensor cores, and the deeper hardware optimisation needed to saturate them.

With the advancement of LLM coding agents, the gap between intuition and machine code is the smallest it's been since the inception of CUDA, making optimal low precision kernels and numerics more attainable than ever.

This is a call to the community to be optimistic and act with urgency in hardware aware ML. LLMs are driving another industrial revolution, and growing demand for tokens from more capable models makes efficient computation essential. As uncle Jensen says: “Now is the most important time. Just dedicate yourself to now.”

  1. These headline ratios compare sparse FP4 with dense BF16: 20/2.5 PFLOPS on GB300 and 50/4 PFLOPS on Rubin. They therefore combine precision and sparsity gains. The dense-only ratios are 6× and 8.75×. GB300 also supports sparse BF16 at 5 PFLOPS per GPU, giving a sparse-to-sparse ratio of 4×. ↩
  2. Here, A4W4 means four-bit activation and weight operands for matrix multiplication, with wider accumulators, scales and optimiser state. Peak FLOPS alone do not establish application speed or cost: those gains have to survive the complete training or serving workload. ↩

References

  1. NVIDIA. GB300 NVL72 specifications.
  2. NVIDIA. Vera Rubin NVL72 specifications.
  3. Format-Aware Fusion for Fast FP4 Pretraining.
  4. Fast Polynomial Transcendentals for LLMs.
  5. Pretraining Large Language Models with NVFP4.
  6. UE5M3 FP4 Block Scaling for Stable Language Model Pretraining.
  7. MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations.
  8. Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling.
  9. Elucidating the Design Space of FP4 Training.
  10. Beyond 2:4: Exploring V:N:M Sparsity for Efficient Transformer Inference on GPUs.
  11. Accelerating Transformer Pre-training with 2:4 Sparsity.
  12. CASPA. Jensen Huang: Career Tips for the Age of AI.