Mastering Low-Precision AI: FP8 and FP4 Support Across Frameworks in Mid-2026

In mid-2026, FP8 and FP4 have become essential tools for making large-scale AI training and inference more efficient . FP8 uses two main formats-E4M3 for better precision on activations and weights, and E5M2 for wider dynamic range on gradientswhile NVIDIA’s NVFP4 takes things further with 4-bit values and micro-block scaling (shared FP8 scales per 16 elements plus a tensor-level scale). These formats dramatically cut memory use and increase throughput on modern GPUs compared with traditional BF16 or FP16, making it possible to train and serve bigger models on the same hardware. The benefits are clear: roughly 2× memory savings with FP8 and up to 3.5× with NVFP4, higher Tensor Core performance, and better energy efficiency. The trade-offs come from reduced numerical range and precision, which can lead to accuracy loss or instability unless carefully managed with techniques such as delayed scaling, stochastic rounding, Hadamard transforms, and selective quantization that skips sensitive layers. When these methods are applied properly, accuracy often stays within 1–2 % of higher-precision baselines on real workloads. Research has moved quickly from the foundational 2022 FP8 paper to 2025 studies showing stable FP4 pre-training of multi-billion-parameter models. Hardware support is mature for FP8 on Hopper GPUs and reaches its peak on Blackwell with native NVFP4 and MXFP8 acceleration. Among frameworks, PyTorch currently leads with native float8 dtypes, Transformer Engine for production training, and TorchAO for optimized inference. JAX offers solid support through Transformer Engine, TensorFlow/Keras provides simpler quantize-to-FP8 options but relies more on TensorRT for high performance, and libraries such as bitsandbytes remain useful for complementary 4-bit memory savings. Practical adoption is already strong for both training and inference, especially when teams start with proven recipes, monitor scaling factors, and prototype on smaller models. Workarounds for r