Skip to content
ai

Mixed precision

Mixed Precision Training

Definition

Mixed precision training performs the forward and backward passes in lower-precision formats (FP16 or BF16) while maintaining a full-precision (FP32) master copy of weights for parameter updates. This reduces memory usage and increases throughput on modern GPUs without significant accuracy loss.

Gradient scaling prevents underflow of small gradient values in FP16.

Can this ship? Are we exposed? Can you prove it?

Every CISO is asking these three questions. We answer them with facts, not guesses, from repository to runtime.