Mixed precision

Mixed precision training performs the forward and backward passes in lower-precision formats (FP16 or BF16) while maintaining a full-precision (FP32) master copy of weights for parameter updates. This reduces memory usage and increases throughput on modern GPUs without significant accuracy loss.

Gradient scaling prevents underflow of small gradient values in FP16.