Skip to content
ai

fp16

16-bit Floating Point

Definition

FP16 is a half-precision floating-point format that uses 16 bits (10-bit mantissa, 5-bit exponent) compared to 32-bit FP32. It halves memory usage and increases throughput on GPUs with tensor core support, but has a narrower dynamic range that can cause overflow or underflow during training.

Mixed precision training typically keeps a master copy of weights in FP32 while performing forward and backward passes in FP16.

Can this ship? Are we exposed? Can you prove it?

Every CISO is asking these three questions. We answer them with facts, not guesses, from repository to runtime.