ai
GPTQ
Generative Pre-trained Transformer Quantization
Definition
GPTQ is a one-shot post-training quantization method for LLMs that uses second-order information (Hessian approximation) to minimize quantization error layer by layer. It achieves high-quality 4-bit and 3-bit quantized models with minimal accuracy loss, enabling large models to run on single consumer GPUs.
GPTQ is widely used with the AutoGPTQ library for serving quantized open-source models.
Can this ship? Are we exposed? Can you prove it?
Every CISO is asking these three questions. We answer them with facts, not guesses, from repository to runtime.