Skip to content
ai

GPTQ

Generative Pre-trained Transformer Quantization

Definition

GPTQ is a one-shot post-training quantization method for LLMs that uses second-order information (Hessian approximation) to minimize quantization error layer by layer. It achieves high-quality 4-bit and 3-bit quantized models with minimal accuracy loss, enabling large models to run on single consumer GPUs.

GPTQ is widely used with the AutoGPTQ library for serving quantized open-source models.

Can this ship? Are we exposed? Can you prove it?

Every CISO is asking these three questions. We answer them with facts, not guesses, from repository to runtime.