training

Quantization

Model Weight Quantization

Making AI models smaller without making them much dumber

Reading level

PRACTITIONER — Technical context

Quantization reduces bit-width of model weights from FP32/FP16 to INT8 or INT4. Post-training quantization (PTQ) methods: GGUF k-quants, GPTQ (Hessian-based), AWQ (activation-aware). Quality tradeoff: INT4 (Q4_K_M) achieves ~98% of FP16 perplexity at 30% memory. Quantization-aware training (QAT) fine-tunes with simulated quantization for higher accuracy.

Real-world example

Llama 3.1 70B in FP16: 140GB RAM needed. After Q4_K_M quantization: 42GB RAM, with only ~2% benchmark degradation. A $3,000 MacBook Pro with 64GB can now run it — vs needing a $50,000 multi-GPU server.

traininginferencelocalhardware