infrastructure
GGUF
GGML Unified Format
The file format that lets LLMs run on any device
Reading level
PRACTITIONER — Technical context
GGUF (successor to GGML, August 2023) is the container format used by llama.cpp for quantized LLM inference. It stores model weights, tokenizer vocabulary, architecture metadata, and quantization config in a single self-contained file. Quantization levels (Q2_K through F16) trade bits-per-weight against quality — Q4_K_M (4.83 bpw) is the community default, achieving near-FP16 quality at 30% of the memory footprint.
Real-world example
A Llama 3.1 70B model in FP16 requires ~140GB of RAM. The Q4_K_M GGUF version is ~42GB — small enough for a MacBook Pro with 64GB or an RTX 4090. Ollama auto-selects the right quantization for your hardware.
infrastructurequantizationlocalllama.cpp