infrastructure

vLLM

Virtual LLM Serving Library

The production inference engine behind most LLM APIs

Reading level

PRACTITIONER — Technical context

vLLM (UC Berkeley) is the de facto standard for production LLM serving. Its core innovation is PagedAttention — managing GPU KV cache like OS virtual memory, eliminating fragmentation and increasing utilization from ~30% to 90%+. Enables continuous batching (adding requests mid-batch), achieving 10–24× higher throughput vs naive HuggingFace serving. Supports tensor parallelism across multiple GPUs.

Real-world example

A single H100 GPU with naive HuggingFace inference: ~100 requests/min. With vLLM PagedAttention + continuous batching: ~2,400 requests/min — 24× improvement, cutting cost-per-request by 96%.

infrastructureproductionCUDA