Hardware & Inference — The full stack
From chip to token — every layer of the LLM inference stack
Apple Silicon vs NVIDIA GPUs · MLX and Core ML · llama.cpp, vLLM, TGI, TensorRT-LLM · GGUF quantization — everything you need to understand and run AI models on any hardware.
01 — Apple Silicon
The best consumer hardware for local AI
Apple Silicon's unified memory architecture eliminates the CPU↔GPU bottleneck that limits CUDA-based local inference. The GPU and CPU share the same physical DRAM — no PCIe transfers, no memory copies, no bottleneck.
Why this matters for LLMs
LLM inference is memory bandwidth-bound, not compute-bound. Every generated token requires loading all model weights from memory. A Mac M4 Max at 546 GB/s rivals an A100 (2 TB/s) once you account for the 4-bit quantization available locally — effective bandwidth for a Q4 model is comparable. And the Mac does it silently, on battery.
Frameworks for Apple Silicon
MLX is Apple's open-source machine learning framework designed from scratch for the unified memory architecture of Apple Silicon. Unlike PyTorch or JAX which were designed for CUDA GPUs, MLX treats the CPU and GPU as a unified device — no data copies between memory pools. This makes it dramatically more efficient for inference on Mac hardware.
pip install mlxCore ML is Apple's framework for deploying ML models on iOS, iPadOS, macOS, watchOS, and tvOS. It automatically routes computation to the most efficient processor — CPU, GPU, or Neural Engine. Models converted to .mlpackage format run with optimized memory access and power efficiency. Used by on-device Siri, autocorrect, and third-party apps.
02 — NVIDIA GPUs
The industry standard for AI training and serving
CUDA's ecosystem dominance makes NVIDIA the default for production AI. From RTX 4090 for local development to GB200 NVL72 racks for hyperscale serving — the full range.
03 — Inference Software Stack
Runtimes, serving engines, and harness tools
The layer between your model weights and a running API. Choose based on hardware, scale, and latency requirements.
Inference Stack Layers (top → bottom)
llama.cpp is Georgi Gerganov's C/C++ implementation of LLM inference that runs on virtually any hardware with no GPU required. It introduced the GGUF quantization format, enabling 4-bit quantized models to run on consumer hardware. It's the engine behind Ollama, LM Studio, and dozens of other tools.
brew install llama.cpp GitHub Key capability: Universal cross-platform inference with GGUF quantization
vLLM is the leading open-source LLM serving framework for production use, developed at UC Berkeley. Its core innovation is PagedAttention — a memory management algorithm inspired by OS virtual memory that increases GPU KV cache utilization from ~30% to over 90%, enabling much higher throughput at lower cost.
pip install vllm GitHub Key capability: PagedAttention — 24× higher throughput vs naive HuggingFace inference
mlx-lm is Apple's official LLM package built on top of MLX. It provides CLI tools and a Python API for running HuggingFace models on Apple Silicon with full Metal GPU acceleration and unified memory. Supports QLoRA fine-tuning directly on Mac — no CUDA GPU needed.
pip install mlx-lm GitHub Key capability: Native Apple Silicon LLM inference + QLoRA fine-tuning on Mac
TensorRT-LLM is NVIDIA's production inference library that compiles LLMs into highly optimized CUDA kernels. It applies operator fusion, INT8/FP8 quantization, and in-flight batching automatically. Used by NVIDIA Triton inference server for serving at data-center scale.
docker pull nvcr.io/nvidia/tritonserver GitHub Key capability: Operator fusion + FP8 quant = maximum tokens/sec on NVIDIA hardware
TGI is HuggingFace's open-source LLM serving solution. It implements continuous batching, tensor parallelism across multiple GPUs, token streaming, and quantization (GPTQ, AWQ, EETQ). It powers the HuggingFace Inference Endpoints service and is widely used in enterprise deployments.
docker run ghcr.io/huggingface/text-generation-inference GitHub Key capability: Continuous batching + tensor parallelism + streaming
Ollama wraps llama.cpp with a user-friendly CLI, automatic model management, and a local REST API that mirrors OpenAI's API. One command to pull and run any model from their library. Auto-detects and uses Metal (Mac), CUDA (NVIDIA), or falls back to CPU.
brew install ollama && ollama run llama3.1 GitHub Key capability: One-command model management + OpenAI-compatible local API
04 — Quantization
GGUF levels, AWQ, GPTQ — choosing the right format
Quantization lets you trade a small amount of quality for a large reduction in memory and compute. Q4_K_M has become the community default — 4× smaller than FP16 with ~98% of the quality.
AWQ is a 4-bit quantization method that identifies and protects the 1% of weights most important for model quality. Unlike GPTQ (which quantizes uniformly), AWQ achieves better perplexity at the same bit width. Widely supported by vLLM and TGI.
GPTQ is a post-training quantization method that uses second-order information (Hessians) to minimize quantization error layer by layer. Produces INT4 models with minimal accuracy loss. Widely used for HuggingFace model deployment.