Understanding RAG: Retrieval-Augmented Generation
Learn how RAG works from first principles — embeddings, vector stores, chunking, and prompt construction — with practical examples for building your own knowledge-grounded AI system.
Understanding RAG: Retrieval-Augmented Generation
RAG is the most practical technique for grounding AI responses in your own data. Instead of hoping the model memorized your documents during training, RAG retrieves relevant context at inference time and provides it directly to the model.
The Core Problem RAG Solves
Large language models have two big limitations:
1. Knowledge cutoff — they don't know about things that happened after training
2. Hallucination — they will confidently invent information they don't know
RAG addresses both by making the model answer from your documents, not from memory.
How RAG Works
User Query
↓
Embed query → vector
↓
Search vector store for similar document chunks
↓
Insert top-k chunks into prompt
↓
LLM generates answer grounded in retrieved chunks
↓
Return answer + citations
Step 1: Chunking Documents
You can't embed an entire document as one unit. Split it into chunks:
from langchain.text_splitter import RecursiveCharacterTextSplittersplitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_text(document_text)
Chunk size matters. Too small → insufficient context. Too large → noise and cost. 300–800 tokens is a good starting point.
Step 2: Creating Embeddings
Embeddings convert text into dense vectors that capture semantic meaning:
from openai import OpenAIclient = OpenAI()
def embed(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
Step 3: Storing in a Vector Database
import chromadbclient = chromadb.Client()
collection = client.create_collection("my-docs")
collection.add(
documents=chunks,
embeddings=[embed(c) for c in chunks],
ids=[f"chunk-{i}" for i in range(len(chunks))]
)
Step 4: Retrieval + Generation
def answer(question: str) -> str:
# Retrieve relevant chunks
results = collection.query(
query_embeddings=[embed(question)],
n_results=5
)
context = "\n\n".join(results["documents"][0])
# Generate grounded answer
messages = [
{"role": "system", "content": f"Answer using only this context:\n\n{context}"},
{"role": "user", "content": question}
]
return llm.chat(messages)