architecture
Transformer
Transformer Architecture
The invention that made modern AI possible
Reading level
PRACTITIONER — Technical context
The Transformer (Vaswani et al., 2017 'Attention Is All You Need') replaced RNNs with self-attention layers that compute pairwise relationships between all tokens in parallel. The architecture: multi-head self-attention → layer norm → feed-forward network → layer norm. Positional encodings add sequence information. Encoders (BERT) process context; decoders (GPT) generate autoregressively; encoder-decoders (T5) do both.
Real-world example
When you ask an AI 'What did the man say who wore the red hat?' — the transformer uses attention to correctly link 'man', 'red hat' and 'say' together even if they're far apart in the sentence.
architecturefundamentalsresearch