training

RLHF

Reinforcement Learning from Human Feedback

How AI learns to be helpful and safe from human ratings

Reading level

PRACTITIONER — Technical context

RLHF consists of three phases: 1) Supervised fine-tuning (SFT) on demonstration data, 2) Reward model training — humans rank responses, a neural reward model learns to predict human preference scores, 3) RL optimization — the LLM policy is updated using PPO (or similar) to maximize the reward model's score, with a KL-divergence penalty to prevent over-optimization ('reward hacking').

Real-world example

OpenAI used RLHF to make ChatGPT much more helpful and less harmful than the raw GPT base model. Human raters gave feedback on thousands of responses, and the model learned what 'good' answers looked like.

trainingalignmentfundamentals