Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) is a method for teaching a model human preferences by training it directly on pairs of ranked responses — a "better" answer and a "worse" one — without the separate reward model and reinforcement-learning loop that traditional RLHF requires. Introduced in 2023, it reframes preference tuning as a simpler supervised-style objective, which makes it cheaper, more stable, and easier to run than a full RLHF pipeline. For builders, DPO is why fine-tuning a model on your own preference data has become far more approachable: instead of standing up an RL training stack, you collect examples of good and bad outputs for your use case and optimize the model toward the good ones. Many open-weight instruction-tuned models are now aligned with DPO or its variants. Practical note: DPO is only as good as your preference pairs — noisy, inconsistent, or unrepresentative comparisons teach the wrong lesson, so invest in clean, deliberate labeling rather than sheer volume.