In a base language model (the output of pre-training), the neural net has learned to predict the next token over a ton of training text data. However, it has not yet learned how to be helpful (e.g. respond to user queries or follow instructions). For example, with just pretraining, the model could just continue the sentence instead of answering. Other failure modes include: topic drift and rambling, as they are trained to continue generate tokens autoregressively (which can go on indefinitely); and an inability to refuse harmful requests.
The general post-training landscape looks like taking the pretrained model, adding an SFT stage (instruction following), taking Human Evals (preference data), training a reward model for quality scoring, and then using RL/DPO to actually finetune the pretrain the model for actual usefulness.
- SFT: Used to teach instruction format where the inputs are (prompt, golden response) pairs
- Reward Modeling: used to learn to score responses where the inputs are triplets of (prompt, response A, response B) and a human rater chooses which response is better. Here, the data is
- RLHF: used to maximize expected reward while staying close to the SFT model where the input is the base model and reward $$ \mathcal{L}_{\mathrm{RLHF}} = -\mathbb{E}\left[\mathrm{reward}(\text{response})\right]
\beta , \mathrm{KL}!\left(\pi_{\text{new}} ,|, \pi_{\text{old}}\right)
For RLHF, [[Proximal Policy Optimization (PPO)]], GRPO, DPO are commonly used. ## Notes **Foundations** - [[Reinforcement Learning from Human Feedback (RLHF)]] - [[Reward Model Training]] - [[RLAIF]] **Supervised fine-tuning** - [[Supervised Fine-Tuning]] - [[SFT Data Quality]] - [[Catastrophic Forgetting]] **Safety and evaluation** - [[Constitutional AI]] - [[Alignment Evaluation]] **On-policy RL** - [[PPO for RLHF]] - [[KL Penalty Estimators]] - [[GRPO]] - [[Dr GRPO]] - [[PTX Loss]] **Direct preference optimization** - [[DPO]] - [[DPO Variants]] - [[DPO Implicit Reward]] **Sampling-based** - [[RAFT and Rejection Sampling]] - [[Pass@k and Majority Voting]] **Implementation** - [[Preference Data Pipeline]] - [[Sequence Log-Probabilities]]