In a base language model (the output of pre-training), the neural net has learned to predict the next token over a ton of training text data. However, it has not yet learned how to be helpful (e.g. respond to user queries or follow instructions). For example, with just pretraining, the model could just continue the sentence instead of answering. Other failure modes include: topic drift and rambling, as they are trained to continue generate tokens autoregressively (which can go on indefinitely); and an inability to refuse harmful requests.
The general post-training landscape looks like taking the pretrained model, adding an SFT stage (instruction following), taking Human Evals (preference data), training a reward model for quality scoring, and then using RL/DPO to actually finetune the pretrain the model for actual usefulness.
- SFT: Used to teach instruction format where the inputs are (prompt, golden response) pairs
- Reward Modeling: used to learn to score responses where the inputs are triplets of (prompt, response A, response B) and a human rater chooses which response is better. Here, the data is
- RLHF: used to maximize expected reward while staying close to the SFT model where the input is the base model and reward $$ \mathcal{L}_{\mathrm{RLHF}} = -\mathbb{E}\left[\mathrm{reward}(\text{response})\right]
\beta , \mathrm{KL}!\left(\pi_{\text{new}} ,|, \pi_{\text{old}}\right)