QUIZ
Post-training
8 questions on Post-training, from 8 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering.
Question 1 of 8
In supervised fine-tuning for chat, why is the loss masked on the prompt tokens?
- A. So it learns to write replies, not to imitate user prompts
- B. It isn't masked: prompt tokens are always trained on
- C. To save GPU memory during fine-tuning
- D. Because prompts are random text that carries no useful signal for training
View answer
✓ A. So it learns to write replies, not to imitate user prompts
Only the assistant's tokens are targets; the prompt is context.
Lesson: SFT and Instruction Tuning
Question 2 of 8
How can a reward that isn't differentiable, such as 'the answer is correct', still train an LLM?
- A. Policy gradients raise the probability of high-reward outputs
- B. By turning the reward into a label for supervised training
- C. It can't: rewards must be differentiable to train with gradient descent
- D. By backpropagating through the grader's code
View answer
✓ A. Policy gradients raise the probability of high-reward outputs
You only need the log-probability of what was sampled; a baseline reduces the variance of the updates.
Lesson: Policy Gradients for LLMs
Question 3 of 8
What goes wrong if you optimise a learned reward model too hard?
- A. The reward model improves along with the policy
- B. The policy exploits the reward model's blind spots
- C. Training stops early because rewards stop changing
- D. Nothing: a higher reward score always means better answers
View answer
✓ B. The policy exploits the reward model's blind spots
A KL penalty to the reference model and early stopping limit this over-optimisation.
Lesson: Reward Models and RLHF (PPO)
Question 4 of 8
How does DPO differ from RLHF with PPO?
- A. It needs no preference data of any kind
- B. It only works for image generation models
- C. It trains on preference pairs directly, no reward model
- D. It uses more models, adding a critic network on top of standard RLHF
View answer
✓ C. It trains on preference pairs directly, no reward model
DPO's loss implicitly defines the reward, which makes training simpler and more stable.
Lesson: DPO and Preference Optimisation
Question 5 of 8
In GRPO, used to train reasoning models, what replaces a learned value (critic) model?
- A. No baseline at all: raw rewards are used directly
- B. Each answer's reward relative to its group's mean
- C. A second LLM that predicts each answer's expected value
- D. Human ratings of each sampled answer
View answer
✓ B. Each answer's reward relative to its group's mean
The group mean acts as the baseline, which saves training a critic.
Question 6 of 8
What does distillation mean for LLMs?
- A. Training a small model on a big model's outputs
- B. Quantising the weights to fewer bits each
- C. Removing some layers from a model at random
- D. Compressing the training dataset so that it takes less space
View answer
✓ A. Training a small model on a big model's outputs
Learning from a strong teacher's traces can beat training the small model with RL on its own.
Question 7 of 8
How does LoRA make fine-tuning cheap?
- A. It skips the backward pass for most layers
- B. It trains every weight in the model, but in 8-bit precision
- C. It removes layers that don't change during training
- D. It freezes the base and trains small low-rank matrices
View answer
✓ D. It freezes the base and trains small low-rank matrices
Only a tiny fraction of parameters train. QLoRA also keeps the frozen base in 4-bit to fit on one GPU.
Question 8 of 8
Why do jailbreaks still work on safety-trained models?
- A. Safety training is never applied to released models
- B. Refusal training misses some phrasings, like role-play or codes
- C. Models are simply unable to refuse any request they're given
- D. Jailbreaks change the model's weights at runtime
View answer
✓ B. Refusal training misses some phrasings, like role-play or codes
Defence needs several layers: training, input and output filters, and limiting what the model can do.
Lesson: Safety Training and Jailbreaks