AIML Rounds

QUIZ

Post-training

8 questions on Post-training, from 8 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering.

  1. Question 1 of 8

    In supervised fine-tuning for chat, why is the loss masked on the prompt tokens?

    1. A. So it learns to write replies, not to imitate user prompts
    2. B. It isn't masked: prompt tokens are always trained on
    3. C. To save GPU memory during fine-tuning
    4. D. Because prompts are random text that carries no useful signal for training
    View answer

    ✓ A. So it learns to write replies, not to imitate user prompts

    Only the assistant's tokens are targets; the prompt is context.

    Lesson: SFT and Instruction Tuning

  2. Question 2 of 8

    How can a reward that isn't differentiable, such as 'the answer is correct', still train an LLM?

    1. A. Policy gradients raise the probability of high-reward outputs
    2. B. By turning the reward into a label for supervised training
    3. C. It can't: rewards must be differentiable to train with gradient descent
    4. D. By backpropagating through the grader's code
    View answer

    ✓ A. Policy gradients raise the probability of high-reward outputs

    You only need the log-probability of what was sampled; a baseline reduces the variance of the updates.

    Lesson: Policy Gradients for LLMs

  3. Question 3 of 8

    What goes wrong if you optimise a learned reward model too hard?

    1. A. The reward model improves along with the policy
    2. B. The policy exploits the reward model's blind spots
    3. C. Training stops early because rewards stop changing
    4. D. Nothing: a higher reward score always means better answers
    View answer

    ✓ B. The policy exploits the reward model's blind spots

    A KL penalty to the reference model and early stopping limit this over-optimisation.

    Lesson: Reward Models and RLHF (PPO)

  4. Question 4 of 8

    How does DPO differ from RLHF with PPO?

    1. A. It needs no preference data of any kind
    2. B. It only works for image generation models
    3. C. It trains on preference pairs directly, no reward model
    4. D. It uses more models, adding a critic network on top of standard RLHF
    View answer

    ✓ C. It trains on preference pairs directly, no reward model

    DPO's loss implicitly defines the reward, which makes training simpler and more stable.

    Lesson: DPO and Preference Optimisation

  5. Question 5 of 8

    In GRPO, used to train reasoning models, what replaces a learned value (critic) model?

    1. A. No baseline at all: raw rewards are used directly
    2. B. Each answer's reward relative to its group's mean
    3. C. A second LLM that predicts each answer's expected value
    4. D. Human ratings of each sampled answer
    View answer

    ✓ B. Each answer's reward relative to its group's mean

    The group mean acts as the baseline, which saves training a critic.

    Lesson: Reasoning Models: RLVR, GRPO and Test-Time Compute

  6. Question 6 of 8

    What does distillation mean for LLMs?

    1. A. Training a small model on a big model's outputs
    2. B. Quantising the weights to fewer bits each
    3. C. Removing some layers from a model at random
    4. D. Compressing the training dataset so that it takes less space
    View answer

    ✓ A. Training a small model on a big model's outputs

    Learning from a strong teacher's traces can beat training the small model with RL on its own.

    Lesson: Distillation: How Small Models Learn from Big Ones

  7. Question 7 of 8

    How does LoRA make fine-tuning cheap?

    1. A. It skips the backward pass for most layers
    2. B. It trains every weight in the model, but in 8-bit precision
    3. C. It removes layers that don't change during training
    4. D. It freezes the base and trains small low-rank matrices
    View answer

    ✓ D. It freezes the base and trains small low-rank matrices

    Only a tiny fraction of parameters train. QLoRA also keeps the frozen base in 4-bit to fit on one GPU.

    Lesson: LoRA and QLoRA: Parameter-Efficient Fine-Tuning

  8. Question 8 of 8

    Why do jailbreaks still work on safety-trained models?

    1. A. Safety training is never applied to released models
    2. B. Refusal training misses some phrasings, like role-play or codes
    3. C. Models are simply unable to refuse any request they're given
    4. D. Jailbreaks change the model's weights at runtime
    View answer

    ✓ B. Refusal training misses some phrasings, like role-play or codes

    Defence needs several layers: training, input and output filters, and limiting what the model can do.

    Lesson: Safety Training and Jailbreaks