AIML Rounds

QUIZ

Pretraining

7 questions on Pretraining, from 5 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering.

  1. Question 1 of 7

    How does one sentence of 8 tokens become 8 training examples?

    1. A. Every position predicts its next token, all in one pass
    2. B. The sentence is copied 8 times into the dataset
    3. C. Each word is treated as a separate document with its own label
    4. D. It doesn't: one sentence is one example
    View answer

    ✓ A. Every position predicts its next token, all in one pass

    Next-token prediction makes every position a target, which is why pretraining is so data-efficient per sequence.

    Lesson: The Next-Token Objective

  2. Question 2 of 7

    Why is deduplication important for pretraining data?

    1. A. Models can't read the same piece of text twice
    2. B. Only to save disk space on the training cluster
    3. C. Duplicates cause memorisation, waste compute, leak tests
    4. D. Duplicates help, because they reinforce the high-quality text
    View answer

    ✓ C. Duplicates cause memorisation, waste compute, leak tests

    Removing near-duplicates improves quality per token and reduces regurgitation.

    Lesson: Pretraining Data: Deduplication, Filtering and Mixtures

  3. Question 3 of 7

    By the Chinchilla scaling result, roughly how many training tokens per parameter is compute-optimal?

    1. A. It doesn't matter
    2. B. About 1,000
    3. C. About 1
    4. D. About 20
    View answer

    ✓ D. About 20

    Earlier models were undertrained. Chinchilla found model size and tokens should grow together, around 20 tokens per parameter.

    Lesson: Scaling Laws: Kaplan vs Chinchilla

  4. Question 4 of 7

    Why was a small model like Llama 3 8B trained on about 15 trillion tokens, far beyond 'compute-optimal'?

    1. A. To test how many GPUs the cluster could use
    2. B. Over-training a small model makes it better and cheaper to serve
    3. C. By mistake: the run was left going too long
    4. D. Small models need much more data than large ones just to work at all
    View answer

    ✓ B. Over-training a small model makes it better and cheaper to serve

    Compute-optimal minimises training cost; when inference dominates total cost, a smaller, longer-trained model wins.

    Lesson: Scaling Laws: Kaplan vs Chinchilla

  5. Question 5 of 7

    Why does training a 70B model with Adam in mixed precision need around a terabyte of memory?

    1. A. ~16 bytes per parameter: weights, grads, FP32 copy, Adam
    2. B. Activations alone take a terabyte at this model size
    3. C. The vocabulary's embedding table is extremely large at this size
    4. D. The training dataset itself must sit in GPU memory
    View answer

    ✓ A. ~16 bytes per parameter: weights, grads, FP32 copy, Adam

    70B × 16 bytes ≈ 1.1 TB before activations, so the model is sharded across many GPUs with FSDP or ZeRO.

    Lesson: Training at Scale: Mixed Precision, Parallelism, ZeRO/FSDP

  6. Question 6 of 7

    What do ZeRO and FSDP do?

    1. A. They compress the training data so that it takes up less GPU memory
    2. B. They lower the learning rate as training goes on
    3. C. They shard parameters, gradients and optimiser state
    4. D. They skip the backward pass for frozen layers
    View answer

    ✓ C. They shard parameters, gradients and optimiser state

    Each GPU holds only a slice and gathers the rest when needed, cutting memory per GPU.

    Lesson: Training at Scale: Mixed Precision, Parallelism, ZeRO/FSDP

  7. Question 7 of 7

    The loss spikes suddenly at step 40,000. Which is a standard response?

    1. A. Ignore it; loss spikes always recover on their own
    2. B. Roll back, skip bad batches or lower LR, add z-loss
    3. C. Delete the optimiser state and keep training on
    4. D. Double the learning rate to push through the spike fast
    View answer

    ✓ B. Roll back, skip bad batches or lower LR, add z-loss

    Spikes often come from exploding logits or attention scores; stabilisers keep them bounded.

    Lesson: Training Instabilities: Loss Spikes, z-loss and QK-norm