AIML Rounds

QUIZ

Generation and LLM evals

6 questions on Generation and LLM evals, from 6 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering.

  1. Question 1 of 6

    What does raising the sampling temperature do?

    1. A. It flattens the distribution; rare tokens appear more
    2. B. It removes all randomness from sampling
    3. C. It changes the model's weights for that request
    4. D. It makes outputs shorter and more focused on the likeliest words
    View answer

    ✓ A. It flattens the distribution; rare tokens appear more

    Logits are divided by T before the softmax: higher T means more diverse and riskier text.

    Lesson: Decoding: Temperature, Top-k, Top-p and Min-p

  2. Question 2 of 6

    How does min-p sampling choose which tokens to keep?

    1. A. Only the single most likely token
    2. B. Tokens above a fraction of the top token's probability
    3. C. Tokens until their probabilities add up to p, in order of likelihood
    4. D. The k tokens with the highest probability
    View answer

    ✓ B. Tokens above a fraction of the top token's probability

    The cut-off adapts to how confident the model is, unlike a fixed top-k or top-p.

    Lesson: Decoding: Temperature, Top-k, Top-p and Min-p

  3. Question 3 of 6

    Why does chain-of-thought prompting help on multi-step maths problems?

    1. A. It lowers the temperature automatically
    2. B. It makes the model larger for that question
    3. C. It lets the model search the web for the answer while it writes
    4. D. Writing steps gives it room to compute before answering
    View answer

    ✓ D. Writing steps gives it room to compute before answering

    Each generated step becomes context for the next, so hard problems are broken into easier ones.

    Lesson: In-Context Learning and Chain-of-Thought

  4. Question 4 of 6

    Why do LLMs hallucinate with confidence?

    1. A. They read the internet live while answering and repeat its errors
    2. B. They are lazy and choose the easiest answer
    3. C. Temperature is always set too high
    4. D. Training rewards a plausible guess over 'I don't know'
    View answer

    ✓ D. Training rewards a plausible guess over 'I don't know'

    Ground answers in sources, allow abstaining, and measure it, rather than relying on the model to know its limits.

    Lesson: Hallucination: Why LLMs Make Things Up

  5. Question 5 of 6

    A new model scores suspiciously high on a public benchmark. What should you suspect?

    1. A. Contamination: test questions were in its training data
    2. B. It was evaluated on faster GPUs than the other models
    3. C. Nothing: it's simply a much better model overall
    4. D. The benchmark is too hard for the other models tested
    View answer

    ✓ A. Contamination: test questions were in its training data

    Use held-out or freshly written test sets and check small score gaps for statistical noise.

    Lesson: Benchmarks and Contamination

  6. Question 6 of 6

    Roughly how many FLOPs does it take to train a model with N parameters on D tokens?

    1. A. About N² · D operations
    2. B. About 6 · N · D
    3. C. About D / N operations
    4. D. About N + D operations
    View answer

    ✓ B. About 6 · N · D

    About 2ND for the forward pass and 4ND for the backward pass is the standard back-of-the-envelope estimate.

    Lesson: LLM Practice 1: Pretraining and Scaling Maths