AIML Rounds

QUIZ

LLMOps

7 questions on LLMOps, from 11 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering.

  1. Question 1 of 7

    What makes operating LLM apps different from classic ML ops?

    1. A. There are no deployments to manage, only API calls to outside vendors
    2. B. LLMs never change once they're deployed
    3. C. Nothing: it's exactly the same as classic ML ops
    4. D. Prompts, models and data all change it; outputs need evals
    View answer

    ✓ D. Prompts, models and data all change it; outputs need evals

    Version prompts and configs like code, and test changes against evaluation sets.

    Lesson: LLMOps: What's Different

  2. Question 2 of 7

    Users say the assistant 'got worse this week', but all dashboards are green. What should you have been recording?

    1. A. Only the error rates and status codes returned by each endpoint
    2. B. CPU usage on the servers
    3. C. Nothing; 'worse' is too subjective to measure
    4. D. Full traces plus quality signals like evals and feedback
    View answer

    ✓ D. Full traces plus quality signals like evals and feedback

    Uptime and latency can be fine while answers degrade; you need traces to find what changed.

    Lesson: Tracing and Observability for LLM Apps and Agents

  3. Question 3 of 7

    Your provider retires your model in 60 days, and the replacement scores higher on benchmarks. How do you migrate safely?

    1. A. Switch on day 59, when the old model stops
    2. B. Run your own evals, adjust prompts, roll out gradually
    3. C. Keep using the old model after it's retired
    4. D. Trust the higher benchmark scores and switch at once
    View answer

    ✓ B. Run your own evals, adjust prompts, roll out gradually

    Public benchmarks don't measure your task; your evals and a staged rollout do.

    Lesson: Model Upgrades and Retirements

  4. Question 4 of 7

    Why can't input filters alone stop prompt injection?

    1. A. Filters would block all normal text as well
    2. B. Prompt injection isn't a real risk in practice
    3. C. Attacks vary endlessly, so limit what the model can do
    4. D. Filters are too slow to run on every request at production scale
    View answer

    ✓ C. Attacks vary endlessly, so limit what the model can do

    Treat model output as untrusted, give tools least privilege, and require approval for risky actions.

    Lesson: Guardrails and Prompt Injection

  5. Question 5 of 7

    Which approach typically cuts an LLM bill the most without losing quality?

    1. A. Turn off logging and tracing to save on storage
    2. B. Route easy requests to small models, cache, batch
    3. C. Always use the biggest model to avoid retries
    4. D. Raise max tokens, so each answer comes back in one call
    View answer

    ✓ B. Route easy requests to small models, cache, batch

    Most traffic doesn't need the strongest model; routing and caching remove most of the cost.

    Lesson: Cost and Latency: Routing, Caching and Batching

  6. Question 6 of 7

    Your LLM provider returns overload errors at peak time, and your retries make it worse. What should the client do?

    1. A. Switch off timeouts, so each call can wait for as long as needed
    2. B. Retry immediately in a tight loop
    3. C. Send every request twice to improve the odds
    4. D. Back off with jitter, respect Retry-After, cap retries
    View answer

    ✓ D. Back off with jitter, respect Retry-After, cap retries

    Synchronised instant retries create a retry storm; spreading and limiting them lets the service recover.

    Lesson: LLM API Reliability: Rate Limits, Retries and Fallbacks

  7. Question 7 of 7

    A RAG app keeps citing an outdated policy page that was updated last week. Where is the likely problem?

    1. A. The LLM's training data contains the old policy
    2. B. The temperature is set too high for questions about company policy
    3. C. The UI is caching an old answer
    4. D. The index wasn't refreshed, so old chunks are retrieved
    View answer

    ✓ D. The index wasn't refreshed, so old chunks are retrieved

    Incidents like stale citations point at the data pipeline, not the model.

    Lesson: GenAI App Scenarios: Set 1