QUIZ
LLMOps
7 questions on LLMOps, from 11 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering.
Question 1 of 7
What makes operating LLM apps different from classic ML ops?
- A. There are no deployments to manage, only API calls to outside vendors
- B. LLMs never change once they're deployed
- C. Nothing: it's exactly the same as classic ML ops
- D. Prompts, models and data all change it; outputs need evals
View answer
✓ D. Prompts, models and data all change it; outputs need evals
Version prompts and configs like code, and test changes against evaluation sets.
Lesson: LLMOps: What's Different
Question 2 of 7
Users say the assistant 'got worse this week', but all dashboards are green. What should you have been recording?
- A. Only the error rates and status codes returned by each endpoint
- B. CPU usage on the servers
- C. Nothing; 'worse' is too subjective to measure
- D. Full traces plus quality signals like evals and feedback
View answer
✓ D. Full traces plus quality signals like evals and feedback
Uptime and latency can be fine while answers degrade; you need traces to find what changed.
Question 3 of 7
Your provider retires your model in 60 days, and the replacement scores higher on benchmarks. How do you migrate safely?
- A. Switch on day 59, when the old model stops
- B. Run your own evals, adjust prompts, roll out gradually
- C. Keep using the old model after it's retired
- D. Trust the higher benchmark scores and switch at once
View answer
✓ B. Run your own evals, adjust prompts, roll out gradually
Public benchmarks don't measure your task; your evals and a staged rollout do.
Lesson: Model Upgrades and Retirements
Question 4 of 7
Why can't input filters alone stop prompt injection?
- A. Filters would block all normal text as well
- B. Prompt injection isn't a real risk in practice
- C. Attacks vary endlessly, so limit what the model can do
- D. Filters are too slow to run on every request at production scale
View answer
✓ C. Attacks vary endlessly, so limit what the model can do
Treat model output as untrusted, give tools least privilege, and require approval for risky actions.
Lesson: Guardrails and Prompt Injection
Question 5 of 7
Which approach typically cuts an LLM bill the most without losing quality?
- A. Turn off logging and tracing to save on storage
- B. Route easy requests to small models, cache, batch
- C. Always use the biggest model to avoid retries
- D. Raise max tokens, so each answer comes back in one call
View answer
✓ B. Route easy requests to small models, cache, batch
Most traffic doesn't need the strongest model; routing and caching remove most of the cost.
Question 6 of 7
Your LLM provider returns overload errors at peak time, and your retries make it worse. What should the client do?
- A. Switch off timeouts, so each call can wait for as long as needed
- B. Retry immediately in a tight loop
- C. Send every request twice to improve the odds
- D. Back off with jitter, respect Retry-After, cap retries
View answer
✓ D. Back off with jitter, respect Retry-After, cap retries
Synchronised instant retries create a retry storm; spreading and limiting them lets the service recover.
Lesson: LLM API Reliability: Rate Limits, Retries and Fallbacks
Question 7 of 7
A RAG app keeps citing an outdated policy page that was updated last week. Where is the likely problem?
- A. The LLM's training data contains the old policy
- B. The temperature is set too high for questions about company policy
- C. The UI is caching an old answer
- D. The index wasn't refreshed, so old chunks are retrieved
View answer
✓ D. The index wasn't refreshed, so old chunks are retrieved
Incidents like stale citations point at the data pipeline, not the model.
Lesson: GenAI App Scenarios: Set 1