GenAI App Scenarios: Set 1

Answer: stale documents are still in the index (no versioning or deletion on update)

1. Check what was retrieved for the failing questions: are old policy chunks among the top results?
2. Re-indexing added the new policy but didn't remove the old one, so both compete; the old one may even match the wording better.
3. Fix: version documents, delete or mark superseded chunks, filter retrieval by 'current' metadata, and show the policy date in citations; add these questions to the golden set.

Follow-up: How do you stop this happening again? Make ingestion idempotent per document ID, alert on duplicate policies, and run the eval set after every re-index.

Answer: use schema-constrained decoding (structured outputs / guided decoding), then validate values

1. Prompted JSON fails occasionally, and more often with larger schemas; retries add cost and tail latency.
2. Switch to the provider's strict structured outputs or the engine's guided decoding with the JSON Schema, so every output parses.
3. Keep Pydantic validation for values (ranges, enums, cross-field rules) and log violations.

Follow-up: Accuracy on refund reasoning dropped after adding strict JSON. Why? Format constraints can hurt reasoning: add a reasoning field before the decision fields, or reason first and extract in a second call.

Answer: the fine-tune taught style but also over-fitted to chat content: forgetting plus learned confident guessing

1. Training on chats where agents stated amounts teaches the model to state amounts, even when it doesn't know the policy (fine-tuning on unknown facts increases hallucination).
2. Narrow data can erode general abilities (forgetting), especially with many epochs or a high learning rate.
3. Fix: remove or mask factual specifics in training targets, keep facts in RAG/tool calls, mix in general instruction data, use fewer epochs / lower rank, and compare against the base model on the full eval set.

Follow-up: How would you have caught it before launch? A golden set with policy-accuracy checks and general-skill cases, run on the base and fine-tuned models side by side.

Answer: look for lost prompt-cache hits, longer prompts or outputs, and routing changes

1. Break cost down by input/output tokens, cached vs uncached input, and model tier per day; find when it jumped.
2. Common causes: a prompt edit placing dynamic text (date, user name) at the start, killing prefix caching; RAG retrieving more or longer chunks; outputs getting longer; a router threshold change sending more traffic to the big model; retry loops.
3. Fix the specific cause, add cost-per-request alerts and dashboards by component.

Follow-up: How would you prevent silent cost regressions? Track tokens and cost per request in the eval/CI run for each release, and alert on cache-hit-rate drops.

Answer: indirect prompt injection; treat retrieved content as untrusted data and restrict what it can influence

1. Delimit retrieved content and instruct the model it is data, not instructions (helps, not sufficient).
2. Discounts and refunds must come from trusted systems via tools/code, not from text the model read; output checks can block unverified promises.
3. The bigger worry: if the same pattern reaches a tool (issue refunds, send messages), untrusted text could trigger actions. Enforce least privilege and human approval for high-impact actions; red-team with injected content.

Follow-up: How do you detect it at scale? Scan ingested content with an injection classifier, log tool calls with the content that triggered them, and add injected documents to the eval set.

Answer: don't ship yet: 30% of claims aren't supported by retrieved context, so correctness comes partly from the model's memory

1. Low faithfulness with high correctness means the model often answers from pre-training knowledge, not your documents; it will be confidently wrong when policies change or for company-specific facts.
2. Check context recall: if needed facts often aren't retrieved, fix retrieval (chunking, hybrid search, re-ranking); if they are retrieved but ignored, tighten the prompt (answer only from context, cite clauses, say 'not found').
3. Re-measure both metrics, and add policy-change test cases where memory and documents disagree.

Follow-up: What test would expose the risk directly? Change a policy value in a test index (e.g. cap ₹150 → ₹100) and check whether answers follow the documents.

Answer: the average hides failures on a hard minority of requests that the router misclassifies

1. Judge averages are dominated by easy questions; misrouted hard cases (disputes, multi-issue complaints) fail and get escalated, which costs agent time.
2. Analyse escalated conversations by route and intent; evaluate the router's precision on hard cases, not just overall quality.
3. Fix: raise the threshold or add rules for hard intents, use a cascade (escalate to the big model when the small model's answer fails a check), and optimise cost per resolved conversation including agent cost.

Follow-up: Which metric should the team have optimised? Total cost per resolved conversation (LLM + human), with quality measured on stratified slices, not just the overall average.