Agentic AI Practice

Answer: no: it's a fixed workflow (classify → route); use a classifier or a single LLM call with structured output

1. The path doesn't depend on intermediate results; there's one decision with a fixed set of outcomes.
2. A single structured-output call (or a fine-tuned small classifier) is cheaper, faster, easier to test and evaluate.
3. Reserve agent loops for parts that need investigation with tools, like resolving the routed ticket.

Follow-up: What would make it agentic? If handling required looking things up and deciding next steps based on what was found.

Answer: about 0.98³⁰ ≈ 55%; shorten the task and add error detection and recovery

1. 0.98^30 = e^(30·ln 0.98) ≈ e^(−0.606) ≈ 0.55.
2. Shorten: move deterministic steps into code or tools, so the model makes fewer decisions.
3. Detect and recover: validate tool arguments and results, retry failed steps, verify with checks; per-step reliability compounds.

Follow-up: What if errors are correlated (a wrong plan early)? Independent retries help less; add checkpoints that re-plan and human escalation.

Answer: load fewer tools per step (tool search / grouping), make tools distinct and well-described, and evaluate tool selection

1. Retrieve only the relevant tools for the current task (tool search, RAG over tool descriptions), or route to sub-agents with smaller toolsets; RAG-MCP raised selection accuracy from 13.6% to 43.1% this way.
2. Merge overlapping tools, rename them clearly, add examples and parameter constraints (enums, IDs).
3. Build a tool-selection test set and track wrong-tool and bad-argument rates per change.

Follow-up: What does each extra tool cost even when unused? Its definition (~100–300 tokens) on every call, plus distraction.

Answer: 0.99¹⁵⁰ ≈ 22% clean runs; add per-step checkpoints, retries with backoff and idempotent tools

1. Clean run probability = 0.99^150 ≈ e^(−1.51) ≈ 0.22.
2. Restarting from scratch repeats most of the work (expected executions ≈ (1 − 0.99^150) / (0.01 × 0.99^150) ≈ 350 for 150 steps).
3. Checkpoint after each step (≈ 152 executions), retry transient errors with backoff, and make side-effecting tools idempotent so replays are safe.

Follow-up: Which step types need idempotency keys? Anything with side effects: refunds, emails, writes to external systems.

Answer: not necessarily: measure consistency (pass^k), end-state correctness, and cost; run each case several times

1. A single run hides flakiness: some cases are solved only sometimes. Run each case k times (e.g. 5) and report pass^5 alongside pass@1.
2. Make sure 'success' means the right end state (refund recorded correctly), not a plausible message.
3. Report cost and latency per resolved case, and break down failures by type (tool errors, policy mistakes, escalations).

Follow-up: Why might production be worse than the eval? Distribution shift (new issues, phrasing), injected content, tool outages; monitor online resolution and escalation rates.

Answer: it has the lethal trifecta; an injected email can make it exfiltrate documents. Remove or gate a leg.

1. Private data (documents, inbox) + untrusted content (incoming email) + external communication (send email) = exfiltration path via prompt injection.
2. Controls: require user confirmation for every outbound email (with a clear preview), restrict recipients (allow-list or only replying to existing threads), block auto-rendering of external links/images.
3. Architecturally: process untrusted emails in a quarantined model without tools, extract only structured fields; keep the privileged agent's plan independent of email content.
4. Least-privilege scopes for document access, logging and anomaly alerts on outbound data volume.

Follow-up: Why not just tell the model to ignore instructions in emails? Injection is precisely overriding instructions; the defence must be outside the model.

Answer: tightly coupled edits by agents with partial context will conflict; prefer one agent owning the change, with sub-agents for read-only exploration

1. Each sub-agent makes assumptions (interfaces, names, data formats) the others can't see; merging produces inconsistencies (Cognition's 'share context' principle).
2. Multi-agent works best for independent, read-heavy work (research, search, analysis), as in Anthropic's research system.
3. Suggest: a single agent writes the change; parallel sub-agents may explore the codebase, run tests or review, returning summaries; or define interfaces first and only then parallelise.

Follow-up: How would you measure whether the multi-agent version is better? Same task set, compare end-state success (tests pass), pass^k, wall-clock time and total tokens.