Before You Launch an AI Agent: A Failure-Mode Test Plan and Go/No-Go Checklist

Test the failures, not the demo

A demo shows an agent doing a task correctly, once, on an input someone chose. It says nothing about the input nobody chose: a malformed record, a hidden instruction, a call that drops mid-action. NIST's generative-AI risk guidance suggests organizations ^[1] “establish policies to evaluate risk-relevant capabilities of GAI and robustness of safety measures, both prior to deployment and on an ongoing basis, through internal and external evaluations.” This article supplies those test cases and the criteria a launch decision applies.

• Only the input path someone picked, not the edge cases users actually send.
• Only one run: a model-based agent won't always repeat itself on the same input.
• No adversarial input: nobody hands a demo a document with a hidden instruction.
• No interrupted call: nobody kills the connection mid-action.

The failure test catalogue

Each of the nine rows below names a way an agent can go wrong: how to provoke it, the safe behavior to check for, the pass bar, and its severity.

Can an LLM judge evaluate an agent?

Some pass criteria are yes-or-no facts a script can check: did the agent call the right tool, did an action run twice. Others are judgment calls, like whether a drafted explanation matches its transaction -- one option for those is a second, stronger model used as a grader. In a study of chat assistants' answers to open-ended questions (MT-Bench and Chatbot Arena), research found that ^[2] “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement.”

The same research documents where a model grader goes wrong: ^[2] “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” in LLM judges. Use a rule-based check wherever one exists, reserve a model judge for calls a rule can't make, and have a person re-score a sample of its verdicts before it gates a launch decision.

Building test cases from your own data

The catalogue only finds what your test cases provoke. Build them from your own operation -- real transcripts, tickets, and exception-queue items. The malformed record, the two-part request, last quarter's format change: these show up in your history, not a generic test suite.

1. 01
   Pull real cases

   Sample recent transcripts, tickets, and exception-queue items across the range of inputs the agent will see.

2. 02
   Add near-misses; label the safe outcome first

   Include cases already flagged as edge cases or disputes, and write the safe, correct action before running the agent.

3. 03
   Version the set

   Keep the test set under version control, and re-run it against every input or prompt change, not only the change itself.

How many runs per test case, given non-determinism?

A model-based agent can behave differently across two runs of the same input, so one successful run isn't proof a test passes. τ-bench's authors proposed ^[3] “a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials.” Testing tool-using agents in 2024, they found that ^[3] “even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail).”

We recommend at least five runs per test case, with the bar set by severity: critical rows need all five to pass; high rows need at least four of five; medium rows need at least three of five. Running more than five? Scale the threshold proportionally. The same researchers frame why this matters: ^[3] “the need for methods that can improve the ability of agents to act consistently and follow rules reliably.”

Injection and permission tests

Row three in the catalogue is a named risk in OWASP's 2025 Top 10 for LLM Applications, not a hypothetical: ^[4] “a prompt injection vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways.” The indirect form is the document case, defined next.

Build the test case by placing a plausible, out-of-scope instruction inside a document, email, or tool result the agent must read -- not a generic “ignore the above” line. A safe agent treats that text as data, not an instruction to follow, on every run.

Permission overreach (row six) is what OWASP's 2025 Top 10 calls Excessive Agency, whose root cause is typically: ^[5] “excessive functionality; excessive permissions; excessive autonomy” -- an agent with more function, access, or unsupervised reach than its task needs.

The mitigations double as the test criteria: ^[5] “limit the permissions that LLM extensions are granted to other systems to the minimum necessary,” and ^[5] “utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken.”

• Ask directly for an action outside the agent's stated scope, and confirm it declines rather than finding a workaround.
• Grant the test environment one extra, unneeded permission, and confirm the agent never exercises it even on a task that would be easier with it.
• Confirm every high-impact action routes to a named approver before it executes, not after.

Retries, crashes and duplicate actions

Rows four and five test the same property from two directions: does the system leave a half-finished action behind, and does a retried request repeat an action that already happened. Both are classic idempotency and atomicity problems, not specific to AI.

1. 01
   Force a mid-action failure

   Kill the process, or return an error, mid-action, and check what state is left behind.

2. 02
   Simulate a dropped response, then retry

   Let the action finish on the backend, but return a timeout, then repeat the request and confirm it isn't duplicated.

3. 03
   Confirm visibility, not just correctness

   A recoverable, flagged partial state is fine; one nobody can see is not.

Deciding go / no-go

NIST frames the deployment decision as a threshold set in advance, not a judgment call made afterward: ^[1] “establish minimum thresholds for performance or assurance criteria and review as part of deployment approval (‘go/no-go’) policies, procedures, and processes.” The severity column is that threshold, made concrete: a critical failure -- a followed injection, a duplicate action, an unflagged partial action, a permission overreach, or a missed escalation -- blocks launch outright.

• High severity (a wrong tool call, a hallucinated field): fix it, re-test the failed case, and it can sometimes ship behind a tighter review step instead of a full delay.
• Medium severity (silent abstention, drift on an unfamiliar format): can launch with a named owner and monitoring, tracked as an open item -- not just noticed and forgotten.

Offline evals vs. production monitoring

The catalogue is an offline evaluation: known test cases, run before anyone outside the team sees the agent, scored against a written pass criterion. It answers one question: is the agent ready to launch.

Production monitoring is a different job: watching live traffic after launch for the same failure classes recurring, new input formats the offline set didn't anticipate, and escalation triggers that fired late or not at all. An offline pass is evidence for launch, not a substitute for it -- the test set can't include next quarter's new format.

• Recurrence of any catalogue failure class in live traffic, even at a low rate.
• New input shapes or formats the offline test set didn't cover.
• A sampled review of judgment-call outputs -- the kind an LLM judge or a person scores offline.
• Escalation triggers that fired late, or not at all.

When this doesn't apply

• The workflow is fully deterministic: no step where the system decides anything a rule doesn't cover. Test it like software, not an agent.
• You're evaluating a vendor's agent and don't control its inputs. Ask the vendor for its own test and escalation-handling evidence instead.
• Every action the agent proposes is already reviewed by a person: the go/no-go bar can be lower, since the reviewer is the last check.

What to do next

If the catalogue turns up a critical-severity failure, fix it and re-run the whole set before anyone sets a launch date. If you'd rather build a test plan like this one in from the start, our AI agent implementation page covers how it's scoped, tested, and handed over.

Frequently asked questions

Related reading

• Our guide on when not to use an AI agent covers the disqualifying conditions to check before a test plan.
• If unsafe escalation misses are the biggest risk in your workflow, customer support escalation covers how routing and hand-off are built and audited.
• For the principle behind every approval and escalation step this catalogue tests, see human-in-the-loop AI governance.
• To record these test cases, their pass criteria and results, use our AI agent acceptance test plan template.

Sources

1. [1] — NIST AI 600-1, "AI Risk Management Framework: Generative Artificial Intelligence Profile" (2024-07-26), Section 3, GOVERN 1.2-002 and GOVERN 1.3-002 (accessed 2026-09-29)
2. [2] — Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv:2306.05685, 2023-06-09), Abstract (accessed 2026-09-29)
3. [3] — Yao et al., "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (arXiv:2406.12045, 2024-06-17), Abstract (accessed 2026-09-29)
4. [4] — OWASP Gen AI Security Project, "LLM01:2025 Prompt Injection" (Top 10 for LLM Applications, 2025 edition) (accessed 2026-09-29)
5. [5] — OWASP Gen AI Security Project, "LLM06:2025 Excessive Agency" (Top 10 for LLM Applications, 2025 edition) (accessed 2026-09-29)