At a glance
- A pilot is a decision instrument: scope an AI automation pilot so a named person can make a build/kill/redesign call from its result, not so it can look impressive.
- The scoping worksheet sets eleven fields to agree in writing before the pilot starts, from the decision it informs to its exit criteria.
- Set the baseline and the success and failure thresholds before you have a single result to defend against them.
- A worked, synthetic example shows the worksheet filled in; a list of anti-patterns shows how pilots quietly stop meaning anything.
A pilot is a decision instrument
A pilot without a decision attached to it isn't a pilot — it's a demo. Before you scope anything, write down what decision the result will change, and who makes that call. A real pilot is scoped, run, and measured so that when it ends, one named person can look at one specific number and decide: build it, kill it, or redesign it.
That discipline is uncomfortable: it forces commitments before results exist — a baseline, a success/failure threshold, and a boundary narrow enough to run fast, honest enough to mean something. The worksheet below makes those commitments explicit.
The scoping worksheet
Fill in every row below before you touch the workflow. Each is a commitment, agreed in writing by the people who'll judge the result — not a placeholder to fill in later. The right-hand column shows the kind of vague answer that turns a pilot into an argument.
| Field | What a good answer looks like | A common bad answer |
|---|---|---|
| Decision the pilot informs | Names the exact decision waiting on the result — fund it, kill it, or choose between two approaches — and who makes that call. | "See if AI can help with X." No decision is attached, so any result leads nowhere. |
| Workflow boundary (in/out) | Lists the exact steps in scope, the steps excluded, and where exceptions route instead of being attempted. | "The whole invoice process" or "scheduling." Anything that isn't a short, specific list isn't a boundary. |
| Volume and sample | States how many cases will run, over what defined sample or period, and why that slice was chosen. | "We'll run it for a while and see." Without a defined sample, the result can't be trusted or explained. |
| Data access needed | Lists every system the pilot must read or write, the access level, and who approves it before day one. | Assuming access will "probably be fine" and finding the integration gap in week two. |
| Baseline metric | The manual process's current, measured performance on the same metric the pilot will be judged on. | No baseline, or a guess ("it's usually about a day") standing in for a measurement. |
| Success threshold | A specific number or range that counts as success, set before results exist and signed by the decision-maker. | Deciding after the results arrive whether they were "good enough." |
| Failure threshold | The specific result that ends the pilot, including any failure mode that means stop immediately. | No failure threshold, so a weak pilot just runs longer instead of ending. |
| Human review design | Specifies what a person checks, at what rate, and what they're allowed to override, with a record kept of every override. | "A person will review it," with no rate, no criteria, and no record. |
| Duration driver | The event that ends the pilot — the sample size reached, or a decision date the sponsor needs — not a fixed calendar length. | An open-ended "we'll know it when we see it," with no defined end condition. |
| Owner | One named person, not a team, accountable for the result and for making the go/kill/redesign call. | "The project team decides together," so no one is accountable for the call. |
| Exit criteria | States what happens after the pilot for each outcome — proceed, stop, or redesign and re-test — agreed before it starts. | Deciding what "success" earns only after seeing whether the result looks good. |
Setting success and failure thresholds before you start
The instinct is to run the pilot first and judge the numbers afterward. We recommend the opposite: agree the metric, the baseline, and the threshold before a single case runs, and measure it under conditions that resemble real operation, not whatever conditions make the demo look best.
NIST's AI Risk Management Framework treats measurement as its own function for AI systems generally: it "employs quantitative, qualitative, or mixed-method tools, techniques, and methodologies to analyze, assess, benchmark, and monitor AI risk and related impacts"[1], and holds that "AI systems should be tested before their deployment and regularly while in operation"[1].
The GOV.UK Service Manual sets the order the same way: in beta, teams "start to gather data on how successful the service is based on the success metrics you identified in alpha"[2]. The metrics come first, from the earlier phase. We recommend the same discipline for a pilot: a workflow with no measured baseline has a gap the pilot can't retroactively supply.
Choosing the boundary
The GOV.UK Service Manual's guidance for the alpha phase (the prototyping stage before a public beta) makes the same boundary decision explicit: teams "do not have to prototype the user's entire wider journey," and "often it makes sense just to focus on the areas you think will be most challenging"[3]. An automation pilot works the same way: scope it to the step that actually tests your open question, not the whole workflow end to end.
That guidance frames the narrower scope as deliberate: teams should "build things that are just complex enough to let you test different ideas, not production quality code," and alpha services "should not be available for the public to use"[3]. A pilot's boundary should be just as deliberate: read-only access where you can, and no live customer-facing exposure until the scope has been tested.
That's also why a bounded pilot ends in a decision, not a fade into production. The same guidance says alpha is finished "when you've got a prototype that's substantial enough to help you make a decision about whether to move on to the beta phase or not"[3]. Build that same explicit decision point into your pilot's exit criteria, rather than letting the boundary quietly expand.
- Exception cases that would need new logic — route them to a human and log them, rather than letting them expand the pilot's scope.
- Systems the pilot doesn't need write access to — we recommend starting read-only and asking for write access only when the pilot needs to act.
- Workflow variants that make up a small share of volume — pilot the common path first, and treat rare variants as a separate decision.
- Steps whose current performance is already acceptable: a pilot should target where the workflow actually hurts, not everywhere it touches.
Illustrative example: a filled worksheet
| Field | Filled-in answer for this synthetic pilot |
|---|---|
| Decision the pilot informs | Whether to fund a production coding agent for the shared-services center, or shelve the idea. |
| Workflow boundary (in/out) | In scope: standard, non-PO invoices under a set dollar threshold from the five highest-volume vendors. Out of scope: PO-matched invoices and disputes. |
| Volume and sample | Every qualifying invoice, run in the live queue rather than a held-out test set, until 1,200 have been coded, matching production's exception mix. |
| Data access needed | Read access to the invoice queue and vendor master; write access limited to a coding-suggestion field the reviewer approves. No posting access. |
| Baseline metric | Manual coding accuracy on the same five vendors, from the controller's existing 500-invoice quality-audit sample, logged by hand. |
| Success threshold | Coding accuracy at or above the measured baseline, with reviewer time per invoice at or below the manual average. |
| Failure threshold | Accuracy meaningfully below baseline on two consecutive weekly batches, or an override rate above one in five invoices — either stops the pilot. |
| Human review design | Every coded invoice is checked by the coder who'd have coded it manually; overrides are logged and reviewed weekly by the pilot owner. |
| Duration driver | Ends when the 1,200-invoice sample is coded, not on a calendar date — sooner if the five vendors' volume runs high that period. |
| Owner | The accounts-payable shared-services manager, who reports the result to the controller and makes the go/no-go call. |
| Exit criteria | Meets threshold: expand to the next ten vendors. Fails: return to manual coding and document why. Mixed: redesign the boundary and re-test before deciding. |
Pilot anti-patterns: how pilots stop producing a decision
The avoidable ways a pilot ends without a usable decision are in the scope, not the model. Watch for the handful of mistakes that turn a well-designed pilot into evidence nobody trusts:
- Moving the goalposts mid-pilot: changing the threshold, sample, or review criteria after early results come in turns a test into a foregone conclusion.
- No baseline: comparing output to a vague sense of how things used to go, instead of a measured number, makes any result indefensible.
- The demo pilot: hand-picking easy cases that don't represent the real exception mix inflates the result and hides the work production will actually face.
- Unbounded review: "a person checks it," with no defined rate or record, means nobody can tell whether the system or the reviewer did the work.
- No owner: when the whole team decides together, a mediocre pilot drifts into production because no one is accountable for calling it.
- Treating completion as success: finishing end to end without errors still has to clear the threshold — a clean run and a passing result are separate checks.
When a pilot isn't the right instrument
A pilot is the wrong tool in a few common situations:
- The workflow is fully deterministic: a fixed rule can compute the correct output, so build or configure that system instead of piloting AI accuracy.
- There's no baseline and no time to build one. A pilot can't retroactively supply a measurement that was never taken; fix the measurement gap first.
- Volume is too low for a meaningful sample in a reasonable window. Consider expert review of a small set instead of a volume-based threshold.
- An existing tool already clears the bar, so a custom pilot would just answer a question nobody's asking.
Frequently asked questions
How do you scope an AI pilot?
Name the single decision the pilot's result will inform, then agree its boundary, baseline, and success and failure thresholds in writing before it starts. Use a worksheet, not a project plan: list what's in and out of scope, the data access needed, who reviews the output, and who owns the go/kill/redesign call at the end.
How long should a pilot last?
This article doesn't set a duration, because the right length depends on your workflow's volume and risk. Drive the end date from an event instead of a calendar: the sample size needed for a trustworthy result, or a decision date the sponsor already needs, whichever comes first.
What success criteria should it have?
A specific, numeric threshold set against a measured baseline, agreed before you have a single result to defend it with. Pair it with a failure threshold and any failure mode that stops the pilot immediately, so a weak result ends the test instead of extending it indefinitely.
Why do most AI pilots fail?
This article doesn't estimate how many pilots fail. The avoidable reasons are specific: no decision was attached to the result, no baseline was measured, or the threshold was set after the numbers came in. Fix those three and a pilot produces evidence, not a demo.
What should a pilot scope statement include?
All eleven worksheet fields, filled in and signed off before day one: the decision it informs, the workflow boundary, volume and sample, data access needed, baseline metric, success and failure thresholds, human review design, duration driver, owner, and exit criteria.
Related reading
- Before you scope a pilot, size the opportunity: How to Calculate AI ROI Before You Build prices the status quo so you know whether the workflow is worth piloting at all.
- Haven't run the self-check yet? AI readiness assessment: 12 questions covers the sponsorship and data-access gaps that otherwise show up as pilot failures.
- A pilot that clears its threshold still has to get built: AI agent implementation describes how a system moves from pilot to production, tested and handed over.
- The 360° AI Blueprint ranks AI and automation candidates across your operation by ROI, cost, and time-to-value, so you pilot the highest-value workflow, not just the first one someone suggested.
Sources
- National Institute of Standards and Technology, AI RMF Core, Section 5.3 (Measure) — AI Risk Management Framework 1.0 (2023) (accessed 2026-09-28)
- Government Digital Service, How the beta phase works — GOV.UK Service Manual (accessed 2026-09-28)
- Government Digital Service, How the alpha phase works — GOV.UK Service Manual (accessed 2026-09-28)
More in AI strategy
AI Agents vs. RPA for Hotel Back-Office Operations: When Bots Break and What Replaces Them
AI agents vs RPA for hotel back-office operations: a decision table for OTA statement matching, folio audit and multi-property AP, plus how to phase bots.
Hotel Management CompaniesWhat a 360 AI Blueprint Costs for a Food & Beverage Manufacturing Back Office
What drives food and beverage manufacturing AI cost in the back office: a table of plants, co-packers, deductions and traceability scope, plus how to budget.
Food and Beverage ManufacturersAI Implementation Cost for Multi-Rooftop Dealer Groups: What Changes When You Add a Store
AI implementation cost for multi-rooftop dealer groups: a driver table showing how DMS count, OEM portals, build type and upkeep move cost per added store.
Auto Dealerships