A vendor demo of an AI invoice-review system is built to look good. The question a claims VP or CFO needs answered is different: does it find more confirmed problems than your current process, without burying reviewers in false alarms? A pilot only answers that if it's built to be checked, not just to look promising.
At a glance
- Run the pilot retrospectively, on invoices your team has already paid and already knows the truth about.
- Stratify the sample by vendor and dollar amount, and blind the reviewer to which lines the system flagged.
- Agree what counts as a confirmed finding, and set success criteria, before the pilot starts, not after.
- Track confirmed findings per 100 invoices, dollars confirmed, the false-positive rate, and reviewer minutes, not just what got flagged.
Why most pilots prove nothing
A pilot can prove nothing for reasons that have nothing to do with the model. If the vendor selects the sample, it can end up full of the invoices the system was tuned on. If the reviewer sees the system's flag before forming an opinion, the pilot ends up measuring agreement with the tool rather than accuracy. And if what counts as a “finding” is left loose at the start, it can get redefined once the running total is in.
None of this means invoice leakage isn't worth catching. This article assumes you already agree it is, and focuses on the narrower problem: a pilot whose result you can defend to whoever signs off next.
The pilot protocol
The table below is the design at a glance; the steps that follow turn it into a sequence.
| Design decision | What we recommend | What breaks it |
|---|---|---|
| Sample selection | A closed, already-paid period covering every line of business the system will eventually touch, with the invoice list frozen before anyone reviews a result. | A sample the operations team hand-picked, or one limited to vendors already suspected of problems. |
| Stratification | Split the sample by vendor and by dollar band, so a few large accounts can't dominate the count and small vendors aren't invisible. | One pooled sample where a handful of high-volume vendors decide the whole result. |
| Blinding | Route system-flagged lines and randomly selected comparison lines to the same reviewer queue, with no marker showing which is which. | The reviewer sees the system's flag first, so their judgment blends with the tool's instead of testing it. |
| Ground truth | Write the rule for a confirmed finding before review starts, and name who can overturn a reviewer's call. | The definition of a confirmed finding shifts once the running total looks disappointing. |
| Metrics and success criteria | Confirmed findings per 100 invoices, dollars confirmed, the false-positive rate (the false discovery rate, or flags rejected over all flags raised), and reviewer minutes, each measured against a threshold set in advance. | Reporting only what was flagged, or setting the bar for success after the results are already in. |
Run it as a sequence, with a named owner for each step:
- Scope and freeze the sample
Pick a closed, already-paid period covering every line of business the system will eventually cover. Freeze the list before anyone sees a result; don't add or drop invoices once a stratum's shape is known.
- Stratify by vendor and amount
Split the frozen sample into strata by vendor and by dollar band, so no single vendor or invoice size decides the outcome.
- Assign blinded review
Route flagged lines and randomly selected comparison lines to the same reviewer queue, with nothing showing which is which.
- Write the ground-truth rule
Agree, in writing, exactly what makes a finding confirmed and who can overturn a reviewer's call, before review starts.
- Run the review and log everything
Have reviewers record the decision, the reason, and the minutes spent on every line, flagged and clean, not just the ones that turn something up.
- Score against the pre-agreed criteria
Compute the metrics against the thresholds set before the pilot started, not against whatever the numbers happen to show.
- Reconcile and decide
Tie confirmed findings back to what was actually paid so the total can be checked line by line, then bring that reconciled result, not the vendor's pitch, into the go/no-go decision.
Choosing the sample
Stratified sampling is a divide-and-conquer strategy: the population is split into groups, or strata, chosen so that similar cases sit together, and a second method, usually simple random sampling, is applied within each stratum [1]. For an invoice pilot, group by vendor and by dollar amount: error patterns can differ by vendor, and the money at stake scales with amount.
Pick a period you've already closed the books on, and test it against two conditions rather than assuming a standard length: long enough to include every line of business the system will eventually cover, and short enough that no fee schedule or billing guideline changed while it ran. Freeze the list before review, and resist adding or dropping invoices once you dislike how a stratum looks.
There's no single right sample size; be wary of anyone who quotes one without asking about your volume and variance first. Give each stratum enough invoices that one unusual case can't swing the result, and review the highest-dollar stratum in full, since that's where a missed finding costs most.
Defining a confirmed finding before you start
Ground truth here doesn't mean an external authority's ruling. It means a small set of facts, agreed in advance, that let two reviewers reach the same conclusion from the same evidence. A finding is confirmed only when it meets that written rule, not when it merely gets flagged.
- The amount in question reconciles to a policy term, fee schedule, or billing guideline that was in force on the date of service, not the current version.
- A second reviewer, given the same invoice and the same reference material, reaches the same conclusion independently.
- The finding is written down with the specific rule it failed and the evidence for it, before any dollar total is calculated.
- A rejected flag is recorded too, with the reason, because it's evidence about the rule that produced it, not a result to discard.
Blinding matters most right here. In experimental design, a study is blind when participants don't know which treatment they received, and double-blind when the researchers working with them don't know either, because researchers can unintentionally bias a study [2]. The pilot applies the same idea to the reviewer: one who already knows the system flagged a line can be nudged toward confirming it.
Mixing in unflagged comparison lines, chosen at random from the same strata, is what turns “the system says this is wrong” into an independently reached conclusion.
Metrics and success criteria
Four numbers carry the pilot: confirmed findings per 100 invoices reviewed, dollars confirmed, the false-positive rate — measured here as the share of flags a reviewer rejects, which statisticians call the false discovery rate — and reviewer minutes per invoice, tracked separately for flagged and clean lines. Track all four together; any one alone can be misleading.
| Metric | What it measures | Why it alone isn't enough |
|---|---|---|
| Confirmed findings per 100 invoices | The rate at which the sample turns up findings that pass the ground-truth rule | Says nothing about the false-positive cost of getting there |
| Dollars confirmed | The amount tied to confirmed findings, net of anything a reviewer later reverses | Easy to inflate by counting flagged dollars instead of confirmed ones |
| False-positive rate (false discovery rate) | Flags a reviewer rejects, divided by all flags raised — one minus precision | A system tuned toward zero false positives can also miss real findings |
| Reviewer minutes per invoice | Time spent per invoice, flagged and clean lines tracked separately | A system that finds more but takes longer may not be a net gain |
Success criteria belong in the same document as the sample and blinding rules, signed before review starts by whoever will act on the result: the minimum confirmed-finding rate that justifies moving forward, the maximum false-positive rate (false discovery rate) the review team can absorb, and the reviewer-minutes the system has to beat. Set thresholds after the results are in, and you're no longer running a pilot; you're writing a press release.
Verifying claimed savings follows the same discipline: reconcile findings against what was paid, and report that reconciled figure instead of the total flagged.
What invalidates a pilot
Some pilot results simply can't be trusted, whatever number they produce. Watch for any of the following:
- The sample was chosen, adjusted, or trimmed by anyone with a stake in the pilot's result.
- The definition of a confirmed finding changed after review started, or was never written down.
- The reviewer knew which lines the system flagged before scoring them.
- The comparison is against a baseline, such as what reviewers currently catch, that was never actually measured the same way.
- Findings are reported as recoveries. A confirmed finding tells you what the system would have caught on invoices already paid; actually recovering funds depends on factors the pilot doesn't test, such as your vendor agreement and how much time has passed since payment.
When a pilot isn't the next step
A retrospective pilot is the wrong first move in a few cases. If your invoice volume is low enough that your team already reviews every invoice by hand without falling behind, a pilot mostly proves what you already know. If policy terms and fee schedules aren't yet agreed on, fix that first, since a pilot can only test rules that exist. And if no one can blind-review the sample in a reasonable window, the result won't be trustworthy, whatever the system's performance.
Turning the result into a decision
A pilot produces one thing: a confirmed-finding rate, a false-positive rate, and a reviewer-minutes figure, measured against the thresholds you set before you started. If the result clears them, the next step is scoping the production system, not running another pilot.
If invoice review is one of several automation candidates competing for the same budget, rank it against the others with the same rigor first.
Frequently asked questions
How do I design an invoice audit pilot?
Run it retrospectively, on invoices your team already paid and already knows the truth about. Pull a stratified sample across the lines of business and vendors the system will eventually cover, blind the reviewer to which lines it flagged, agree on what counts as a confirmed finding, and set success criteria before you see a single result.
What sample size do I need for a bill-review pilot?
There's no universal number: it depends on how many invoices you process and how much variance there is between vendors. A practical approach is to stratify by vendor and dollar band, then set each stratum's size large enough that one vendor's swing can't move the whole result, and review the highest-dollar stratum in full.
Should the pilot be blind to the incumbent reviewer?
Yes. If the reviewer knows which lines the system flagged, their judgment blends with the tool's, and the pilot measures agreement, not accuracy. A cleaner design mixes system-flagged lines into a review queue that also includes randomly selected lines, so the reviewer can't tell which is which, and scores every line the same way.
What success criteria should be agreed up front?
Before the pilot starts, in writing: the minimum rate of confirmed findings that would justify moving forward, the maximum false-positive rate the review team can absorb, and the reviewer-minutes the system has to beat per invoice. Setting these after the results are in lets the numbers get reinterpreted to fit whatever the pilot produced.
How long should an invoice audit pilot run?
Long enough to cover a full billing and payment cycle for every line of business in the sample, plus the time reviewers need to work the queue without rushing. There's no fixed length: the right duration depends on your volume, your review capacity, and how many cycles you need before a result stops looking like noise.
How do you verify claimed savings from a pilot?
Don't take the pilot's own flagged total as the answer. Reconcile the sample against what was actually paid, so nothing is silently missed; have a second reviewer confirm findings independently; and separate dollars a reviewer confirmed from dollars merely flagged, since some flags get reversed once a vendor responds. Only the confirmed, reconciled figure belongs in a business case.
Related reading
- For background on why this money goes uncaught in the first place, see invoice leakage patterns.
- Our solution page on insurance invoice analysis covers the production system a pilot like this is designed to test.
- If you're building the business case in parallel, how to calculate AI ROI before you build covers modeling error cost and recovered value before you build.
- The 360° AI Blueprint ranks invoice review against other automation candidates; AI agent implementation covers building the production system once a pilot clears.
More in Insurance operations
What AI Claims Automation Costs: A Cost and ROI Model for Carriers, TPAs and MGAs
AI claims automation cost depends on scope, not a price list. Get the cost drivers, an LAE baseline worksheet, and a payback method for carriers, TPAs and MGAs.
Insurance CompaniesBordereaux Automation for MGAs and Carriers: Ingestion, Validation and Reconciliation Checks
Bordereaux automation for MGAs and carriers: a validation-and-reconciliation check matrix with data source, rule, tolerance, owner and action for every check.
Insurance CompaniesInsurance Commission Reconciliation Automation: AMS Module, Point Tool or Custom AI Agent?
Choosing insurance commission reconciliation automation: a decision matrix for AMS module vs. point tool vs. custom agent, and a discrepancy routing table.
Insurance Companies