False Positives in Insurance Invoice Review: Precision, Recall and Reviewer Workload

Every flag costs reviewer time

A flagging system's value shows up twice: in what it catches, and in what it costs to review. Every line a system flags becomes a task for a person, whether the flag turns out to be a real issue or not. A system tuned to flag loosely can look better on paper while quietly requiring more reviewer hours than the extra findings are worth.

A high false-positive rate also carries a cost that's harder to put on a spreadsheet: reviewer trust. When most flags turn out to be nothing, a noisy queue risks training reviewers to expect that — moving faster through it, skimming, batch-clearing, or waiting for a supervisor to push back before slowing down. We treat that as a risk to plan for, not a measured outcome: if it happens, it's also how a real finding gets missed.

• Reviewer time spent on flags that turn out fine — the same time a true finding would have needed.
• Slower turnaround on invoices that were clean all along, because they're stuck behind the noise.
• A risk, not a certainty: a queue reviewers learn to distrust is also one where they're more likely to miss the flags that matter.

Precision, recall and base rate in plain terms

Three numbers, not two, decide how a flagging system's threshold turns into work. Precision and recall describe how good the flags are once you have them. Base rate describes how many real issues exist in the population before you flag anything. Keep the three separate when you choose a threshold — conflating them makes it easy to misjudge what a given setting will do to your queue.

Precision is the ability of a classifier not to label a negative line as positive, and recall is its ability to find all the positive lines ^[1]. Applied to invoice review, precision is the share of flags that are real issues, and recall is the share of real issues the system actually flags.

Base rate — also called prevalence — is the count of positive-label lines divided by the total count of lines ^[2]. A low base rate matters more than it sounds: when only a small share of lines are genuinely wrong, even a rule that almost never misfires on a single clean line can still end up with most of its flags being clean lines, simply because clean lines vastly outnumber real issues — so the rule's precision on this population ends up low.

From threshold to workload

Loosen the threshold and the system flags more lines. Some of the extra flags are real issues you'd have missed at a stricter setting, which can raise recall. Lowering the threshold also returns more results overall, and precision does not necessarily fall as recall rises — though once a threshold is already reasonable, further loosening usually costs some precision ^[3].

As a working assumption rather than a measured fact, expect a growing share of newly flagged lines to be borderline calls rather than clear issues, so the queue can grow faster than the count of findings worth a reviewer's time.

1. 01
   Flags per period

   Multiply invoice volume by the flag rate at the threshold you're testing. That's the size of the queue for the period.

2. 02
   True findings

   Multiply flags by precision. That's the flags actually worth a reduction, a request for more information, or a denial.

3. 03
   Reviewer time

   Multiply flags by the average minutes a reviewer spends per flag, then convert the total from minutes to hours. That's the demand the queue places on the review team.

4. 04
   Missed findings

   Multiply the population's real issues — volume times base rate — by one minus recall. That's what a stricter threshold leaves on the table.

None of this needs a confirmed base rate in hand to be useful. Run the same four multiplications against a range of assumptions, and you'll see which thresholds are workable long before you commit reviewer headcount to one.

Illustrative example: three thresholds

Illustrative example: the numbers below are synthetic, not benchmarks, and no real precision, recall or flag rate for insurance invoices is claimed. Assume a month of 10,000 invoice lines, of which 500 are genuinely wrong (a five percent base rate), and that a reviewer spends about six minutes on each flag. Three thresholds trade flag rate against precision differently.

The pattern is typical: loosening the threshold buys more true findings and fewer missed findings, at the cost of more reviewer hours and more lines that were never a problem. Run your own volume, base rate and minutes-per-flag through the same four steps before picking a threshold your review team has to live with.

Choosing a threshold with finance

The threshold is a staffing decision as much as a modeling one, which is why finance and the reviewer team both belong in the conversation before it's set. Reviewer hours and missed findings move in opposite directions as the threshold changes, and only finance can say which side of that trade a team's capacity and risk tolerance can absorb.

Monitoring drift after launch

A threshold tuned on last quarter's invoices doesn't stay right forever. Vendor billing patterns change, fee schedules update, and the mix of claim types shifts, all of which can move the base rate and precision even if no one touches the rule. Reviewer decisions are the signal to watch.

• Precision by rule, tracked over time, not only measured once at launch.
• Flag rate and reviewer hours, so a shift in volume or base rate doesn't look like a sudden staffing surprise.
• A sample of invoices that cleared without a flag, so a rule that's gone quiet gets checked too, not only the ones still firing.

When a rule's confirmed rate drifts, that's the signal to revisit the threshold, the rule itself, or both — using the same workload steps above rather than a fresh guess.

When this framing doesn't apply

This workload model assumes flags feed a reviewer's queue, with a human look before anything happens to a payment. It doesn't apply to a hard rule with no ambiguity, such as an invoice tied to a claim number that doesn't exist: that's a lookup, not a probabilistic flag, and it needs no precision/recall threshold. See insurance invoice analysis for how the rules themselves get built. Sizing the review team's overall capacity is a separate exercise from this one threshold's effect on it.

Frequently asked questions

Related reading

• The insurance invoice analysis solution page covers how the flagging rules themselves are built and tested.
• For the four kinds of leakage a flagging system is built to catch, see insurance invoice leakage.
• On why every flag should reach a person before anything changes, see human-in-the-loop AI governance.
• The AI agent implementation service describes how a system like this is scoped, tested, and handed over.

Sources

1. scikit-learn developers, "Precision, recall and F-measures" (User Guide) (accessed 2026-09-28)
2. scikit-learn developers, "Class likelihood ratios" (User Guide) (accessed 2026-09-28)
3. scikit-learn developers, "Precision-Recall" example (accessed 2026-09-28)