When a vendor's accuracy number comes from an internal team labeling the vendor's own documents against a definition of “correct” it chose, none of that has to match your production mix. If you want a number that describes your own workflow, you need your own ground truth dataset for document extraction: a labeled sample of your own documents, held out from anything the system was tuned on.
At a glance
- Build your own labeled sample from your own documents; a vendor's accuracy score doesn't transfer to your production mix.
- Sample by document type, source, quality, and time period so the set reflects production, not whichever batch is easiest to pull.
- Write an annotation guideline, double-annotate a subset, measure agreement, and adjudicate disagreements before trusting the labels.
- Freeze a holdout set nobody touches while tuning the system, and version the dataset the way you'd version code.
- There's no single right size: it follows how rare the fields and error types you need to catch actually are.
Why vendor accuracy numbers don't transfer
A benchmark score describes the benchmark's documents: its scan quality, layouts, and the vendor's idea of a match. Move the same model onto your intake, with different templates and scan quality, and the number no longer describes your documents. The only accuracy figure worth acting on is measured on your documents, against your definitions, by people with no stake in the result.
Can an LLM judge replace ground truth?
An LLM can score outputs quickly for triage. But a model judging another model's extraction can share its blind spots: it can accept a confident-sounding wrong answer and penalize a correct one phrased unexpectedly, and it can't confirm what the source document said. Use an LLM judge to pre-sort a queue; use a human-labeled holdout, checked against the document, to decide whether the system works.
The dataset method
Treat the dataset as an engineering deliverable. The steps below produce one: a matching sample, a shared guideline, and a holdout nobody touches while tuning.
- Define the fields and the unit of truth
List every field the system extracts, its value type, and what counts as correct, including edge cases. This becomes the field-definition table below.
- Draw the sample
Pull documents by type, source, quality tier, and time period so the sample matches production, not whichever batch is easiest to find.
- Write the annotation guideline
Turn the field definitions into instructions a new annotator could follow unaided, with worked examples of correct and incorrect labels.
- Double-annotate a subset
Have two independent annotators label the same documents, blind to each other's answers, before trusting one annotator on the rest.
- Measure agreement and adjudicate
Score the double-annotated subset with a chance-corrected agreement measure. A third reviewer adjudicates disagreements against the guideline, which is revised if a gap turns up.
- Freeze a holdout
Set aside labeled documents nobody uses for prompt or model tuning, and lock the set before development starts.
- Version and schedule a refresh
Tag the dataset with a version, log what changed, and set a refresh cadence tied to how your document mix actually changes.
Step one produces a table like the one below: one row per field, filled in before labeling starts and kept next to the guideline.
| Field | Definition | Value type | Evidence location rule | Edge-case rule (example) |
|---|---|---|---|---|
| document_type | Which pre-agreed category the document belongs to. | Enum | Document structure and content, not the filename | If more than one type could apply, use the guideline's priority order. Example: a scanned form with a purchase-order number and no signature block is labeled purchase_order. |
| effective_date | The date the document's terms take effect, not when it was signed or received. | Date | Body text or a dated clause | If only a signature date appears, or the date requires calculating from a relative phrase, mark the field “not stated” and flag for review. Example: a clause reading “effective next quarter” is labeled “not stated” and flagged. |
| total_amount | The single bottom-line amount the document asks the reader to pay or record, excluding subtotals. | Currency amount | Summary or totals block | If more than one currency appears, label it separately and flag for review. Example: a document shows a subtotal, tax, and a bolded final line; only the final line is labeled. |
Field-level accuracy vs document-level accuracy
Field-level accuracy is the share of individual fields the system got right; document-level accuracy is the share of whole documents where every field was correct. One missed field can fail the whole document, so report both. For how to weight that choice by dollar impact, see invoice extraction accuracy vs. financial accuracy.
Sampling documents that represent reality
A convenience sample — one folder, or documents a reviewer remembers being tricky — describes that folder, not production. Draw the sample along dimensions that actually vary, and record how you drew it.
| Dimension | Why it matters | How to sample it |
|---|---|---|
| Document type | Difficulty and the field set differ by type | Stratify so every type has enough documents to measure separately |
| Source or channel | Scan quality and layout vary by intake channel | Sample proportionally to each channel's real volume |
| Scan or image quality | Poor scans and photos fail differently than clean PDFs | Include a deliberate share of poor-quality documents, even if rare |
| Time period | Templates and formats change over time | Span more than one period so the set isn't tuned to one moment |
Annotation guidelines and adjudication
A guideline keeps two people from producing two different datasets. Write it before annotation starts, and treat disagreement as a sign the guideline is missing something, not noise.
| Section | Contents | Why it's there |
|---|---|---|
| Field definitions | One paragraph per field: meaning, exclusions, value type | The field-definition table, written for a person |
| Worked examples | Two or three annotated documents per field, with an edge case | Examples settle disputes faster than prose |
| Normalization and abstention rules | How to format extracted values, and when to mark a field “not stated” instead of guessing | Prevents formatting differences and invented values from counting as disagreement |
| Escalation path | Who an annotator asks when a document fits no rule | Routes new edge cases to whoever updates the guideline |
Label a meaningful minority of the sample twice, by annotators who don't see each other's answers, before trusting one annotator on the rest. Score that subset with a chance-corrected agreement measure rather than raw percent agreement, which can overestimate true agreement because it doesn't account for raters guessing [1].
Cohen's kappa is one of the most commonly used measures of agreement between raters [1]: it removes the agreement expected from chance, which matters because raters can guess on ambiguous fields and still look like they agree. Cohen's own suggested reading treats 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement [1].
The same source suggests those bands may be too lenient for health-related studies, and simplifies them: any kappa below 0.60 indicates inadequate agreement, with little confidence to place in the result [1]. Our recommended decision rule, once a field's kappa is scored, is below.
- 0.81 and above: move to single annotation for the rest of the sample, with periodic spot re-checks.
- 0.61–0.80: usable, but review the guideline behind any field scoring in this band.
- 0.41–0.60: not yet ready — revise the guideline and re-run double annotation on that field.
- 0.40 and below: stop. The field definition or examples are too unclear to trust.
Where two annotators disagree, a third person adjudicates: seeing both labels, deciding against the guideline, and recording why. A recurring disagreement is a guideline gap, not an annotator problem.
Holdouts and versioning
A holdout is the part of the sample nobody uses while building the system: not for the prompt, a threshold, or a model choice. Check it to make a decision, and it stops being a holdout.
| Split | Used for | Who sees results, and when |
|---|---|---|
| Development set | Iterating on prompts and thresholds | Anyone building the system, any time |
| Validation set | Choosing between finished configurations | The build team, once development is frozen |
| Holdout (test) set | The one reported accuracy number | Results reported once, after freeze; if it's used for a decision, retire it and draw a new holdout |
A year-old holdout describes documents that may no longer arrive. Refresh it on a schedule tied to how your document mix changes — a new vendor template or channel, not a fixed date. Version every release: what changed and which model it was last measured against, the way you'd version code.
| Record | Example | Why it matters |
|---|---|---|
| Version tag | v2026.3 | States exactly which labels a reported score is measured against |
| Change log | Added documents from a new channel; revised a field rule | Explains a score move before anyone assumes the model changed |
| Guideline version | Guideline v1.4, tied to this dataset version | A relabel under an old guideline isn't comparable to a new one |
How big is big enough
There's no single right number; be wary of anyone who gives one. Size follows field rarity, the difference you need to detect, and how many strata your sampling plan created.
| Factor | Effect on size | Practical rule |
|---|---|---|
| Field rarity | A rare field needs more total documents | Set a minimum count per field, not per total |
| Number of strata | Each type, channel, and quality tier needs its own count | Multiply the per-field minimum by your strata count |
| Detectable difference | A finer distinction needs a larger sample | Decide how small a gap changes your decision first |
| Cost of a wrong decision | High-stakes fields need a bigger, checked sample | Size the highest-stakes fields first |
Published document-AI benchmarks are a size check, not a target to copy. FUNSD, a form-understanding benchmark, comprises 199 real, fully annotated scanned forms, split 149 for training and 50 for testing [2]. Its published annotation procedure names the tool and who performed the text-detection annotations, but doesn't report an inter-annotator agreement measure [2] — a reminder to check whether a benchmark you're sizing against reports agreement before trusting its scale.
When you don't need a full ground truth set
- Volume is low enough that a person already checks every document; a sampled evaluation restates what you already know.
- The fields aren't defined yet and nobody agrees what “correct” means. Write the definitions first.
- You need a one-time sanity check before a small pilot, not a standing measurement. A single double-annotated batch may be enough.
- Documents can't leave their source system under your data-handling rules. Solve that access question first.
Frequently asked questions
How do you build a ground truth set?
Define every field and what counts as correct, draw a sample matching your production documents, write an annotation guideline with worked examples, double-annotate a subset and measure agreement, adjudicate disagreements, and freeze a holdout nobody touches while tuning the system.
How many documents do you need?
Enough that your rarest important field and smallest stratum still have a workable count, not one fixed number. Size around field rarity, the document types you measure separately, and how small a difference changes your decision, then check published benchmarks for scale, not as a target.
How do you resolve annotator disagreement?
Have two annotators label the same documents independently, score their agreement with a chance-corrected measure such as Cohen's kappa, and send every disagreement to a third person who adjudicates against the guideline. A recurring disagreement means the guideline is missing a rule, not that an annotator erred.
Field-level vs document-level accuracy?
Field-level accuracy is the share of individual labeled fields the system got right. Document-level accuracy is the share of whole documents where every field was correct. A system can do well field by field and still fail most documents, because one missed field on a multi-field document fails that document; report both.
Can an LLM judge replace ground truth?
Not for the number you report. An LLM judge is useful for fast, rough triage, but it can share the blind spots of the model it's judging and can't confirm what the source document actually said. A human-labeled holdout, checked against the document, should decide whether the system works.
How often should it be refreshed?
On a schedule tied to how your document mix changes — a new vendor template, channel, or policy change — not a fixed calendar date. Version every refresh with a change log and guideline version, so a moved score can be explained instead of assumed.
Related reading
- Our solution page on document review with human approval covers the production system a dataset like this measures.
- For a rulebook-driven document workflow in insurance, see insurance invoice analysis.
- Our article on AI document intelligence for legal and compliance covers the wider document-review workflow this dataset sits underneath.
- The 360° AI Blueprint ranks document intelligence against other automation candidates, and AI agent implementation covers how a system like this is scoped.
More in Document intelligence
340B Program Documentation: Where AI Agents Reduce Audit Risk (Not Replace Compliance Officers)
How AI agents can assemble 340B program documentation, from split-billing records to contract pharmacy files, while your compliance officer keeps each decision.
Pharmacy ChainsChargeback and Returns Dispute Document Automation for E-Commerce Brands
Chargeback and returns dispute automation for e-commerce: a step-by-step dispute-document workflow and evidence table, with submission kept with your staff.
Ecommerce OperationsJoint Interest Billing and Land/Lease Administration Document Automation for Independent Energy Operators
Joint interest billing automation for independent operators: a lease-to-distribution workflow showing what an agent drafts and what staff approve.
Utilities and Energy Companies