Guide · OCR Testing

Test Documents for Benchmarking OCR Services

Published OCR accuracy numbers — 92% here, 96% there — don't transfer to your documents. The only benchmark that matters is one run on a document mix that looks like yours, scored against known ground truth. Here's how to build that test set without uploading a single real document to anyone's cloud.

Why you can't benchmark with real documents

Sending production documents to AWS Textract, Azure Document Intelligence, or Google Document AI for a bake-off means transmitting customer PII to a third party — which triggers exactly the compliance review you were hoping to defer until after picking a vendor. Public datasets dodge the PII problem but fail differently: vendors have tuned against them, and they don't match your document mix.

Synthetic documents solve both: realistic structure and degradation, zero real entities, safe to send to any API.

What the test set needs

A quality gradient, not a cliff

Run the same documents at multiple degradation levels (e.g., 0%, 25%, 50%, 75%) and plot accuracy vs. degradation per vendor. Vendors that tie on clean pages separate dramatically under corruption — that curve, not the clean-page score, predicts production behavior.

The corrupt-text-layer trap

Include PDFs that look clean but carry a corrupted embedded text layer. Tools that shortcut by reading embedded text return garbage on these; tools that genuinely OCR the pixels are unaffected. This single test distinguishes real OCR from PDF text extraction — and many commercial tools fail it.

Ground truth you can score against

Every test document needs machine-readable intended text. DocSet Generator writes manifest.jsonl plus per-document ground-truth sidecars, so scoring is a script (edit distance per page, field-level accuracy) instead of a manual review.

Type coverage

Letters, invoices, forms, tables, receipts, court documents, faxes — vendors' relative strengths vary sharply by type. A 33-type mixed batch reveals per-type winners a single-type test hides.

A reproducible benchmark protocol

# Same seed at four quality levels = identical content, controlled degradation
docsetgen --count 330 --seed 7 --quality 100
docsetgen --count 330 --seed 7 --quality 75
docsetgen --count 330 --seed 7 --quality 50
docsetgen --count 330 --seed 7 --quality 25
# Submit each batch to each vendor; score against manifest.jsonl

Seeded generation makes the benchmark rerunnable — when a vendor ships a model update, regenerate the identical corpus and measure the delta.

Frequently asked questions

Is it safe to upload synthetic documents to cloud OCR services?

Yes. Every name, address, SSN, EIN, and account number is fabricated (with structurally invalid SSNs, reserved 555-01xx phone numbers, and .example domains), so there is no PII exposure regardless of the vendor's data handling.

Can I benchmark Tesseract and other local OCR with the same set?

Yes — the same ground-truth manifest scores any engine, local or cloud. Comparing local Tesseract against cloud APIs on an identical degraded corpus is a common use of the tool.

How many documents make a meaningful benchmark?

A few hundred per quality level per document type of interest is enough to see stable per-type differences. Free tiers (e.g., 500–1,000 pages) usually cover an initial bake-off.