Guide · Annotation Formats

How to Build a Custom DocVQA Dataset

Document Visual Question Answering — asking natural-language questions of a page image — is how modern multimodal models are trained and evaluated on documents. The public DocVQA benchmark is fixed and finite; here's how to build your own QA data over documents you control.

The DocVQA task and format

The benchmark pairs page images with questions and grounded answers — ~50,000 questions over ~12,000 pages, scored with ANLS (normalized Levenshtein similarity). Each record ties a question to a document, page, and answer text:

{
  "questionId": 1142,
  "question": "What is the invoice total?",
  "image": "documents/invoice_0042_p1.png",
  "answers": ["$4,120.50"],
  "answer_page": 1
}

What makes DocVQA-style data expensive is grounding: the answer must actually appear in the document, and good datasets record which words on which page constitute it.

Why synthetic generation fits QA unusually well

Writing questions over real documents requires a human to read each page. But a generator that composes documents from a structured model can derive questions mechanically: it knows the invoice total is $4,120.50 and where those words sit on the page, so "What is the invoice total?" comes with a perfectly grounded answer — word references, page number and all. No annotator, no ambiguity, no transcription errors.

DocSet Generator's DocVQA export does exactly this: questions derived from recorded model fields, with answer word and page references, across 33 document types. Because generation is seeded, the same corpus is reproducible byte-for-byte.

Design guidance for custom QA sets

• Vary question targets: totals, dates, parties, table cells, headings — not just one field type per document class.
• Include degraded pages. QA over clean renders is easy mode; real deployments answer questions over scans. Seeded clean/degraded twins let you measure exactly how much degradation costs your model.
• Test the text-layer shortcut. Corrupt-text-layer documents (clean-looking page, garbage embedded text) expose multimodal systems that quietly read the PDF text layer instead of looking at the image.
• Hold out by document type, not just by page, to measure generalization to unseen layouts.

Frequently asked questions

Can synthetic DocVQA data train a model evaluated on the real benchmark?

Yes, as augmentation. Synthetic QA pairs teach grounding and layout reading at scale; a small real-data fine-tune closes the domain gap. Evaluate on the official benchmark for comparable numbers.

Are answers guaranteed to appear in the document?

Yes — questions are derived from the document model that generated the page, so every answer is grounded by construction, with word and page references recorded.