Guide · Annotation Formats

Building Training Data for LayoutLM Fine-Tuning

LayoutLM and its successors (LayoutLMv2/v3, LiLT) dominate document-understanding leaderboards — but fine-tuning them requires token-level labels with layout coordinates, which is brutal to produce by hand. Here's the data format, the pitfalls, and how to generate it synthetically.

What LayoutLM needs

Unlike plain-text transformers, LayoutLM consumes three aligned streams per token: the token text, a label (for token classification tasks), and a normalized bounding box. The standard convention normalizes coordinates to a 0–1000 integer grid regardless of page size:

{"tokens": ["Invoice", "Number:", "INV-20260114"],
 "labels": ["B-QUESTION", "I-QUESTION", "B-ANSWER"],
 "bboxes": [[84,109,152,130], [158,109,226,130], [240,109,388,130]]}

Boxes are [x0, y0, x1, y1] with x' = x / page_width × 1000. Getting this normalization wrong — using pixel coords, or normalizing against the wrong page dimensions — is the most common silent failure in LayoutLM fine-tuning: training runs, loss falls, and the model learns garbage geometry.

Why hand-labeling fails at this granularity

Word-level labeling with boxes costs several dollars per page and annotators disagree constantly on token boundaries and BIO tags. For a 10,000-page fine-tuning set, you're looking at tens of thousands of dollars and weeks of QA. Synthetic generation eliminates the labeling step entirely: the generator knows every token, its role, and its exact geometry because it placed them.

A synthetic LayoutLM pipeline

1. Generate a mixed-type document set with LayoutLM JSONL export enabled. DocSet Generator emits token/label/normalized-box JSONL per document, derived from a pre-render canonical model.
2. Degrade strategically. Generate clean and degraded twins with the same seed. Degraded pages with clean labels teach robustness; the label stays true to intended text even when the rendered word is corrupted (source-to-rendered token mappings are recorded).
3. Split deterministically. Use type-balanced train/validation/test splits with a fixed split seed so results are reproducible.
4. Fine-tune with HuggingFace transformers — the JSONL loads directly into a token-classification dataset.

Coverage checklist

• Tables with merged cells (row/column spans) — where layout models earn their keep.
• Multi-column layouts, headers/footers, long documents.
• Key/value-dense types: invoices, forms, government documents, certificates.
• Degradation spectrum from clean to heavily corrupted.

Frequently asked questions

Does this work for LayoutLMv3 and LiLT too?

Yes. The token/label/normalized-box structure is shared across the family. LayoutLMv3 additionally consumes the page image — DocSet Generator exports clean page images alongside the JSONL for exactly this.

How are word→line→block relationships handled?

DocSet Generator's annotation schema records explicit line and layout-block boxes with word→line→block parent relationships, plus table/row/cell records — useful for models and post-processing that operate above token level.