When I started building AIDA, the document intelligence system at Equans, I had a mental model of what I was working with. PDFs. Text documents. Machine-readable content sitting in files, waiting to be extracted and indexed.
That model was wrong in almost every specific.
The first thing I did was write a simple PDF text extractor using PyMuPDF. It worked perfectly on the first five documents I tested it on. Clean extraction, correct structure, headings preserved, tables legible. I thought the hard part was done.
Then I ran it on the actual document set.
What was actually in there
About 30% of the documents were scanned PDFs — physical documents that had been photographed or photocopied and saved as image-only PDFs. No text layer. PyMuPDF returned empty strings. The documents existed; the content was invisible to any text-based approach.
Another 20% had text layers, but those layers were either wrong (OCR artefacts from an automated scan-to-PDF process years earlier) or structured oddly — multi-column layouts where the text extraction order jumped between columns mid-sentence, tables where the cells had been extracted as a flat stream of values with no delimiters.
Some documents were combinations of both: a PDF with a text layer that was a degraded OCR pass from the 1990s, overlaid on the scan of the original printed page.
About 15% had been created from CAD software and contained technical drawings with embedded text that was part of the drawing, not searchable document content.
What I had to change
PyMuPDF got replaced by Azure Form Recognizer for the main ingestion pipeline. Form Recognizer does layout analysis — it understands that two columns of text are two columns, not one stream. It reconstructs tables. It runs OCR on image-only PDFs. It's slower and costs money per page, but it produces structured output that the rest of the pipeline can actually use.
The CAD documents got a different treatment: extracted text was flagged with a low-confidence score, and those documents were excluded from the main retrieval index. When a user queried about equipment covered in a CAD manual, the system would note the limitation explicitly rather than returning garbled results.
The most important change was to the evaluation approach. I had been testing extraction quality on a sample of "good" documents. I replaced that with a stratified sample across document types — including the difficult ones. The benchmark numbers got worse immediately and then improved as the pipeline improved. That's the right direction.
What this means more generally
The lesson isn't specific to PDFs. It's about the difference between the data you test on and the data you'll actually encounter. In almost every real-world ML project I've worked on, the hardest part was understanding what the input data actually looks like — not designing the model that processes it.
The clean, well-structured benchmark datasets that ML papers are evaluated on exist because someone cleaned them. The messy, inconsistent, partially-broken data that exists in real organisations is how the world actually is. Building systems that work on the second kind while only testing on the first kind is a way to be confidently wrong.
The PDFs were not what I expected. They were better data about the problem than any dataset I could have designed.