Week 5: Ingesting and Processing PDFs

Welcome

Ingesting and Processing PDFs

Core session 2

• Last week: StatsChat end to end
• This week: how reports become machine-readable
• Key idea: extraction quality shapes retrieval quality

Today’s aims

By the end, you should be able to explain:

• why PDF processing matters for RAG
• why PDFs are difficult for machines
• how StatsChat currently processes PDF reports
• what can go wrong during extraction
• how better ingestion could support better retrieval

Why this matters for your organisation

Many of you may use, adapt or help maintain a StatsChat-like system for your own organisation.

• What publications or data sources would you want users to query?
• Are they PDFs, spreadsheets, web pages, APIs, or a mixture?
• What would be risky if the structure was lost?

Quick question 1

Which source would you prioritise first for a StatsChat-like system?

A. A frequently used official report with clear user demand
B. A very old scanned report with poor image quality
C. A visually impressive factsheet with many charts but little text
D. Any PDF, because all PDFs are equally easy to process

The big idea

A PDF is not the same as evidence

Quick question 2

Why are PDFs often difficult for RAG systems?

A. PDFs are always scanned images
B. PDFs preserve visual layout, but not always semantic structure
C. PDFs cannot contain text
D. PDFs are too large for Python to open

From reports to evidence

The simple version

PDF → text → JSON

Demo: what we expect to see

A simple PyMuPDF workflow

import fitz
import json

pdf = fitz.open("report.pdf")

pages = []
for page_number, page in enumerate(pdf, start=1):
    pages.append({
        "page_number": page_number,
        "page_text": page.get_text()
    })

with open("report.json", "w", encoding="utf-8") as f:
    json.dump({"pages": pages}, f, indent=2)

How StatsChat currently processes PDFs

Current output: what gets preserved

A simplified StatsChat JSON structure

{
  "metadata": {
    "title": "Consumer Price Indices and Inflation Rates",
    "release_date": "2026-03-31",
    "theme": "Prices",
    "url": "https://..."
  },
  "pages": [
    {
      "page_number": 3,
      "page_url": "https://...pdf#page=3",
      "page_text": "Annual consumer price inflation..."
    }
  ]
}

The example PDFs we will use

Quick question 3

If the numbers are extracted but the visual grouping is lost, what is the biggest risk?

A. The PDF has not been downloaded
B. The system may mix up which labels, groups or units belong to which values
C. The LLM will refuse to answer every question
D. There is no risk if all the numbers are present

Three kinds of PDF problem

What can go wrong?

The table problem

Example: table structure lost

Human-readable table → machine-confusing string

A table might visually mean:

But extracted text might look more like:

YEARPER CENT20232024202220212020012345678 97.77.74.56.15.4

Lost structure is a retrieval problem

Quick question 4

A correct table is extracted into JSON, but StatsChat answers from the wrong page. Where is the problem most likely to be?

A. PDF download only
B. PDF extraction only
C. Chunking, retrieval or ranking
D. The user question cannot be answered by RAG

Another lesson: report pages, not just PDFs

A report page can have several useful files

Report page
├── main PDF
├── popular version
├── highlights or presentation
├── speech or launch material
└── spreadsheet tables, where available

Current improvements in StatsChat

The current branch has already improved the ingestion pipeline.

• downloads all PDF links from a report page, not only the first one
• records source URLs through url_dict.json
• handles page-level extraction errors more safely
• can fall back to pdfplumber for selected difficult pages/documents
• includes tests for downloader, conversion and page integrity

Diagnosing failures

When an answer is wrong, where did it go wrong?

From current to better: example 1

Treat report pages as parent records

PDF file
└── page text

Report record
├── metadata
├── primary PDF
├── supporting attachments
├── pages
├── tables
├── figures/captions
└── quality flags

From current to better: example 2

Move from page text to evidence objects

{
  "page_number": 25,
  "page_text": "Figure 3... YEARPER CENT..."
}

{
  "chunk_type": "table",
  "page": 25,
  "caption": "Inflation, 2020-2024",
  "units": "per cent",
  "headers": ["2020", "2021", "2022"],
  "text": "Inflation table with source page"
}

Quick question 5

Which improvement is most useful for table-heavy official reports?

A. Only increase the LLM prompt length
B. Preserve tables as structured evidence objects with captions, units and page references
C. Remove all tables before indexing
D. Assume the LLM can reconstruct tables from any text

Better ingestion checklist

A good ingestion pipeline should preserve:

Anchor examples for today

Takeaways

• PDF processing is where real-world documents enter the RAG pipeline.
• Good ingestion is not just text extraction; it is evidence preservation.
• Tables, headings, metadata and page references matter for official statistics.
• Current StatsChat ingestion works and has improved, but this is still a key area for development.
• Better ingestion and better retrieval need to be designed together.