Splitting, chunking and vectorising

This week is the third core session of the course. Last week focused on PDF ingestion: how official reports become extracted text and JSON. This week looks at the next step: how extracted text becomes searchable evidence.

The session covers splitting, chunking, embedding, and vector stores. These steps determine what evidence the retrieval system can later find and send to the LLM.

The key message is: a chunk is not just a piece of text. It is the unit of evidence the retrieval system can find.

Slide deck

Session overview

In this session, we focus on eight big ideas:

  • the key design question is: what evidence should the LLM receive?
  • extracted text needs to be split into useful retrieval units
  • chunk size and overlap affect how much context is preserved
  • chunks that are too small can lose meaning
  • chunks that are too large can become unfocused and costly
  • chunking methods range from simple fixed-size splitting to structure-aware, semantic and LLM-assisted approaches
  • embeddings turn chunks into numerical representations for similarity search
  • a vector store such as FAISS stores those embeddings so relevant chunks can be retrieved quickly

How this relates to StatsChat

StatsChat currently uses a LangChain-based embedding pipeline. In simplified terms:

  1. JSON conversions are split into page/section-level JSON files.
  2. page_text is loaded with metadata.
  3. lightweight context such as title, date and page number is added to the text.
  4. a recursive text splitter creates chunks.
  5. chunks are embedded using a HuggingFace sentence-transformer model.
  6. embeddings and metadata are saved into a FAISS vector store.

The current approach is sensible for an early RAG prototype, but chunking strategy should not be treated as fixed. Recent work suggests larger chunks can improve answer quality, and modern LLMs can generally handle more context than models available when older RAG approaches were designed. More sophisticated future approaches could use structure-aware chunks for tables/sections, semantic chunking for long narrative passages, and LLM-assisted chunking for complex or messy pages — but these should be evaluated rather than assumed to be better.

Demo scripts

The session includes short code examples shown in the slides. The scripts are also included here for reference:

The optional sentence-transformer demo may require installing packages and downloading a model. It is not required for the main session.

Lab exercise

Week 6 exercise: choosing useful chunks

This short exercise is for the Thursday lab. It should take around 10-15 minutes.

The aim is to practise the main lesson from Tuesday’s session:

A chunk is the unit of evidence the retrieval system can find and pass to the LLM.

You do not need to write code.

Part 1: think about your own source

Use the publication or data source you had in mind from Week 5, or choose a new one.

Answer briefly:

  1. What source would you want to feed into a StatsChat-like system?
  2. What kind of questions would users ask about it?
  3. What would count as a useful evidence unit?
    • a paragraph?
    • a table?
    • a page?
    • a section?
    • a chart plus caption?
  4. What metadata should each chunk carry? For example: title, date, page number, table title, geography, unit, source URL.
Part 2: compare chunking options

Imagine this text has been extracted from a CPI report:

Consumer Price Indices and Inflation Rates - March 2026.
Table 1: One month and twelve month percentage changes by CPI division.
Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%.
Food and non-alcoholic beverages increased by 7.1% over twelve months.
Transport increased by 3.8% over twelve months.
Source: Kenya National Bureau of Statistics, CPI release, March 2026.

A user asks:

What was Kenya’s annual inflation rate in March 2026?

Which chunking option would be best?

Option A: very small chunks
Chunk 1: Consumer Price Indices and Inflation Rates - March 2026.
Chunk 2: Table 1: One month and twelve month percentage changes by CPI division.
Chunk 3: Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%.
Chunk 4: Source: Kenya National Bureau of Statistics, CPI release, March 2026.
Option B: one useful evidence chunk
Chunk 1: Consumer Price Indices and Inflation Rates - March 2026.
Table 1: One month and twelve month percentage changes by CPI division.
Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%.
Source: Kenya National Bureau of Statistics, CPI release, March 2026.
Option C: very large mixed chunk
Chunk 1: Consumer Price Indices and Inflation Rates - March 2026.
Table 1: One month and twelve month percentage changes by CPI division.
Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%.
Food and non-alcoholic beverages increased by 7.1% over twelve months.
Transport increased by 3.8% over twelve months.
Several other sections from the report follow, including notes on methodology, unrelated price divisions, and background information from other pages.
Source: Kenya National Bureau of Statistics, CPI release, March 2026.
Questions
  1. Which option is best for this user question, and why?
  2. What is risky about Option A?
  3. What is risky about Option C?
  4. What metadata would you attach to Option B?
  5. How would your answer change if the user asked a broader question, such as “What drove inflation in March 2026?”
Part 3: bring one point to the lab

Bring one short comment to the lab:

  • one thing that makes a good chunk
  • one risk with chunks that are too small
  • one risk with chunks that are too large
  • or one question about embeddings/vector stores

Short bullet points are completely fine.

Between-session task

Choose one publication or data source from your own organisation. Think about:

  • what a useful chunk would be for that source
  • whether the source contains prose, tables, charts, maps or mixed content
  • whether fixed-size, recursive, structure-aware, semantic or LLM-assisted chunking would be the best starting point
  • what metadata each chunk should carry
  • what might go wrong if the text is split too aggressively
  • what might go wrong if chunks are too large