Week 6: Splitting, Chunking and Vectorising

2026-05-11

Welcome

Splitting, chunking and vectorising

Core session 3

From extracted text to searchable evidence.

  • Week 5: PDFs became extracted text and JSON
  • Week 6: text becomes chunks, embeddings and a vector index
  • Key idea: a chunk is the unit of evidence retrieval can find

Quick question 1

A user asks one question about a 100-page report. What should we send to the LLM?

A. The whole extracted report

B. The first page of the PDF

C. A few relevant chunks with source metadata

D. Nothing — the LLM should answer from memory

ℹ️ RAG works by finding the right evidence, not by sending everything or relying on model memory.

Work backwards from the answer

What does the LLM need to answer well?

A good answer needs:

  • relevant evidence
  • enough context
  • source and page metadata
  • labels, units and dates
  • not too much irrelevant text

So before retrieval, we need:

  • useful chunks
  • embeddings for those chunks
  • a vector store to search them
  • metadata that travels with each chunk

Chunking is where we decide what the retriever is allowed to find.

Where this session fits

Today we focus on the bridge between processed documents and retrieval.

Why this matters for your organisation

Many of you may use or adapt a StatsChat-like system for your own organisation.

  • Your source documents may be long, table-heavy or highly structured
  • Users may ask for definitions, latest figures, trends, caveats or comparisons
  • The system can only retrieve the chunks you create

Keep one publication or data source from your own organisation in mind during the session.

Today’s aims

By the end, you should be able to explain:

  • why we split extracted text into chunks
  • what makes a chunk useful or risky
  • what embeddings represent
  • what a vector store does
  • how StatsChat currently builds its searchable index
  • why chunking choices should evolve as models and evidence needs change

Quick question 2

What is the best description of a “chunk” in a RAG system?

A. A random piece of text cut from a document

B. The unit of evidence the retriever can search for and send to the LLM

C. A whole PDF report

D. A summary written by the LLM

ℹ️ A chunk is the searchable evidence unit. Bad chunks make retrieval and generation harder.

From document to chunks

Illustration of a report being split into evidence chunks before AI search

Splitting is not just a technical convenience. It shapes what evidence the system can later find.

Key terms

Term Simple meaning
Splitting Breaking long documents/pages into smaller pieces
Chunking Creating retrieval-sized evidence units
Embedding Turning a chunk into a numerical representation of meaning
Vector store The searchable database of chunk embeddings
Retrieval Finding the chunks closest to the user’s question

Demo 1: simple chunking

sample_text = """
Consumer Price Indices and Inflation Rates - March 2026.
Table 1: One month and twelve month percentage changes by CPI division.
Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%.
Source: Kenya National Bureau of Statistics, CPI release, March 2026.
"""

def chunk_words(text: str, max_words: int, overlap: int = 0) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + max_words
        chunks.append(" ".join(words[start:end]))
        start = end - overlap
    return chunks

The highlighted lines are where the chunk size and overlap shape the output.

Demo 1 output: small chunks

max_words = 16, overlap = 4

Chunk 1:
Consumer Price Indices and Inflation Rates - March 2026. Table 1: One month and twelve

Chunk 2:
month and twelve month percentage changes by CPI division. Total: weight 100.0000, one month

Chunk 3:
one month change 0.5%, twelve month change 4.4%. Source: Kenya National Bureau of Statistics,

The evidence is split across chunks: table title, row and source note are separated.

Demo 1 output: larger chunk

max_words = 42, overlap = 4

Chunk 1:
Consumer Price Indices and Inflation Rates - March 2026. Table 1: One month and twelve month percentage changes by CPI division. Total: weight 100.0000, one month change 0.5%, twelve month change 4.4%. Source: Kenya National Bureau of Statistics, CPI release, March 2026.

The chunk is longer, but it keeps the question-relevant evidence together.

Quick question 3

What is the biggest risk if chunks are too small?

A Retrieval becomes impossible

B. Important context may be split away from the value or statement

C. The vector store cannot be saved

D. The LLM will always refuse to answer

ℹ️ Small chunks can be precise, but they can lose the heading, table title, unit or source note needed to interpret the evidence.

Chunk size is a trade-off

Too small

  • precise matches
  • lower token cost
  • can lose context
  • headings and values may separate
  • more chunks to search

Too large

  • more context preserved
  • fewer boundary problems
  • can include irrelevant material
  • retrieval may be less focused
  • higher token cost if sent to LLM

The best chunk size depends on document structure, model context window, retrieval method and evaluation results.

Overlap: why repeat some text?

  • overlap reduces hard boundary problems
  • it helps keep context near values
  • too much overlap creates duplication and cost

How chunking is done in practice

There is a spectrum of methods

Method Basic idea When useful
Fixed-size split every N characters/words simple baseline, demos
Recursive try paragraphs, then lines, then words good general default
Structure-aware split by headings, tables, pages or sections official reports
Semantic split where topic meaning changes long narrative text
LLM-assisted ask a model to identify evidence units complex/messy documents

Start simple, then improve where evaluation shows problems.

Practical method: recursive chunking

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    separators=["\n\n", "\n", " ", ""],
)

chunks = splitter.split_text(extracted_text)

This still has a size limit, but it tries not to cut through natural boundaries such as paragraphs and lines.

Structure-aware chunking for official statistics

The chunk should follow the evidence

Content type Better chunking choice
Prose keep paragraphs/sections together
Tables keep caption, headers, rows, units and source note together
Definitions keep definition and caveat together
Factsheets keep visual sections and labels together
Source notes keep them with the evidence they qualify

For official statistics, the best chunk is often a meaningful evidence unit, not a fixed number of characters.

Optional advanced methods

Semantic chunking

  • uses embeddings to detect topic shifts
  • useful for long narrative sections
  • can be harder to debug
  • not always ideal for tables

LLM-assisted chunking

  • can identify sections/tables/caveats
  • can create richer evidence objects
  • slower and more expensive
  • must be evaluated carefully

A future StatsChat pipeline might combine methods rather than use one splitter everywhere.

Good chunk vs bad chunk

Infographic comparing poor chunking and good chunking

A useful chunk should usually be understandable enough to act as evidence.

Quick question 4

Which chunk would be safest for answering “What was annual inflation in March 2026?”

A. Total 100.0000 0.5 4.4

B. Table 1: CPI percentage changes. Total: weight 100.0000, one-month change 0.5%, twelve-month change 4.4%. March 2026.

C. Source: KNBS

D. Food and transport prices...

ℹ️ It keeps the value with the table context, unit meaning and reference period.

StatsChat’s current indexing pipeline

The current branch already carries lightweight document context into chunk text before embedding.

Current StatsChat code pattern

# Simplified from statschat/embedding/preprocess.py

self.text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=self.split_length,
    chunk_overlap=self.split_overlap,
    length_function=len,
)

self.chunks = self.text_splitter.split_documents(self.docs)

self.embeddings = HuggingFaceEmbeddings(
    model_name=self.embedding_model_name,
    model_kwargs={"local_files_only": True},
)

self.db = FAISS.from_documents(self.chunks, self.embeddings)
self.db.save_local(self.faiss_db_root)

This is the core: split documents, embed chunks, save vectors and metadata to FAISS.

A current improvement: add context before chunking

prefix_lines = []

if title:
    prefix_lines.append(f"Title: {title}")
if date:
    prefix_lines.append(f"Release date: {date}")
if page_number:
    prefix_lines.append(f"Page number: {page_number}")

prefix = "\n".join(prefix_lines)
doc.page_content = f"{prefix}\n\n{body}"

This helps otherwise similar numeric chunks carry their title, date and page context into retrieval.

Why StatsChat’s chunking is due a rethink

The current approach is sensible for an early RAG prototype, but it reflects older assumptions.

  • early LLMs had smaller context windows
  • smaller chunks were often safer and cheaper
  • modern LLMs can usually handle more context
  • recent larger-chunk experiments improved answer quality
  • chunking should be tuned with evaluation, not fixed forever

Teaching point: chunking strategy should evolve with models, retrieval methods and evidence needs.

Example: CPI table as a chunking problem

CPI March 2026 table crop

What should stay together?

  • report title
  • reference period
  • table heading
  • row label
  • values
  • units / column meanings
  • page reference

Example: visual factsheet as a chunking problem

ICT factsheet visual section crop

What could go wrong?

  • values separate from labels
  • indicators merge together
  • visual grouping disappears
  • a chunk may mix unrelated sections
  • retrieval may find the right words but weak evidence

Embeddings: the intuition

Text becomes a point in “meaning space”

  • each chunk is converted into a vector: a list of numbers
  • similar meanings should be closer together
  • the user question is embedded in the same space
  • retrieval finds nearby chunks

You do not need to understand every dimension. The useful idea is similarity in meaning.

Vectorising one piece of text

Text:
"Annual inflation was 4.4 percent in March 2026."

Embedding:
[0.08, -0.22, 0.51, 0.17, ..., -0.04]

The real vector usually has hundreds or thousands of numbers. We do not read the numbers directly; we use them to compare meanings.

Vector search

Illustration of vector search finding nearby document chunks

The query is embedded, then the vector store returns chunks with nearby vectors.

Demo 2 output

0.67  Annual inflation was 4.4 percent in March 2026.
0.00  The housing survey describes household conditions.
0.00  Mobile phone ownership was 53.7 percent nationally.
0.00  Food and transport prices changed consumer prices.

The system does not “understand” like a human; it ranks chunks by vector similarity.

Quick question 5

What does an embedding represent in this context?

A. A citation to a PDF page

B. A compressed numerical representation of text meaning

C. A cleaned JSON file

D. A final answer from the LLM

ℹ️ An embedding is a numerical representation used for similarity search.

Vector stores and FAISS

What FAISS does for StatsChat

  • stores embeddings for many chunks
  • searches for vectors close to the user query
  • returns matching chunks and metadata
  • gives retrieval a fast searchable index

FAISS is not the LLM. It is the local search index used before generation.

Quick question 6

A user asks about “latest inflation”. Which choice most affects what evidence reaches the LLM?

A. The PDF file name only

B. The chunking, embeddings, retrieval ranking and date metadata

C. The colour of the report cover

D. The LLM temperature only

ℹ️ The answer depends on chunk quality, semantic similarity, retrieval ranking and recency information.

Beyond RAG: embeddings are useful elsewhere

Your colleague may have used vectorising in a text classification project.

  • in RAG: embeddings help find relevant evidence
  • in classification: embeddings can help group or classify text
  • in deduplication: embeddings can find near-duplicates
  • in clustering: embeddings can reveal themes

Keep this brief: useful context, but today’s main story is RAG retrieval.

Practical improvement ideas

Current / simple

  • character-based chunks
  • fixed chunk size and overlap
  • page-level text
  • generic embedding model

Better / future

  • structure-aware chunks
  • table and figure chunks
  • metadata-rich evidence objects
  • larger chunks where helpful
  • empirical tuning with evaluation

Lab task

Compare chunking options

In the lab, you will look at a short extracted text example and compare possible chunking strategies.

  • Which chunk is most useful for retrieval?
  • Which is safest for the LLM?
  • What metadata should each chunk carry?
  • What could go wrong if chunks are too small or too large?

Takeaways

  • A chunk is the unit of evidence retrieval can find
  • Good chunks preserve meaning, source and enough context
  • Embeddings turn chunks and queries into vectors for similarity search
  • FAISS stores and searches those vectors
  • StatsChat’s chunking should evolve as models and evaluation results improve