Week 7 Lab: Retrieval

2026-05-28

Lab focus

Choosing the right evidence

Today we will practise retrieval as evidence selection.

  • What should a retrieval system return?
  • Which chunks are useful evidence?
  • Which chunks are related but risky?
  • How do we tell retrieval failure from generation failure?

No coding required: the aim is to think like retrieval designers.

Recap

Search finds possible matches.

Retrieval for RAG decides which evidence is good enough to pass to the LLM.

  • relevant enough?
  • current enough?
  • specific enough?
  • not duplicated or distracting?
  • enough evidence to answer safely?

Shared example

User question

What was the annual inflation rate in February 2025?

Imagine a StatsChat-like system retrieves four chunks.

Your task:

  • choose which chunks should go to the LLM
  • decide which chunks should be excluded
  • say whether there is enough evidence to answer

Chunk A

February 2025 CPI release

The annual inflation rate was 5.1 per cent in February 2025.
The Consumer Price Index increased from the previous month.
Source: CPI release, February 2025.

Directly relevant and date-matched.

Chunk B

January 2025 CPI release

The annual inflation rate was 5.3 per cent in January 2025.
Food and non-alcoholic beverages contributed to the annual rate.
Source: CPI release, January 2025.

Semantically similar, but from the wrong month.

Chunk C

CPI methodology note

The Consumer Price Index measures changes in the price of a basket of goods
and services purchased by households. The index is used to monitor inflation.

Useful for definitions, but it does not contain the requested figure.

Chunk D

Older economic survey

Inflation remained stable during 2019, supported by changes in food and fuel prices.
The annual average inflation rate was reported in the Economic Survey 2020.

Related topic, but stale and not answer-bearing.

Discussion 1

Which chunks should go to the LLM?

Discuss:

  • Which chunk should definitely be passed to the LLM?
  • Should Chunk B be included or excluded?
  • What is useful about Chunk C?
  • What makes Chunk D risky?
  • Is there enough evidence to answer safely?

Suggested judgement

  • Chunk A: include — direct answer, correct month/year
  • Chunk B: usually exclude — useful only if comparison is requested
  • Chunk C: exclude for this question — definition, not value
  • Chunk D: exclude — old and not specific

The goal is not “related text”. The goal is answer-bearing evidence.

Quick check

A chunk can be a good search result but weak evidence

Chunk type Search result? Good evidence?
Mentions the right topic Maybe Not necessarily
Comes from the right report Probably Not always
Contains the exact figure, date and unit Yes Usually
Explains the definition but not the value Maybe Depends on the question
Comes from an old release Maybe Risky for “latest” questions

Diagnose the failure

For each situation, decide the main problem.

Situation Main problem?
Right topic, wrong year ?
Right report, wrong page ?
Figure retrieved, but unit/table heading missing ?
Good evidence retrieved, but LLM misreads it ?

Suggested diagnosis

Situation Likely issue
Right topic, wrong year retrieval/date handling
Right report, wrong page retrieval/page or chunk selection
Figure retrieved, but unit/table heading missing chunking/context problem
Good evidence retrieved, but LLM misreads it generation/prompting problem

Many weak answers begin before generation: the model may not have received the right evidence.

If you have your own example

Think of one source from your organisation.

  • What question might a user ask?
  • What exact evidence would be needed?
  • Would keyword search be enough?
  • Would semantic search help?
  • What metadata would matter?
  • Where could retrieval fail?

Fallback example

Monthly labour market bulletin

User question:

What is the latest unemployment rate?

What should the system retrieve?

  • latest labour market bulletin
  • correct unemployment section
  • relevant table or paragraph
  • reference period
  • geography
  • rate, unit and caveats

Improvement plan

For either your own source or the fallback example, choose two improvements.

  • hybrid keyword + semantic search
  • better metadata filters
  • stronger reranking
  • better latest/date handling
  • better table-aware chunking
  • query rewriting
  • retrieval evaluation

Wrap-up

Three takeaways

  1. Retrieval is evidence selection, not just search.
  2. Similar text is not always good evidence.
  3. Official statistics often need vector search, keyword search, metadata, dates, reranking and evaluation together.

Next session

Generation

Next week we move from retrieval to generation:

  • how the LLM uses retrieved evidence
  • how prompts constrain answers
  • how grounded answers are produced
  • when the system should say it does not know