Week 7: Retrieval
2026-05-26
Welcome
Retrieval: finding the right evidence
Core session 4
How StatsChat chooses which chunks to show the LLM.
Quick question 1
A user asks: “What was GDP growth in 2024?” What happens first at query time?
A. The LLM reads every PDF
B. The system searches for relevant evidence chunks
C. The system downloads all reports again
D. The answer is generated from model memory
Suggested answer: B — by query time, the searchable index has already been prepared.
Zooming out
Retrieval is a small step in the whole pipeline, but it controls what evidence reaches the LLM.
The framing for today
Retrieval is evidence selection
In a RAG system, retrieval is the step that decides:
- which chunks are most relevant to the user’s question
- which sources are current, specific and trustworthy enough
- which evidence should be shown to the LLM
- which weak, duplicate, old or distracting chunks should be left out
- whether the system has enough evidence to answer at all
Guiding question: what evidence should StatsChat pass to the LLM for this question?
Retrieval is not just “search”
Search usually means:
- find documents or passages that match a query
- rank them by keyword match, similarity or another score
- show the user a list of possible results
Retrieval for RAG has an extra job:
- find evidence suitable for the LLM to use
- decide which chunks are relevant enough, current enough and specific enough
- pass only a small amount of context into generation
In RAG, retrieval is not just finding results. It is selecting the evidence that shapes the answer.
Search result vs evidence
A chunk can be a good search result but weak evidence.
| Chunk type | Search result? | Good evidence? |
|---|---|---|
| Mentions the right topic | Maybe | Not necessarily |
| Comes from the right report | Probably | Not always |
| Contains the exact figure, date and unit | Yes | Usually |
| Explains the definition but not the value | Maybe | Depends on the question |
| Comes from an old release | Maybe | Risky for “latest” questions |
Retrieval needs to ask: does this chunk help answer this specific question safely?
Why this matters for your organisation
If you built a StatsChat-like tool for your own organisation, retrieval would decide whether users see the right evidence.
- Which publication is most relevant?
- Which page or section contains the answer?
- Is the evidence current enough?
- Does the chunk contain the value, definition, unit and caveat?
- Is the evidence strong enough to answer at all?
Think about your own source
One-minute reflection
Think of one publication or data source from your own organisation.
- What question might a user ask?
- Which document would contain the answer?
- Which page, table, paragraph or chart would be the evidence?
- What metadata would help the system find it?
Retrieval in plain English
Retrieval means:
finding the pieces of source material most likely to help answer the user’s question.
- the user asks a question
- the system searches the prepared knowledge base
- the system returns a small set of evidence chunks
- those chunks become context for the LLM
Quick question 2
What does StatsChat usually retrieve?
A. Whole PDFs
B. Whole websites
C. Text chunks with metadata
D. Only publication titles
Suggested answer: C — the evidence unit is usually a retrieved chunk, carrying metadata such as title, page URL and score.
Why retrieve evidence at all?
Direct LLM / whole PDF
- simple idea
- but weak grounding if no evidence
- whole PDFs can be slow and noisy
- citations are harder to interpret
Retrieval first
- finds official evidence
- selects a small amount of context
- supports references and checking
- reduces noise before generation
For official statistics, grounding, traceability and source quality matter.
Quick question 3
Why retrieve chunks before asking the LLM to answer?
A. Because the LLM cannot generate text without chunks
B. Because it is usually better to give the LLM a small set of relevant evidence than every PDF
C. Because embeddings always produce perfect answers
D. Because keyword search is never useful
Suggested answer: B — the aim is focused, traceable evidence.
Search methods in context
Retrieval is a design space
| Method | Similar to | Useful when |
|---|---|---|
| Keyword search | search box / Ctrl+F | exact terms, titles, acronyms |
| Semantic search | search by meaning | natural-language questions |
| Metadata filters | library catalogue filters | date, theme, geography, report type |
| Reranking | reviewing and reordering results | first search is close but imperfect |
| Hybrid retrieval | combining methods | you want strengths of several approaches |
Keyword vs semantic search
Keyword search
- matches exact words
- strong for titles, acronyms, codes
- transparent and predictable
- misses different wording
Semantic search
- matches related meaning
- strong for natural-language questions
- bridges synonyms and paraphrases
- can retrieve similar but unhelpful text
Why vectorise text?
Semantic search needs a way to compare meanings.
- turn the user question into a vector
- turn each chunk into a vector
- compare the query vector with chunk vectors
- retrieve nearby chunks
This is why embeddings matter: they make chunks searchable by meaning.
Toy numerical example
Made-up vector dimensions
For teaching, imagine each text becomes four numbers:
| Dimension | Rough meaning |
|---|---|
| 1 | inflation / prices |
| 2 | population / demographics |
| 3 | current / date-specific |
| 4 | methodology / definition |
Real embeddings have many more dimensions, and the dimensions are not usually human-labelled like this.
Toy query vector
Question:
What was the latest inflation rate?
Made-up vector:
Toy chunk vectors
| Chunk | Text | Toy vector |
|---|---|---|
| A | Annual inflation was 5.1% in February 2025 | [0.90, 0.00, 0.80, 0.10] |
| B | CPI measures changes in prices over time | [0.70, 0.00, 0.10, 0.90] |
| C | Population growth was 2.3% | [0.00, 0.90, 0.20, 0.00] |
| D | Food prices increased during the quarter | [0.45, 0.00, 0.25, 0.20] |
| E | Annual inflation was 5.3% in January 2025 | [0.85, 0.00, 0.45, 0.10] |
Comparing the vectors
Higher cosine similarity means closer meaning
| Rank | Chunk | Toy cosine similarity |
|---|---|---|
| 1 | A: February inflation rate | 1.00 |
| 2 | E: January inflation rate | 0.96 |
| 3 | D: food prices | 0.92 |
| 4 | B: CPI definition | 0.58 |
| 5 | C: population growth | 0.14 |
This is simplified. The point is that numbers allow the system to rank chunks by closeness to the query.
Visualising the toy example
This picture is simplified to two dimensions. Real vector spaces have many more.
Technical note: L2 vs cosine
Vector search needs a way to measure how close two vectors are.
| Method | Intuition | In retrieval |
|---|---|---|
| L2 / Euclidean distance | straight-line distance between points | lower = closer |
| Cosine similarity | angle / direction between vectors | higher = more similar |
| Inner product | dot product between vectors | higher = more similar |
These are different scoring methods, but they can be closely related.
StatsChat metric
- FAISS index class:
IndexFlatL2 - metric: L2 / Euclidean distance
- vectors: 182,050
- dimensions: 768
- stored vectors are unit-normalised
So StatsChat returns L2-style distance scores: lower scores are better.
Why this behaves like cosine search
For unit-normalised vectors, L2 distance and cosine similarity give the same ranking.
- StatsChat is configured as an L2 FAISS index
- but the stored vectors are unit-normalised
- so it behaves like cosine-similarity search in practice
- changing this should still be tested with retrieval evaluation
Quick question 4
In the toy example, why might Chunk E be risky?
E | Annual inflation was 5.3% in January 2025 |
[0.85, 0.00, 0.45, 0.10] |A. It is about population
B. It is semantically similar but may be older than the requested “latest” rate
C. It contains no numbers
D. It cannot be embedded
Suggested answer: B — it is relevant to inflation, but may not be the best evidence if the user asked for the latest figure.
Current StatsChat retrieval path
FAISS is the first-stage vector search. It is not the whole retrieval method.
What StatsChat retrieval currently does
At query time, StatsChat roughly does the following:
- embeds the user’s question using the same embedding model used for chunks
- searches the FAISS vector store for nearby chunk vectors
- filters out weak matches using distance thresholds
- applies latest/date, deduplication and reranking logic
- selects a smaller set of chunks as context for the LLM
- returns retrieved chunks as references/provenance
FAISS finds candidate chunks. The wider retrieval pipeline decides which chunks are useful enough to use.
Current implementation pattern
# 1. Search FAISS for candidate chunks
matches = db.similarity_search_with_score(
query=question,
k=candidate_k,
)
# 2. Keep candidates below a distance threshold
matches = [m for m in matches if m[-1] <= similarity_threshold]
# 3. Rerank and select a smaller result set
retrieved_docs = rerank_results(question, matches)
retrieved_docs = retrieved_docs[:k_docs]
# 4. Select context chunks for generation
context_docs = select_generation_documents(retrieved_docs, k_contexts)Simplified pseudocode. The important pattern is: search → filter → rerank → select context.
StatsChat retrieval knobs
| Setting | Simple meaning |
|---|---|
k_docs |
maximum retrieved references returned |
k_contexts |
maximum chunks passed to the LLM |
similarity_threshold |
filter out weak search matches |
answer_threshold |
decide when answer evidence may be too weak |
document_threshold |
decide when no suitable documents were found |
These are design choices: changing them affects quality, cost and user trust.
“Latest” is not simple
For official statistics, users often ask for the latest figure.
- latest publication?
- latest data point?
- latest annual value?
- latest monthly release?
- latest revised value?
- latest for this geography or indicator?
A system may need semantic search, dates, metadata and reranking to handle this well.
Reranking and recency
First search, then reorder
- semantic search gives candidate chunks
- reranking judges which candidates look most useful
- recency logic can favour newer publications where appropriate
- metadata helps distinguish similar reports from different dates
This is especially important when older and newer reports use very similar wording.
Quick question 5
A user asks for the “latest unemployment rate”. What makes retrieval difficult?
A. The word “latest” may require date-aware ranking
B. The latest publication may not contain the latest data point
C. Several publications may mention unemployment
D. All of the above
Suggested answer: D — this is why retrieval for official statistics needs more than simple similarity.
Thresholds: when should the system say “not enough”?
Retrieval is also about deciding when evidence is too weak.
- low-quality matches can mislead the LLM
- returning no answer can be safer than forcing an answer
- thresholds help decide what counts as a suitable match
- thresholds must be tuned with evaluation
What the LLM sees
The LLM does not see the whole collection.
- it sees the user question
- it sees selected context chunks
- those chunks carry source metadata
- generation quality depends heavily on retrieved context
Bad retrieval can lead to bad answers even if the LLM itself is strong.
Quick question 6
If the retrieved chunks do not contain the answer, what is the primary failure?
A. Retrieval failure
B. Generation failure
C. Frontend failure
D. Cost failure
Suggested answer: A — the LLM can only work with the evidence it is given.
Retrieval failure types
| Failure type | Example | Why it matters |
|---|---|---|
| Old but similar | inflation report from 2022 for a “latest” question | semantically close but stale |
| Right report, wrong page | methodology page instead of figure/table | source family is right, evidence is wrong |
| Similar but not answer-bearing | text about exports but no export value | related topic is not enough |
These are retrieval failures even if the final generated answer is where we notice the problem.
Evidence selection activity
Which chunks should go to the LLM?
Question:
What was the annual inflation rate in February 2025?
In the next slide, imagine these are the top retrieved chunks.
Your task: choose which evidence should be passed to the LLM, and which should be excluded.
Retrieved chunks
A. February 2025 CPI release
The annual inflation rate was 5.1 per cent in February 2025...
B. January 2025 CPI release
The annual inflation rate was 5.3 per cent in January 2025...
C. CPI methodology note
The Consumer Price Index measures changes in prices of goods and services...
D. Economic Survey 2020
Inflation remained stable during 2019...
Quick question 7
Which chunk should definitely be passed to the LLM?
A. February 2025 CPI release
B. January 2025 CPI release
C. CPI methodology note
D. Economic Survey 2020
Suggested answer: A — it directly answers the question and matches the month/year.
Activity discussion
Suggested judgement
- A should definitely go to the LLM
- B may be useful only if comparison is needed
- C helps definition questions, not this figure question
- D is old and not directly relevant
This is the central retrieval judgement: not all relevant-looking evidence is useful evidence for the question.
Beyond the current approach
How retrieval could improve
- hybrid search: combine keyword and semantic results
- metadata filters: date, theme, geography, publication type
- better reranking: judge candidate chunks more carefully
- query rewriting: turn vague questions into better search queries
- table-aware retrieval: keep headings, units and values together
- evaluation-driven tuning: test retrieval against known questions
Hybrid retrieval
Why combine keyword and semantic search?
Keyword helps with
- exact titles
- acronyms
- codes
- named indicators
- publication series
Semantic helps with
- natural-language questions
- synonyms
- related wording
- broad topic search
- user uncertainty
For official statistics, a strong system may need both.
Quick question 8
Which improvement would most directly help if the system finds the right report but wrong page?
A. Better page-level retrieval or reranking
B. Removing metadata
C. Increasing LLM temperature
D. Ignoring dates
Suggested answer: A — the retrieval system needs to locate the answer-bearing page or chunk, not just the correct publication.
Retrieval choices are trade-offs
| Choice | Benefit | Risk |
|---|---|---|
| Fewer chunks | cheaper, less noise | may miss evidence |
| More chunks | broader context | more irrelevant/conflicting text |
| Semantic search | handles natural language | similar may not mean useful |
| Keyword search | transparent exact matching | misses different wording |
| Metadata filters | strong control | needs reliable metadata |
| Reranking | better ordering | more complexity |
How do we know retrieval is good?
Evaluate it
- create test questions with known evidence
- check whether the right chunk/report/page is retrieved
- check whether the latest/correct edition is preferred
- inspect failure cases
- tune chunking, metadata, search and reranking
Retrieval quality should be measured, not guessed.
Back to your organisation
For your own source, ask:
- What would count as the right evidence?
- Would users know exact terminology?
- Would dates or revisions matter?
- Would keyword search be enough?
- What metadata would help?
- What failures would be most risky?
Thursday lab exercise
Choosing the right evidence
In the lab, we will practise retrieval judgement.
- choose a source from your own organisation
- identify a question a user might ask
- decide what would count as the right evidence
- compare several retrieved chunks
- decide which chunks should be passed to the LLM
- discuss likely retrieval failures and improvements
The aim is not to write code. The aim is to think like a retrieval designer.
Takeaways
- Retrieval is evidence selection
- StatsChat retrieves chunks, not whole PDFs
- Vector search helps with natural-language questions
- StatsChat uses L2 FAISS scores, but unit-normalised vectors make this cosine-equivalent in practice
- Similar evidence is not always sufficient evidence
- Official statistics need date, metadata and source awareness
- Retrieval can be improved through hybrid methods, reranking and evaluation
Next session
Generation
Next week we will look at what the LLM does with the retrieved evidence.
- how prompts constrain answers
- how retrieved context is used
- how grounded answers are produced
- when the system should refuse or say it does not know
