Generation, prompting, and grounded answers

Published

3 June 2026

Last week focused on retrieval: finding the chunks of text most likely to contain useful evidence. This week focuses on what happens next: how StatsChat uses those chunks to generate a grounded answer.

The key message is:

Retrieval selects the evidence. Generation uses that evidence to produce an answer.

A good RAG answer is not just fluent. For official statistics, it should be grounded in retrieved evidence, clear about its source, and easy for a user to check.

Slide deck

Session overview

This session uses StatsChat as the main example, but also broadens out to modern LLM applications more generally.

We cover:

  • how generation follows on from retrieval;
  • how the StatsChat interface presents an answer and references;
  • what happens behind the scenes when the system generates an answer;
  • why thresholds, prompts, structured output, and highlighting matter;
  • how to recognise common generation failure modes;
  • how StatsChat fits into a wider family of LLM-powered applications.

Learning aims

By the end of the session, you should be able to explain where generation fits in a RAG system, describe what the LLM receives, inspect whether an answer is supported by references, and connect StatsChat to broader LLM application patterns.

From retrieval to generation

A simple query-time view of StatsChat is:

User question
   ↓
Retrieval selects relevant chunks
   ↓
Generation uses those chunks to produce an answer
   ↓
The system returns an answer and references

Week 7 treated retrieval as evidence selection. The retriever decides which chunks should be passed forward.

Week 8 focuses on evidence use. The generator receives the selected chunks and tries to produce an answer that is justified by them.

Stage Main question Output
Retrieval Which chunks should the LLM see? Retrieved chunks with metadata
Generation What answer can be justified from those chunks? Answer, structured response, references/highlights

The LLM is not searching all KNBS publications during generation. It sees only the context the application gives it. This is why retrieval and generation quality are closely linked.

Live demo: StatsChat answering a question

During the session, we will use the StatsChat interface to ask:

What was Kenya’s real GDP growth rate in 2024?

When we inspect the answer, we will check whether it gives the correct figure, year, and measure; whether the reference supports the claim; and whether the wording adds anything unsupported. The aim is to practise reading an AI-generated answer critically.

What happens behind the scenes?

retrieved context + question → prompt → LLM
   → structured response → highlighting → final answer

Generation is not just “ask the model”. The application decides what context to pass in, how to instruct the model, what format to request, how to parse the response, and how to connect the answer back to the evidence.

Thresholds: deciding whether to answer

StatsChat uses retrieval scores to help decide whether the evidence is strong enough to support an answer.

These are distance scores, not probabilities: lower scores mean a stronger semantic match; higher scores mean a weaker match.

The intended behaviour has three zones:

Zone Evidence strength Behaviour
A Strong enough Return an LLM answer and references
B Weak or partial Suppress the LLM answer, but show potentially useful references
C Too weak Suppress both answer and references

Zone B is important: the system can avoid synthesising an answer while still showing possibly useful evidence for a human to inspect.

Prompting: rules for answer behaviour

A prompt is not just a nicely worded question. In StatsChat, it defines the model’s role, evidence boundary, refusal behaviour, and output format. In plain English, the StatsChat prompt tells the model to answer from official KNBS/statistical context, consider dates, avoid unsupported advice or forecasts, refuse when evidence is insufficient, and return exact phrases for highlighting.

For statistical questions, the prompt also warns the model not to pick the wrong number when the context contains several figures. For example, it should not use a comparison figure such as “compared with 6.3% last year” when the question asks for the current period’s headline figure.

Structured output and highlighting

StatsChat does not only ask the LLM for a paragraph. It asks for structured information, such as whether an answer was provided, what the answer is, and which exact phrases support highlighting.

{
  "answer_provided": true,
  "answer": "Kenya's real GDP grew by 4.7 per cent in 2024.",
  "highlighting": ["real Gross Domestic Product (GDP) grew by 4.7 per cent"]
}

Highlighting helps the user move from “the AI said this” to “this passage supports the answer”. But it is not a guarantee: users still need to check that the highlighted phrase supports the exact claim.

Failure modes in generation

Generation can fail even when the retrieved evidence is broadly relevant. Common failure modes include:

Failure mode Example
Unsupported claim The answer adds a cause, forecast, or recommendation not present in the evidence
Wrong figure The model chooses a comparison number rather than the figure asked for
Missing context The answer gives “4.7%” without saying what it measures or which year it refers to
Weak refusal The model answers a future or out-of-scope question instead of saying it cannot answer
Poor traceability The cited passage is related to the topic but does not support the exact claim

A useful diagnostic question is:

Is the issue with retrieval, generation, the source document, or the way the result is presented to the user?

Broader pattern: modern LLM applications

StatsChat is one example of a wider pattern:

user request → application adds context/tools/rules
   → LLM produces text or structured output
   → application checks, formats, displays, or acts

Modern LLM applications often combine models with documents, APIs, databases, user interfaces, permissions, validation, and human review. Examples include customer support assistants, coding assistants, data assistants, document extraction systems, and workflow assistants.

The model is only one component. The surrounding system determines what context the model receives, what actions it can take, and how users can check the result.

Tiny coding demo

We will also use a tiny demo to demystify what an LLM application is doing.

The demo shows three versions of the same task:

  1. ask a plain question;
  2. add a short piece of context and tell the model to use only that context;
  3. ask for structured output that an application could parse.

The simplest reliable way to run the demo is the mock mode, which does not need an API key or internet access:

python tiny_llm_generation_demo.py --mock

For teaching, the notebook version is usually easier to walk through step by step. The script is a backup if you want a quick command-line run.

Two ways to host the LLM

The StatsChat repo uses labels such as “cloud” and “local”, but for teaching it is clearer to talk about hosting options.

Hosting option Meaning
External-hosted LLM The model is hosted by an external provider such as OpenAI, Google, Anthropic, or Mistral
Self-hosted LLM KNBS hosts the model on its own machines or rented servers

The generation pattern stays the same either way:

question + retrieved context → prompt → model → answer

What changes is who runs the model, where data is sent, what hardware is needed, how fast the system is, and what operational responsibilities KNBS takes on.

Beyond StatsChat: agents and automation

Some LLM systems go beyond a single answer: they can call tools, use APIs, schedule tasks, keep memory, delegate subtasks, and ask for human approval before acting.

StatsChat is not an agent, but it is also not “just a chatbot”. It is an LLM inside a wider system. For official statistics, this raises questions about what should be automated, what should require review, and what should never be delegated to a model.

Summary

The main takeaways are:

  • retrieval selects evidence;
  • generation uses evidence;
  • a good RAG answer should be grounded, structured, and checkable;
  • thresholds help decide when to answer, show references, or refuse;
  • prompts define the model’s task and boundaries;
  • modern LLM applications are systems, not just model calls.

Next week, we bring the pipeline together by asking how to evaluate systems like StatsChat and how participants might adapt the approach for their own NSO context.

Lab

Between-session exercise: inspecting a generated answer

In this exercise, you will inspect a StatsChat-style answer and decide how trustworthy it is.

You do not need to use the StatsChat interface. Use the example below.

User question

What was Kenya’s real GDP growth rate in 2024?

Generated answer

Kenya’s real GDP growth rate in 2024 was 4.7 per cent.

Most relevant publication(s)

2025 Economic Survey Released on: 01 May 2025 Page: PDF page 71 Semantic distance: 0.46

Context preview:

In 2024, Kenya’s real Gross Domestic Product (GDP) grew by 4.7 per cent compared to a revised growth of 5.7 per cent in 2023.

2025 Facts and Figures Released on: 01 May 2025 Page: PDF page 31 Semantic distance: 0.32

Context preview:

In 2024, Kenya’s real Gross Domestic Product (GDP) grew by 4.7 per cent compared to a revised growth of 5.7 per cent in 2023.

2024 Kenya Vital Statistics Report Abridged Version Released on: 01 June 2025 Page: PDF page 7 Semantic distance: 0.50

Context preview:

Between 1962 and 1989, the country’s intercensal growth rate was 3.4 per cent. This declined to 2.9 per cent in 1999…

Questions

  1. Is the generated answer supported by the retrieved evidence?
  • A. Yes, clearly
  • B. Partly, but important information is missing
  • C. No, the evidence does not support it
  • D. Not sure
  1. Which retrieved source best supports the answer?
  • A. 2025 Economic Survey
  • B. 2025 Facts and Figures
  • C. 2024 Kenya Vital Statistics Report Abridged Version
  • D. All three equally
  1. One retrieved source is less useful for answering the question. Which one?
  • A. 2025 Economic Survey
  • B. 2025 Facts and Figures
  • C. 2024 Kenya Vital Statistics Report Abridged Version
  • D. None of them
  1. Why is that source less useful?
  • A. It is about population growth, not GDP growth
  • B. It is older than the others
  • C. Its semantic distance is too low
  • D. It gives the answer but not the source
  1. Here is a weaker alternative answer:

Kenya’s GDP growth slowed in 2024 because agriculture and services performed less strongly than in 2023.

What is the main problem with this answer?

  • A. It uses the wrong year
  • B. It gives a causal explanation that is not clearly supported by the provided evidence
  • C. It should use US English instead of British English
  • D. It does not mention the publication title
  1. In one or two sentences, explain whether you would be comfortable using the original generated answer in a briefing or report. What would you check first?

Lab Solutions: suggested answers and explanation

I’m sorry that we are not able to hold the live lab this week. Instead, please use the notes below to review the between-session exercise. The aim of the exercise was to practise inspecting a generated answer, checking whether it is grounded in the retrieved evidence, and identifying possible weaknesses in the answer or references.

Exercise recap

The user asked:

What was Kenya’s real GDP growth rate in 2024?

The generated answer was:

Kenya’s real GDP growth rate in 2024 was 4.7 per cent.

The retrieved evidence included two directly relevant sources:

In 2024, Kenya’s real Gross Domestic Product (GDP) grew by 4.7 per cent compared to a revised growth of 5.7 per cent in 2023.

This appeared in both the 2025 Economic Survey and 2025 Facts and Figures.

A third retrieved source, the 2024 Kenya Vital Statistics Report Abridged Version, mentioned intercensal growth rates rather than GDP growth, so it was less useful for this question.

Suggested answers

1. Is the generated answer supported by the retrieved evidence?

Suggested answer: A. Yes, clearly

The generated answer is directly supported by the retrieved evidence. The context says that in 2024, Kenya’s real Gross Domestic Product grew by 4.7 per cent. The generated answer repeats that claim accurately and includes the correct year, measure, and unit.

A slightly stronger answer might say:

Kenya’s real Gross Domestic Product (GDP) grew by 4.7 per cent in 2024.

This version keeps the full term “Gross Domestic Product (GDP)”, which makes the statistical measure especially clear.

2. Which retrieved source best supports the answer?

Suggested answer: A or B are both reasonable

Both the 2025 Economic Survey and 2025 Facts and Figures directly support the answer, because both include the relevant statement that Kenya’s real GDP grew by 4.7 per cent in 2024.

There is a useful judgement point here:

  • 2025 Facts and Figures has the lower semantic distance in the example, so it may be the closer retrieval match.
  • 2025 Economic Survey may be a fuller and more detailed source, depending on how the answer will be used.

So the best answer is probably A or B, with an explanation. The important point is that both sources directly support the generated answer.

3. One retrieved source is less useful for answering the question. Which one?

Suggested answer: C. 2024 Kenya Vital Statistics Report Abridged Version

This source is less useful because it does not answer the GDP question. It refers to intercensal growth rates, which are about population change between censuses, not real GDP growth.

This is a good example of a retrieved source that is statistically themed and contains growth-rate language, but is not actually the right evidence for the question.

4. Why is that source less useful?

Suggested answer: A. It is about population growth, not GDP growth

The question asks about real GDP growth. The less useful source refers to intercensal growth rate, which is a different concept. Even though both involve percentages and growth, they describe different statistical measures.

This is one reason why users should inspect references rather than relying only on the generated answer. A retrieved document can look relevant at first glance while still not supporting the exact claim being made.

5. What is the main problem with the weaker alternative answer?

The weaker alternative answer was:

Kenya’s GDP growth slowed in 2024 because agriculture and services performed less strongly than in 2023.

Suggested answer: B. It gives a causal explanation that is not clearly supported by the provided evidence

The evidence supports the claim that real GDP growth was 4.7 per cent in 2024, compared with 5.7 per cent in 2023. So it is reasonable to say that growth was lower in 2024 than in 2023.

However, the weaker answer goes further and gives a causal explanation: it says growth slowed because agriculture and services performed less strongly. That causal claim is not clearly established by the provided evidence.

A safer answer would be:

Kenya’s real GDP grew by 4.7 per cent in 2024, compared with a revised growth of 5.7 per cent in 2023.

This version stays close to the evidence and avoids adding an explanation that the retrieved context does not clearly support.

6. Would you be comfortable using the original generated answer in a briefing or report? What would you check first?

A good answer might say:

Yes, I would be reasonably comfortable using the answer because the figure is directly supported by the retrieved evidence. Before using it in a briefing or report, I would open the source page, confirm the exact wording, check the year and unit, and cite the publication properly.

The key checks are:

  • Does the evidence contain the exact figure?
  • Is the year correct?
  • Is the statistical measure correct?
  • Is the unit clear?
  • Is the cited source actually relevant?
  • Does the answer avoid adding unsupported interpretation?

Main lesson

This example is a relatively good generated answer because the answer is short, clear, and directly supported by the retrieved evidence.

However, the exercise also shows why inspection matters. One of the retrieved sources was not actually useful for the GDP question, and the weaker alternative answer added a causal explanation that was not clearly supported.

A good RAG answer should not only be fluent. It should be:

  • grounded in the retrieved evidence;
  • clear about the measure, period, and unit;
  • traceable to a relevant source;
  • careful not to add unsupported claims.