Generation: Turning Evidence into Grounded Answers

Today’s focus

Last week:

Retrieval is evidence selection.

This week:

Generation is evidence use.

The question is now:

What answer can we safely produce from the retrieved chunks?

Learning aims

By the end of today, you should be able to:

• explain how generation follows retrieval;
• describe what the LLM receives in StatsChat;
• distinguish grounded answers from general chatbot answers;
• explain why prompts, thresholds, and structured output matter;
• inspect whether an answer is supported by references;
• connect StatsChat to broader modern LLM applications.

Query-time RAG

User question
   ↓
Retrieval
   ↓
Selected chunks
   ↓
Generation
   ↓
Answer + references

Retrieval decides what the LLM gets to see.

Generation decides what can be said from what the LLM sees.

User-facing view vs system view

User sees:

Question → Answer → References

System does more:

Question → Retrieval → Context selection → Prompt
         → LLM → Structured response → Highlighting
         → Answer + references

What the LLM sees

The LLM does not see every KNBS document.

It sees something like:

Instruction:
Answer using official KNBS context only.

Context:
[small number of retrieved chunks]

Question:
What was Kenya’s real GDP growth rate in 2024?

Thresholds before the prompt

StatsChat uses retrieval scores to help decide whether to answer.

Important:

• these are distance scores, not probabilities;
• lower is better;
• a weak match may mean the system should not synthesise an answer.

Threshold zones

Better match                                      Worse match
0 ───────── answer threshold ───────── document threshold ─── 2.0

Zone A: answer + references
Zone B: no answer, but show possible references
Zone C: no answer and no references

Zone B is useful because weak evidence may still help a human user.

Prompting is not magic

A prompt defines the model’s task and boundaries.

For RAG, it should say:

• what role the model has;
• what evidence it may use;
• when it should refuse;
• how specific/cautious the answer should be;
• what output format is expected.

StatsChat prompt in plain English

The StatsChat prompt tells the model to:

• answer from official KNBS/statistical context;
• be concise, professional, impartial, and use British English;
• consider dates in the question and publication title;
• refuse if context is unrelated or insufficient;
• avoid advice, forecasts, causal claims, or subjective judgements unless directly stated;
• return exact phrases for highlighting.

StatsChat prompt: figure-selection rules

For statistical questions, the prompt also says:

• match the metric and unit in the question;
• prefer evidence from a publication with the right period;
• do not use comparison figures as headline answers;
• do not substitute one measure for another.

Example risk:

“3.5% in February 2025, compared with 6.3% in February 2024”

Prompt quick check

Multiple choice

Question: “What was inflation in February 2025?”
Context: “Inflation was 3.5% in February 2025, compared with 6.3% in February 2024.”

Which figure should the model choose?

A. 6.3%
B. The largest number
C. 3.5%
D. Both numbers as the answer

Grounded vs ordinary chatbot answer

Structured output

Instead of just asking for a paragraph, an application may ask for fields:

{
  "answer_provided": true,
  "answer": "Kenya's real GDP grew by 4.7% in 2024.",
  "highlighting": ["real Gross Domestic Product (GDP) grew by 4.7 per cent"]
}

Highlighting and traceability

StatsChat can ask the model for exact supporting phrases.

Then the system searches for those phrases in the retrieved chunks.

This helps the user move from:

“The AI said this.”

To:

“This passage supports the answer.”

Highlighting example

Answer

Kenya’s real GDP grew by 4.7 per cent in 2024.

Supporting phrase

“real Gross Domestic Product (GDP) grew by 4.7 per cent”

UI effect

• highlight the phrase in the retrieved chunk;
• link back to the publication/page;
• help the user verify the claim.

Traceability check

Multiple choice

A highlight says “GDP grew by 4.7 per cent”, but the answer says “services grew by 4.7 per cent”. What is the issue?

A. The highlight proves the answer is correct
B. The answer may use the right number for the wrong measure
C. There is no issue
D. The system has found no documents

When should StatsChat not answer?

Examples:

• context is unrelated;
• date or period does not match;
• question asks about another country;
• user asks for policy advice;
• evidence contains several similar figures;
• question asks for future or unpublished data.

Refusal check

Multiple choice

Which question should StatsChat probably refuse unless the retrieved KNBS context directly supports it?

A. “What was Kenya’s real GDP growth rate in 2024?”
B. “What was the CPI inflation rate in February 2025?”
C. “Should Kenya change interest rates next month?”
D. “Which publication reports GDP growth in 2024?”

Failure mode 1: unsupported claim

Evidence

“Real GDP grew by 4.7 per cent in 2024.”

Generated answer

“GDP grew by 4.7%, mainly because of new government policy.”

Problem

The growth figure is supported, but the cause is not.

Failure mode 2: wrong figure

Question

“What was inflation in February 2025?”

Evidence

“Inflation was 3.5% in February 2025, compared with 6.3% in February 2024.”

Generated answer

“Inflation was 6.3% in February 2025.”

Problem: the model selected the comparison figure.

Failure mode 3: missing context

Generated answer

“It was 4.7%.”

Better answer

“Kenya’s real GDP grew by 4.7% in 2024.”

Problem

The weak answer gives a number but not the country, measure, or period.

Failure mode 4: weak refusal

Question

“What will Kenya’s GDP growth be in 2027?”

Retrieved context

A 2024 report with historical figures only.

Generated answer

“GDP growth will be 4.7%.”

Problem: the system should not answer a future question from historical evidence.

Failure mode 5: poor traceability

Generated answer

“Real GDP grew by 4.7% in 2024.”

Reference/highlight shown

“The Economic Survey includes national accounts statistics.”

Problem

The reference is related to GDP, but it does not support the exact figure.

Activity: classify the failure

Multiple choice

Answer: “Inflation was 6.3% in February 2025.”
Evidence: “Inflation was 3.5% in February 2025, compared with 6.3% in February 2024.”

What is the main failure?

A. Unsupported causal claim
B. Wrong figure
C. Poor UI colour choice
D. Hosting problem

Broader pattern: modern LLM apps

StatsChat is one example of a wider pattern:

User request
   ↓
Application adds context, tools, rules, or examples
   ↓
LLM produces text or structured output
   ↓
Application checks, formats, displays, or acts

The LLM is a component inside a system.

Examples of modern LLM apps

What makes these apps different from chatbots?

Many LLM apps add:

• context: documents, records, history;
• tools: APIs, search, databases, code execution;
• structure: JSON, schemas, forms;
• rules: permissions, refusals, validation;
• interfaces: citations, buttons, review screens;
• evaluation: tests, monitoring, human review.

Tiny LLM app: plain call

question = "What was Kenya's real GDP growth rate in 2024?"

response = client.responses.create(
    model="...",
    input=question,
)

print(response.output_text)

This is a general chatbot-style call.

Tiny LLM app: add context

context = """
In 2024, Kenya's real Gross Domestic Product (GDP)
grew by 4.7 per cent.
"""

prompt = f"""
Answer the question using only the context below.

Context:
{context}

Question:
{question}
"""

Tiny LLM app: ask for structure

prompt = f"""
Use only the context below to answer the question.

Return JSON with:
- answer_provided
- answer
- evidence

Context:
{context}

Question:
{question}
"""

Now the output is easier for an application to parse.

Two ways to host the LLM

The clearer teaching language is:

The generation pattern is the same either way.

Hosting trade-offs

A modular system can change hosting option later.

Beyond StatsChat: agents and automation

Some LLM systems can:

• call tools or APIs;
• remember useful context;
• schedule or monitor tasks;
• delegate subtasks;
• ask for human approval before acting.

StatsChat is not an agent, but it is also not “just a chatbot”.

Discussion: what should an NSO automate?

Multiple choice

Which are good candidates for an NSO LLM system?

A. Drafting a summary with citations for human review
B. Automatically publishing statistics without review
C. Suggesting relevant publications for a user question
D. Changing official figures based on model output

Between-session exercise

Before the lab, complete a short answer-inspection task.

In this exercise, you will inspect a StatsChat-style answer and decide how trustworthy it is.

You do not need to use the StatsChat interface. Use the example provided on the course site.

Key takeaways

• Retrieval selects evidence.
• Generation uses evidence.
• A good RAG answer should be grounded, structured, and checkable.
• Prompts define the model’s task and boundaries.
• Thresholds help decide when to answer or refuse.
• Modern LLM systems combine models with context, tools, APIs, interfaces, and evaluation.

Next week

Next week we bring the pipeline together:

• How do we evaluate systems like StatsChat?
• What kinds of failure should we test for?
• What would need to change for your own NSO?
• When is a RAG system good enough to use?