Week 4: StatsChat Overview

2026-04-28

Welcome

StatsChat Overview

Core session 1

From RAG in general to one real system for official statistics.

  • Last Week: overview of RAG
  • This Week: one real example of a RAG system (StatsChat)

Where this session fits

Last week

  • LLMs
  • RAG
  • grounding
  • hallucination risk

Today

  • StatsChat end to end
  • what happens once
  • what happens per question
  • where it fits

Today’s aims

By the end, you should be able to explain:

  • what StatsChat is for
  • how the pipeline works at a high level
  • which steps happen once and which happen for each question
  • what the outputs really mean
  • why evaluation and human judgement still matter

Quick check

In official statistics, what matters the most?

A. A fluent answer
B. A fast answer
C. A traceable answer

Suggested answer: C — all matter, but traceability is essential.

A real question

What was Kenya’s annual inflation rate in March 2026?

A good answer needs to check:

Definition

What measure of inflation?

Date

Which month and comparison period?

Caveats

What drove the change? What does “drove” mean?

Example: what we would look for

User question

In March 2026, what was Kenya’s annual inflation rate, and what drove it?

Evidence to verify

  • annual CPI inflation: 4.4%
  • March 2026 compared with March 2025
  • main divisions named in the report
  • source: KNBS CPI bulletin, March 2026

Why this is not just “find a number”

We need to know what the number means

  • Is it annual or monthly inflation?
  • Is it from an official KNBS source, with the right date?
  • Are “drivers” rates, contributions, or a narrative explanation?
  • What should the user inspect before trusting the answer?

Manual search vs StatsChat

Manual route

StatsChat route

StatsChat should make the route to evidence faster — not remove the need to check sources.

Why StatsChat?

Question about Kenya → KNBS evidence → grounded answer

The aim is to support verification, not replace judgement.

StatsChat in one sentence

Questions about Kenya, grounded in KNBS evidence

StatsChat helps users answer questions about Kenya and Kenyan statistics by finding relevant KNBS evidence and using it to generate a grounded answer.

Example questions

  • What was inflation in March 2026?
  • Which report supports that figure?

Evidence base

  • official KNBS publications
  • retrieved report passages
  • references back to sources

Two halves of the system

The important split

Prepare once

  • download reports
  • convert PDFs
  • chunk text
  • create embeddings
  • build search index

Answer each question

  • receive question
  • retrieve chunks
  • send context to LLM
  • return answer
  • return references

Prepare once

Making reports searchable

This mainly happens before users ask questions, or when new reports are added.

Answer each question

The query-time flow

This is the part the user experiences directly.

Quick check

Which pair happens for each user question?

A. PDF download + JSON conversion
B. Chunking + embedding
C. Retrieval + generation
D. None of the above

Suggested answer: C — retrieval and generation happen each time a user asks a question.

Query-time input

What does the system receive?

Main input

In March 2026, what was Kenya’s annual inflation rate, and what drove it?

Implementation options

These may include:

  • search scope: latest or all
  • debug mode for technical inspection

For most users, the important input is simply the question. Implementation options matter more for developers, evaluators, or system maintainers.

Query-time output

What does the system return?

User-facing

  • answer
  • references

For developers/evaluators

  • retrieved chunks
  • source metadata
  • scores
  • debug details, if requested

“Debug details” means extra information for developers or evaluators, such as what was retrieved and how the model response was parsed.

Example output shape

A simplified StatsChat response

{
  "question": "In March 2026, what was Kenya’s annual inflation rate, and what drove it?",
  "answer": "Annual CPI inflation was 4.4% in March 2026...",
  "references": [
    {
      "title": "Consumer Price Indices and Inflation Rates – March 2026",
      "page_content": "...annual CPI inflation was 4.4 per cent...",
      "metadata": {"source": "KNBS", "period": "March 2026"},
      "score": 0.23
    }
  ]
}

The exact API details matter less today than the basic idea: answer + retrieved evidence to inspect.

Key terms

Source side

  • publication / PDF
  • JSON conversion
  • metadata

Search side

  • chunk
  • embedding
  • vector store
  • reference

What is a reference?

Usually not “the whole PDF”

In StatsChat, a reference is usually a retrieved chunk with metadata linking it back to a source document.

For the inflation example, a useful reference should help us check:

  • the KNBS publication
  • the March 2026 date
  • the relevant passage or page

This matters because the answer is only as good as the retrieved evidence.

Quick check

In this system, “references” are usually:

A. the model’s memory
B. whole PDFs only
C. retrieved chunks with metadata
D. citations generated after the answer

Suggested answer: C — retrieved chunks with metadata.

One question through the system

The LLM is not answering alone: it is answering with retrieved context.

Useful, improving prototype

What it shows well

  • real RAG system
  • practical case study
  • improving results
  • useful for adaptation planning

What still needs care

  • not a production service yet
  • needs evaluation before rollout
  • PDF processing can improve
  • retrieval can improve

The balanced message

Useful prototype, improving system, needs evaluation

StatsChat helps us understand what a real RAG system can do — and where careful evaluation and further development are still needed.

The aim is not to oversell it, but also not to undersell the underlying approach.

Quick check

Where does StatsChat fit best?

A. Replacing official publication processes
B. Helping users navigate and verify information in official reports
C. Producing final figures without checking sources
D. Answering any question about Kenya from the open web

Suggested answer: B — helping users find, navigate, and verify evidence from official reports.

Where StatsChat fits

Good fit

  • navigating report collections
  • finding definitions
  • locating evidence
  • repeated questions

Use with caution

  • weak evidence
  • subtle definitions
  • latest figures
  • high-stakes publication use

Quick check

Where can a StatsChat-style system fail?

A. document processing
B. retrieval
C. generation
D. all of the above

Suggested answer: D — failure can happen at any stage of the pipeline.

Optional discussion

What about AI search summaries?

Screenshot of a Google AI Overview answer about Kenya GDP

Useful, but check carefully

  • good for discovery
  • may use non-KNBS sources
  • may mix source types
  • may make verification harder

Connected topic

Can NSOs make their content easier for AI systems to use well?

  • clearer metadata
  • accessible HTML pages
  • well-structured publications
  • consistent page titles and summaries
  • explicit source and date information

This is a connected topic, not the main focus of today’s session.

Takeaways

  • StatsChat helps answer questions about Kenya and Kenyan statistics using KNBS evidence
  • it helps to split the system into prepare once and answer each question
  • references are based on retrieved evidence, not just model memory
  • the current prototype is useful and improving, but not production-ready
  • evaluation and human judgement matter throughout

Exercise for the lab

Before the lab, bring:

  1. one question you would want a StatsChat-like system to answer
  2. what documents it would need
  3. one reason you might trust — or distrust — the answer

Short bullet points are enough.