Evaluation: how do we know whether StatsChat is working?

Slide deck

Session overview

This final core session brings the course together by looking at automated evaluation. We use the StatsChat testing workflow as the main case study: an audited set of KNBS questions is run through StatsChat, answers and references are recorded, and metrics are produced to show where the system works and where it needs improvement.

The main message is:

Evaluation is the feedback loop that lets us improve a RAG system safely.

The session is not just about one accuracy number. It is about how to design an evaluation set, how to separate answer quality from retrieval quality, how to test refusal behaviour, and how to use failures to improve the system.

We will also make the session more concrete by starting from a live or static StatsChat example. A single UI answer can be turned into an evaluation row: the user question becomes the query, the source passage becomes the expected evidence, and the inspection questions become repeatable checks.

Learning aims

By the end of the session, you should be able to:

explain the automated evaluation workflow used for StatsChat;
describe what a useful RAG benchmark row should contain;
distinguish answer metrics, guardrail metrics, and retrieval metrics;
explain why unanswerable questions are part of evaluation;
interpret benchmark results as development evidence rather than final proof;
draft evaluation questions for a StatsChat-like system in your own NSO context.

Why evaluation brings the course together

The course has followed the StatsChat pipeline from documents to answers:

PDFs → JSON conversion → chunks → embeddings → retrieval → generation → references/UI

Each week focused on one part of the evidence chain. Evaluation asks whether the whole chain is working.

Course theme	Evaluation question
Ingestion	Did we preserve the evidence?
Chunking / embedding	Did we create useful evidence units?
Retrieval	Did we select the right evidence?
Generation	Did the model use the evidence safely?
Evaluation	Can we measure and improve the whole system?

Evaluation helps us move beyond a few impressive demos. It lets us ask systematically where the system works, where it fails, and what should be improved next.

The StatsChat evaluation workflow

The automated evaluation workflow can be understood as a repeatable loop:

1. Start with audited questions and expected answers
2. Run StatsChat on each question
3. Record the generated answer and returned references
4. Compare against expected answers, documents, pages, and should-answer flags
5. Produce metrics and inspect failures
6. Improve the system and rerun the benchmark

This makes evaluation useful during development. After changing retrieval settings, prompts, model provider, thresholds, or the document corpus, the same benchmark can be run again to see whether performance improved or regressed.

What a benchmark needs

A useful RAG benchmark is more than a list of questions. A good row should include enough information to check both the answer and the evidence.

Field	Purpose
`query_id`	Stable identifier for tracking rows over time
`query_text`	The user question
`golden_answer`	Expected answer, if answerable
`should_answer`	Whether the system should answer or refuse
`relevant_doc_ids`	Expected source document(s)
`evidence_pages`	Expected page(s), where available
`source_text`	Human-audited evidence span or quote

The benchmark itself needs quality assurance. Questions should be clear, expected answers should be verifiable, source evidence should be specific, and ambiguous rows should be revised or excluded. Automated evaluation scales human judgement; it does not replace it.

Benchmark questions can come from several places: real user questions, questions written by subject experts, previous system failures, AI-drafted questions reviewed by humans, or a backwards process where we start from a source document/table and create a question around it. A small benchmark is useful for development, but a robust audited benchmark should grow over time toward hundreds of high-quality rows with good coverage across sources, dates, question types, and guardrail cases.

What the current StatsChat evaluation showed

The April benchmark contained 74 rows: 61 answerable and 13 unanswerable. The best full benchmark run achieved strong results, including correct refusal on all unanswerable rows.

However, the most important lesson is not just the final number. Earlier smaller benchmarks gave a more optimistic picture. When the benchmark was expanded to include under-tested report families, answerable accuracy dropped. This showed that the original set was not representative enough. Further retrieval/routing improvements then recovered performance on the expanded set.

This is a central evaluation lesson:

A benchmark should grow toward the real variety of user questions and source documents.

Useful metric families

For teaching purposes, it is useful to group metrics into three families.

Metric family	What it asks
Answer metrics	Did the system give the expected answer when it should answer?
Guardrail metrics	Did the system refuse when it should not answer?
Retrieval metrics	Did the system return the right document, page, or evidence?

A single overall accuracy score is not enough. A system can retrieve the right document but generate a bad answer. It can answer correctly but cite a weak reference. It can refuse a question that was actually answerable. Evaluation should help us diagnose these differences.

What automated evaluation can and cannot tell us

Automated evaluation is powerful because it is repeatable. It can track improvements, compare system changes, detect regressions, and make failures easier to inspect.

But it cannot fully replace expert judgement. It depends on the quality of the benchmark, may miss real user needs, and cannot by itself decide whether a system is ready for public release.

A useful way to read evaluation results is:

Evaluation finding	Possible next decision
High answer accuracy, weak page hit	Improve page-level grounding before public use
Strong refusals, weak answerable accuracy	Improve retrieval/generation for known facts
Good benchmark, small sample	Expand benchmark before relying on result
Repeated LLM variability	Repeat runs or stabilise generation
Weak guardrail performance	Keep internal-only until refusals improve

Improving evaluation in future

The current automated evaluation system is a strong first step, but future evaluation could improve by:

expanding the benchmark toward several hundred audited rows across more report families and question types;
adding more unanswerable, policy, causal, future, and cross-country boundary cases;
creating a held-out evaluation split as the benchmark grows;
repeating runs to measure LLM variability;
improving page-level and evidence-span checks;
capturing real user feedback and turning failures into new benchmark rows;
evaluating different user contexts, such as internal analysts versus public users.

The wider lesson is that evaluation should evolve with the system and with real usage.

Evaluation in another NSO setting

A different NSO may need to test different things: document formats, languages, update cycles, source systems, user groups, data governance rules, and acceptable risk.

A small first benchmark might include:

one monthly inflation question;
one annual GDP question;
one definition or methodology question;
one latest-publication question;
one table-heavy question;
one revised-figure question;
one refusal or guardrail question;
one non-PDF source question, if relevant.

The key is that evaluation should match the intended use. A tool for internal evidence search can have different requirements from a public-facing statistical assistant.

Mini-project: adapting StatsChat to your context

The mini-project is a short adaptation plan rather than a technical build. It does not need to be completed before the Thursday lab; the lab will help you start or refine it.

Your plan should cover:

Use case and user group — who would use the system, and for what task?
Likely document sources — which publications, PDFs, spreadsheets, web pages, databases, or APIs would it need?
Key risks and challenges — where might the system fail, and what would worry you most?
What you would keep from StatsChat — which parts of the current approach are useful for your context?
What you would change — what would need to be adapted for your sources, users, languages, infrastructure, or governance?
How you would evaluate success — what questions, metrics, user feedback, or pilot criteria would show whether it is working?

You can still include a small set of evaluation questions, but the aim is broader: to think through how a StatsChat-like system would need to be adapted and tested in a real NSO setting.

Thursday lab

The Thursday session will be an evaluation question design clinic. You will not need to have a finished mini-project before the lab. Instead, bring an idea for a use case or a few possible questions, and we will discuss how to turn them into a useful evaluation set.

Between-session task

Week 9 between-session exercise: start your adaptation plan

This short task prepares for the Thursday evaluation design / mini-project clinic. You do not need to finish the mini-project before the lab.

Task

Choose one possible StatsChat-like use case for your own organisation or statistical area.

Write brief notes on:

Use case and user group — who would use it, and for what?
Likely document sources — what publications, tables, web pages, databases, or APIs would it need?
Key risk or challenge — what is one thing that could go wrong?
One evaluation question — what is one question you would use to test the system?
One guardrail question — what is one question the system should refuse or answer cautiously?

A few bullet points are enough. The Thursday lab will help you turn these notes into a more complete mini-project plan.

Mini-project brief

Mini-project: adapting StatsChat to your context

The mini-project is a short adaptation plan for how a StatsChat-like system might be useful in your own NSO or statistical area. It does not need to be completed before the Thursday lab. The lab is a clinic to help you start or refine the plan.

This is not a technical build. A one-page outline is enough. Bullet points are fine.

Suggested structure

1. Use case and user group

Who would use the system, and for what?

Examples:

internal analysts searching official publications;
public users looking for headline figures;
staff answering user enquiries;
managers preparing briefings;
methodology teams explaining definitions and methods.

2. Likely document sources

What would the system need to search?

Consider:

PDF publications;
statistical bulletins;
Excel tables;
HTML pages;
methodology documents;
databases or APIs;
internal guidance, if appropriate.

3. Key risks and challenges

Where might the system fail?

Consider:

poor PDF/table extraction;
old report editions being retrieved;
ambiguity around “latest” figures;
multiple similar numbers in one source;
unsupported policy advice or forecasts;
language/terminology issues;
traceability and user trust;
data governance or hosting constraints.

4. What you would keep from StatsChat

Which parts of the current approach seem useful for your context?

For example:

RAG over official publications;
references and source previews;
refusal behaviour for unsupported questions;
automated evaluation with audited questions;
separate retrieval and generation checks;
internal pilot before public release.

5. What you would change

What would need to be adapted?

For example:

source formats;
metadata fields;
chunking strategy;
languages;
prompts and terminology;
hosting model;
user interface;
governance or review process.

6. How you would evaluate success

What would show that the system is working well enough?

You could include:

5–10 example evaluation questions;
at least one question the system should refuse or answer cautiously;
expected sources or evidence;
answer accuracy and refusal behaviour;
source traceability checks;
user feedback from an internal pilot;
response time, cost, or maintainability requirements.

Suggested length

One page is enough. The most useful output is a clear plan that explains what you would build, what you would keep or change, and how you would know whether it is working.

Summary

Evaluation closes the loop. Across the course, you have learned how to reason about the whole RAG evidence chain: source documents, processing, chunking, retrieval, generation, references, and evaluation.

That is useful whether you adapt StatsChat directly or simply need to use AI systems more critically. A useful RAG system is not just built once; it is tested, diagnosed, improved, monitored, and adapted to real documents and users.