How to Build a Golden Dataset for Repeatable AI Evaluations

A golden dataset is a curated, version-controlled collection of test cases, where each item records an input your AI application must handle and, usually, a reviewed definition of what a correct response looks like. It is the fixed reference you run against before every meaningful change, so a prompt edit, model swap, or retrieval tweak can be judged against something more reliable than impressions. The concept is straightforward; the hard part is keeping the set representative for months while prompts, models, and users all change.

This guide is aimed at practitioners who already have some evaluation in place, perhaps a handful of manual checks, a spreadsheet, or a scoring script, and want a set they can trust and reuse. It covers what belongs in a golden dataset, how to source and annotate items, how to size and maintain the set, and how to use it to compare prompt versions.

What a golden dataset is, and what it is not

According to Langfuse’s guide to golden dataset evaluation, a golden dataset is a collection of test cases where each item records an input the application should handle and, usually, a reviewed definition of a correct response. The word “golden” signals that the items are curated and trusted: someone with domain knowledge confirmed the expected output, or deliberately decided the item does not need one because a reference-free check such as format, safety, or tone is the pass criterion. That trust is the entire value of the set. When an experiment shows a score drop, you want to know whether the regression is real rather than whether the test itself was sloppy.

The AI Evaluation Playbook frames the same artifact as a set of records representing an optimal or ideal user interaction with the system, effectively a performance target. It is used to benchmark the AI system over time or to compare performance across variants of that system. Each record typically includes a sample input, a paired output, and associated labels.

It helps to separate the golden dataset from things it is often confused with. Raw production logs are unlabeled and uneven in quality. Public benchmark suites measure general capability, not your product’s specific behavior. Unit tests check deterministic code paths, not probabilistic model output. A golden dataset sits between these: it is small enough to review by hand, specific enough to be meaningful for your application, and durable enough to be re-run on demand.

Scope and size: one behavior at a time

Langfuse recommends scoping a golden dataset to one component or behavior and sizing it for its job. Tens of items are enough for a fast pull-request gate, while a full regression set may run to hundreds or a thousand. Those two figures describe different jobs rather than a single target, and treating them as one number is a common source of friction.

A broad dataset that covers everything is slow to run and slow to review, which discourages people from running it at all. A narrow dataset that covers one prompt is fast but blind to regressions elsewhere in the pipeline. The practical resolution is layered: maintain a small gate set of high-signal cases that runs on every change, plus a larger regression set that runs before releases or after significant model changes. Both can live in the same store, separated by tags or a split field.

Before collecting items: rubric, metrics, and scorers

The AI Evaluation Playbook places the golden dataset after three earlier decisions: a rubric describing desirable system behaviors, metrics that quantify performance, and scorers that compute those metric values. The dataset exists to verify whether the solution is actually improving along the rubric’s dimensions. Skipping this ordering is expensive, because annotators without a rubric will disagree with each other, and a dataset without metrics cannot produce a comparable score.

The playbook also notes that creating the dataset is often the most time-consuming part of a first-level evaluation, and that it requires a cross-functional team: domain experts, annotators, product owners, and quality assurance. Inputs and outputs are usually annotated by human raters who either write ideal reference answers or define how a given output should be scored against the rubric. Budget for that review time explicitly rather than assuming it can be absorbed.

Where the items come from

Langfuse’s guidance on sourcing is direct: build from real traces first, with CSV imports and synthetic generation second. Real traces carry the messiness of actual usage, including the awkward phrasing, missing context, and edge cases that synthetic prompts tend to smooth away.

For applications that already exist in some form, the playbook describes a past-transaction approach. If human support staff have answered user queries in the past, for example, high-quality and representative question-answer pairs can be extracted from those non-AI interactions to form the golden dataset. LLMs can be used to pre-process or clean that historical data, but domain experts should remain involved in deciding what counts as a good answer.

A workable sourcing order looks like this: production traces from the component you are evaluating, then historical human-handled interactions, then imported CSVs from support tickets or review queues, and only then synthetic generation to fill gaps in coverage. Synthetic items are useful for probing specific failure modes, but they should be labeled as synthetic so their provenance stays visible.

A step-by-step build procedure

1. Choose one component or behavior. A retrieval step, a summarizer, or a classification prompt. Do not start with the whole pipeline.
2. Write the pass criteria first. Decide whether each item needs a reference answer or a reference-free check such as format, safety, or tone, following the distinction Langfuse draws.
3. Harvest candidate inputs. Pull real traces and historical interactions for that component. Aim for coverage of known failure modes rather than volume.
4. Clean and deduplicate. Near-duplicate items inflate scores and waste review time. LLM-assisted cleanup is acceptable at this stage if a human checks the result.
5. Annotate. A domain expert writes the expected output or the scoring rule for each item, consistent with the rubric.
6. Record metadata. Store the source trace identifier, the date, the component, and any tags that will let you filter later.
7. Validate the schema. Langfuse lists schema validation, deduplication, and item versioning as the maintenance essentials, and the same three apply at creation time.
8. Split the set. Tag items as gate or regression so the fast path stays fast.
9. Pin a version and record baseline scores. Without a baseline, later deltas mean nothing.

An illustrative example

The following is a schematic example of how items might be stored, not a record of a tested system. It uses JSON Lines, one item per line, which is easy to diff in version control.

{"id": "sum-014", "component": "ticket-summarizer", "input": "<raw support thread>", "expected": "<reviewed summary>", "criteria": ["lists all action items", "invents no dates", "under 120 words"], "source_trace": "prod-2026-09-14-...", "split": "gate", "item_version": 3}

The important properties here are the explicit criteria, the trace of origin, the split, and the item version. If a reviewer later decides the expected summary was wrong, bumping item_version preserves the history of why a score changed rather than silently rewriting the past.

Maintaining the set as production evolves

Langfuse’s central warning is that golden datasets fail when they stay frozen. Prompts, models, and users all change, so a set that was representative six months ago may now be measuring the wrong distribution. The recommended maintenance loop is to grow the dataset from production failures rather than from periodic rewrites: when something breaks in the wild, add a reviewed item that captures it, so the same failure cannot return unnoticed.

Three practices keep that loop from degrading. Schema validation catches malformed or missing fields before they reach a scoring run. Deduplication prevents near-identical items from dominating the score. Item versioning records changes to expected outputs so that a score change can be attributed to the system rather than to an edited test.

Comparing prompt versions with the dataset

Langfuse describes the comparison method precisely: run one experiment per prompt version against the same pinned dataset version, then read score deltas against a designated baseline. Two details do the heavy lifting. First, the dataset version is pinned, so every version is measured against identical items. Second, a baseline is designated in advance, so the direction of improvement is defined before results arrive.

The failure mode to avoid is changing the prompt and the dataset at the same time. If items are added or expected outputs edited between runs, a score difference cannot be attributed to the prompt. If you must update the dataset, re-run the baseline against the new pinned version before comparing anything else.

Limitations and tradeoffs

A golden dataset is not free. Annotation is the most time-consuming part of a first-level evaluation and depends on domain experts whose time is scarce, as the playbook notes. Small sets are fast but noisy, and a score difference of one or two items may not be meaningful. Reference-free checks can verify format, safety, and tone, but they cannot detect a confidently stated factual error, so items that require factual grounding still need reference answers or a retrieval-aware scorer.

There is also a subtler risk: repeatedly tuning prompts against the same items can overfit the system to that set, producing gains that do not generalize. Rotating in fresh production items and keeping a held-out portion of the set out of day-to-day tuning both reduce that risk. Finally, a golden dataset complements rather than replaces red-teaming and production monitoring; it verifies known behaviors, while those practices look for unknown ones.

A practical next step

Start small and concrete. Pick one component, pull twenty to thirty real traces from it, write the pass criteria, and have someone with domain knowledge annotate them. Run one baseline experiment, record the scores, and pin the version. Then add a gate split and a regression split, and commit to adding one reviewed item every time production surfaces a failure. That loop, more than the initial size of the set, is what makes evaluations repeatable.

If you are working through this on a local model or inside a WordPress automation stack, the constraints differ mainly in tooling rather than method: the same scoping, annotation, and versioning decisions apply. Readers are welcome to share their component, environment, dataset size, and the first score delta they observe in the community discussion.