Golden Dataset for LLM Evaluation: How to Build One From Production Logs

A golden dataset is the reference set of representative inputs, with labels or grading rules, that you score every LLM model and prompt change against. Build it from production logs: redact them, sample across every workflow, remove near-duplicates, add labels, and keep updating it as your traffic changes.

The payoff shows up at upgrade time. On IrisAgent's own support eval set, GPT-6.1 Sol costs a third of GPT-5.6 Sol's output price, yet it scored 90.6 overall against 93.3. Without a golden dataset, that upgrade ships on the strength of the launch post and the lower bill.

You already know you need evals. The hard part is the set itself, because most teams write one for a pilot and never touch it again. This guide shows you how to build a golden dataset from real traffic, how big it needs to be, and how to stop it going stale. If you want to skip the build, AutoEval generates a golden eval set from your LLM call logs and keeps it up to date.

Key Takeaways

• Start from real traffic. Production logs contain the phrasing, typos, and language mixes that hand-written examples miss.

• Stratify before you sample. A random sample over-represents your easiest, highest-volume path and under-represents what breaks.

• Size the set to the decision. At 90% accuracy, 100 items gives a margin of about 6 points. 1,000 items brings it to about 2 points.

• Keep passing items and add every new failure. Passing items prove behavior still holds, and fresh failures are your highest-value additions.

• Treat staleness as the main risk. Strong eval scores next to weakening production signals mean the set no longer matches your traffic.

What Is a Golden Dataset?

A golden dataset is a fixed, versioned collection of test inputs paired with the expected result or a grading rule for each one. In LLM evaluation, it works as your yardstick. You run every candidate model or prompt on the same items, grade the outputs the same way, and compare the scores.

Each item needs more than an input and an answer, however. LLM outputs vary, so many items need a rubric instead of one exact string. A useful item records where it came from and why it is in the set:

You'll also see the terms golden set, eval set, and reference dataset. They describe the same idea. Teams tend to say "golden" when the labels have passed human review and the set serves as the agreed source of truth.

How It Differs From Similar Datasets

Why a Golden Dataset Beats Spot Checks and Benchmarks

A golden dataset turns "this model feels better" into a number you can defend in a meeting. Public leaderboards test generic tasks, so they can't tell you whether a model follows your refund policy or replies in your customer's language.

Consider the Arena result above in more detail. GPT-6.1 Sol costs $10 per 1M output tokens against $30 for GPT-5.6 Sol. However, its answer accuracy came in at 95.3% against 98.8%, and its cited sources score at 70.6% against 76.5%. The answer suites behind those numbers use anonymized real tickets, split into tickets that should be answered and tickets that must be declined. A generic benchmark has no way to surface that gap on support work.

Here is a hypothetical example of the failure this prevents. Maya runs AI platform at a mid-size SaaS company and tests prompt changes against 30 examples she wrote during the pilot. In March, her team launches in Germany. Her set contains no German tickets, so a prompt edit that breaks German replies passes every check. She learns about it three weeks later from a support escalation.

The vendors agree on the direction. OpenAI's evaluation guide tells teams to cover typical, edge, and adversarial cases, and warns against datasets that don't reflect real traffic. Anthropic's guide says evals should match the distribution of your real-world tasks.

How Big Should a Golden Dataset Be?

Start with 20 to 50 reviewed items, then grow to a few hundred or more as decisions get more expensive. Langfuse's guide suggests about 10 items to explore one issue and 100 to 1,000 for CI checks on larger changes.

The right size depends on the smallest score gap you need to detect. The table below uses the standard 95% margin of error for a model scoring around 90%. Treat it as a planning estimate, because comparing two models on identical items is somewhat more precise.

Two practical rules follow from that math. First, the margin applies to each slice you read. A 500-item set with 40 German tickets gives you a weak German score, however strong the overall number looks. Second, don't over-read small gaps. The Support Model Arena treats a gap of a point or two as noise, so ties share a rank.

Volume only helps when items differ, though. Anthropic favors more questions with automated grading over fewer hand-graded ones, while Langfuse says coverage matters more than raw count, since near-duplicates add little. Both hold: add volume, then remove the repeats.

Want to see this on your own calls? Get a sample eval report from AutoEval by sending 10 to 20 example calls. Nothing changes in your call path.

How to Build a Golden Dataset From Production Logs

These eight steps take you from raw logs to a set you can trust. Run them once per workflow rather than mixing everything into one pile.

1. Name the Decision the Set Must Support

Write down the decision first: approving a model upgrade, catching prompt regressions, or picking a cheaper model for one workflow. The decision sets your required margin of error and which metrics matter. A ticket tagging set and a grounded answer set need different labels, so keep them as separate slices.

2. Export and Redact Your Logs

Pull the system prompt, the user prompt or conversation, and the output for each call. Then strip PII and secrets before anyone reviews them. Also check whether your data agreements allow this use. For support data, that includes names, emails, account numbers, and anything pasted from a customer's screen.

3. Stratify Before You Sample

Split your logs by workflow, language, customer segment, and outcome, and then sample inside each group. A plain random sample mirrors your traffic, so your most common and easiest question dominates. Sample rare but costly groups above their traffic share, and report their scores separately so they don't distort the overall number.

4. Pull In the Failures on Purpose

Good logs tell you where to look. Add calls that led to thumbs-downs, escalations, repeat contacts, retries, or blocked outputs. Langfuse ranks recent failures as the most valuable new items for a set. In support, that means tickets a human had to reopen or correct.

5. Remove Near-Duplicates

Cluster similar inputs by meaning, using embeddings or a judge model, and keep the hardest one or two from each cluster. Otherwise, 200 password reset tickets will drown out the 10 billing disputes that decide whether a model is safe to ship. Removing them also lowers the cost of every run.

6. Add Labels and Rules With Your Domain Experts

Have the people who know the right answer label the first batch. For support, that is a senior agent or QA lead, not the ML engineer. Use the lightest label that works:

1. Exact match for routing and tagging, where one label is correct.

2. Must-decline flag for questions your knowledge base cannot answer, where a pass means the model hands off.

3. Reference answer when a correct reply exists and wording can vary.

4. Plain-English rules for format, tone, citations, and policy, such as "never promise a delivery date."

5. Correctness rubric for everything else, scored by a judge model.

Expect your criteria to shift during this step. Research by Shankar and colleagues found a circular dependency: "users need criteria to grade outputs, but grading outputs helps users define criteria." So label a first batch, revise the rules, and only then label the rest. Remove items where your experts cannot agree, or flag them and keep them out of the headline score.

7. Fill the Gaps With Edge and Adversarial Cases

Logs miss what hasn't happened yet. Add prompt injection attempts, inputs that mix languages, overly long pastes, and questions about products you don't sell. Synthetic items are fine here, as long as they get the same review as real ones and you label their source. Our seven-part test plan for AI support agents lists more cases worth covering.

8. Calibrate Your Judge, Then Record a Baseline

If a judge model grades your outputs, test the judge first. Run it on your human-labeled items and compare the verdicts, because OpenAI advises calibrating automated scoring against human feedback. Anthropic recommends a judge that differs from the model that generated the output. Finally, score your current production model, and save that result with the dataset version.

How to Keep a Golden Dataset From Going Stale

A golden dataset starts decaying the day you finish it, since your products, policies, and users keep changing. Maintenance is the real cost, and it is where most sets fail.

Here is another hypothetical. Dev's team freezes a set of 300 items in January. By June, they have added a billing agent and retired a legacy plan. The set still shows 94%, while escalations on billing tickets are climbing. Nothing in the eval is wrong. It's measuring a product that no longer exists.

Use these rules to avoid that outcome:

• Add failures weekly. Route production failures through review and into the set.

• Give new workflows their own slice. Add the slice before launch, not after the first complaint.

• Retire deliberately. Archive items whose behavior no longer exists or whose expected answer is now wrong.

• Keep passing items. A set of only hard failures tells you nothing about regressions on what works.

• Version everything. Pin a version when you compare runs, because scores on different versions are not comparable.

• Keep a fast subset. Run tens to low hundreds of items per change, and the full set nightly or before a release.

• Watch for drift. Compare your eval scores to production signals like escalations and groundedness. When they disagree, refresh the set first.

Common Golden Dataset Mistakes

Most golden dataset problems come from a handful of repeat errors, and all of them are cheap to fix early:

1. Writing every item by hand. Your team writes clean questions. Your customers don't.

2. Sampling only recent happy paths. The scores look great until the first hard ticket arrives.

3. Leaving out must-decline cases. A model that answers everything looks helpful and hallucinates on the rest.

4. Reading only the overall score. One number hides a failing language or workflow.

5. Editing items to make a run pass. Review set changes the way you review code.

6. Testing on data used for fine-tuning. The model has seen those answers, so its score is inflated.

7. Ignoring price and latency. A two-point gain rarely justifies triple the cost, and you can't judge that without both numbers.

How AutoEval Builds and Maintains Your Golden Dataset

If maintaining all of this isn't where your team wants to spend its time, AutoEval does the work. It is the evaluation module of Iris Control, IrisAgent's LLM control plane.

AutoEval builds your golden eval set from your own call logs. It removes duplicate and near-duplicate inputs, balances the set across workflows, tenants, languages, and edge cases, and retires inputs that no longer match your product. New workflows get their own slice as you launch them.

A judge model then returns a correctness verdict, a score from 0 to 5, and an explanation for each output, plus pass, fail, or not applicable for each rule you write in plain English. You can replay the set on any model, including in-house and fine-tuned ones, with cost and latency next to every score. It runs on a copy of your logs, never on your request path, so it adds no latency.

A code-first framework or an observability platform is the better fit if you want tests in your own CI or deep trace debugging, and your team has the time to own the set.

Next Steps

A golden dataset earns its keep the first time it stops a bad upgrade. To recap:

• Build it from redacted production logs, stratified by workflow, language, and outcome.

• Size each slice to the smallest score gap you need to detect.

• Add failures and must-decline cases, and have domain experts label them.

• Version the set, retire what no longer applies, and watch for drift.

This week, export 100 recent calls from your highest-volume workflow and label them with your senior agent or domain expert. You will find your first missing rule within the hour. When you are ready to compare models on those calls, see how we rank LLMs for customer support as a worked example.

Prefer not to maintain the set yourself? Send 20 example calls for a free AutoEval sample report. We grade them against default rules and yours, then replay them on leading models, with price.