Evals on your own traffic, always on
By the IrisAgent team · Last updated
Without evals, every LLM change is a guess
Prompt edits, model upgrades, and new workflows all ship on a hunch. Here is what that costs.
Customers find the failures first
Spot checks and thumbs-downs cover a sliver of traffic. A bad prompt change can run for weeks before anyone notices.
Every model release is a fire drill
A new model ships every few weeks. Testing one by hand takes an engineer days, so most releases go untested or get adopted on a launch post.
You pay for the top model everywhere
No one can prove a cheaper model is good enough, so the most expensive one stays on every workflow.
Benchmarks don't know your rules
Public leaderboards test generic tasks, not your formats, your policies, your products, or the languages your customers write in.
Ship on evidence, not intuition
AutoEval answers the three questions every AI team gets asked, with numbers from your own traffic.
Know it works
Every production call is evaluated for correctness and against your rules, so regressions show up the day they start, not when a customer complains.
Upgrade in days, not weeks
Replay your real prompts on any new model, including your own fine-tuned or self-hosted ones, and ship the upgrade on evidence.
Stop overpaying
Find the cheapest model that clears your quality bar for each workflow, with cost and latency next to every score.
It grades your call logs against default and custom rules
Default rules check every output for correctness against its own prompt, from day one. Custom rules check what your team cares about, written in plain English. Then replay the same prompts on any model before you switch.
- 1
Capture every call
The system prompt, the user prompt, and the model's output, from your logs or the Iris Control endpoint.
- 2
Grade correctness
A judge model checks whether each output correctly and completely answers the prompt it was given, with a score and an explanation.
- 3
Check your rules
Rules you write in plain English, such as "always include a link" or "reply in the user's language", pass or fail on every output.
- 4
Build the eval set
Pick the inputs that matter from your own calls: a workflow, a tenant, last month, or only the failures. Duplicates drop out.
- 5
Replay on every model
The same inputs run on each model you consider, including your own, and every output is graded the same way, with cost and latency.
- 6
Decide the upgrade
Switch only when a model beats what you run today on your own prompts and your own rules. Nothing changes without your approval.
Eval set · support-answers
Upgrade check
Current primary
in production
New release
candidate
Hold: grounded rate below current
Current fallback
in production
A golden eval set that stays live, not a spreadsheet that goes stale
Most eval sets are written once for a pilot and stop matching production within months. AutoEval keeps yours current, so every model comparison runs on a set that looks like this month's traffic.
Built from your real traffic
Inputs come from your own call logs, so the set reflects what your users actually ask today.
No redundancy
Duplicate and near-duplicate inputs are removed, so one common question can't skew the score.
Diverse by design
Balanced across workflows, tenants, languages, and edge cases, including the calls that failed.
Grows with your product
Launch a new use case or workflow and it gets its own slice of the set. Inputs that no longer match your product are retired.
Custom rules that fit each job
Any LLM call with a prompt and an output can be graded: answers, classifiers, agent steps, generated content, and voice transcripts.
Grounded answers
Support answers, internal search, RAG
Answer accuracy, hallucination resistance, cited sources, grounding score
Routing and tagging
Intent routing, ticket tagging, classification
Accuracy per label and which labels get confused with each other
Agents and tool use
Billing agents, triage agents, multi-step workflows
Right tool, right arguments, task completed, steps taken
Generated content
Scripts, summaries, drafts
Correctness against the prompt, plus your rules for format, tone, length, and required content
Voice and multilingual
Voice agents, conversations that switch languages mid-sentence
Language fidelity, code-mixed comprehension, turn accuracy, and response time per turn
Always next to price
Every workflow is scored on cost per call and response time too, so quality is never read without what it costs.
Bring your own model
In-house and fine-tuned models are graded on the same prompts and rules as frontier models. See where yours wins, where it falls behind, and which model should back it up.
A report for every new release
When a provider ships a model, replay your prompts on it and see whether it beats what you run today on your rules, and what it would cost at your volume.
Your rules, in plain English
Write the checks your outputs must pass, such as "always include a help article link" or "never promise a delivery date". Every output passes or fails each rule, with an explanation.
Newer and cheaper is not always better
On IrisAgent's own golden support eval set, a newer release at a third of the output price still scored below the model it would replace. Without evals, that upgrade ships. AutoEval holds it.
| Support eval set | GPT-5.6 Sol (today) | GPT-6.1 Sol (new release) |
|---|---|---|
| Output price per 1M tokens | $30 | $10 |
| Overall score | 93.3 | 90.6 |
| Answer accuracy | 98.8% | 95.3% |
| Cited sources | 76.5% | 70.6% |
Source: IrisAgent Support Model Arena, October 15, 2026.
Start light, go deeper when it pays off
AutoEval never needs to sit on your request path. Logs are redacted for PII and secrets before storage, and Iris Control can run in your own VPC.
| How you start | What you send | On your request path? | What you get |
|---|---|---|---|
| Sample eval report | 10 to 20 example calls: system prompt, user prompt, output, and your rules | No | A sample report: correctness and rule results, and the same prompts replayed on leading models, with price |
| Log copy | A recurring export of your LLM call logs | No | Evals on every call, a golden eval set kept current, reports by workflow, model, and rule, and model comparisons |
| Iris Control endpoint | Point your OpenAI-compatible client at Iris Control | Yes, server side | Everything above captured automatically, plus routing, failover, spend limits, and redaction |
Start with 20 examples
Send 10 to 20 example calls as a JSONL file: the system prompt, the user prompt, your model's output, and any rules the output must follow. We return a sample eval report, and the same prompts replayed on leading models, with price. Nothing changes in your call path.
AutoEval is part of Iris Control, IrisAgent's LLM control plane for cost, failover, and security on every LLM call.
Any questions?
We got you.
Run your first evals free
Send us 20 example calls. We grade them against default rules and yours, and replay them on leading models, with price.