Support Model Arena
Which LLM actually wins on customer support? We score models on real tickets inside IrisAgent — resolution when the answer exists, hallucination resistance when it does not, plus latency and cost. Same retrieval. Same tools. Same prompts. Only the model changes.
Four axes that matter in support
Every model runs inside the same production IrisAgent stack. Same retrieval, same tools, same prompts. Only the model changes.
On real support tickets that have a correct answer in the knowledge base, does the model resolve correctly?
On tickets that should not be answered from the KB, does the model correctly decline or hand off instead of inventing?
p50 / p90 / p99 response time on the resolution suite.
Published input and output price per 1M tokens at the time of the run.
We also run FAQ / intent-matching evals in the same package. They inform routing; the public table below keeps the buyer-facing Pareto clear: resolve correctly, decline correctly, stay fast enough, stay affordable.
This week's read
Snapshot from September 22, 2026. Rankings reshuffle; the system around the model is the product.
100% resolution with 95% hallucination resistance — tied at the top of the board with Claude Opus 5.
98.8% resolution with perfect hallucination resistance on the decline suite.
97.6% resolution, 95% hall resistance, p50 3.14s.
96.5% / 100% at $0.15 / $0.50 per 1M tokens.
Leaderboard
Sorted by resolution, then hallucination resistance, then p50 latency. Balance score is the simple average of resolution % and hallucination resistance %.
| # | Model | Resolution | Hall. resist. | Balance | Latency p50 | $/1M in · out |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 5Anthropic | 100.0% | 95.0% | 97.5 | 9.65s | $5 · $25 |
| 2 | Claude Fable 5Anthropic | 100.0% | 95.0% | 97.5 | 9.93s | $10 · $50 |
| 3 | GPT-5.5OpenAI | 100.0% | 85.0% | 92.5 | 5.60s | $5 · $30 |
| 4 | GPT-5.6 SolOpenAI | 98.8% | 100.0% | 99.4 | 6.50s | $5 · $30 |
| 5 | Gemini 3.5 FlashGoogle | 98.8% | 90.0% | 94.4 | 7.47s | $1.50 · $9 |
| 6 | Gemini 3 FlashGoogle | 98.8% | 70.0% | 84.4 | 8.61s | $0.5 · $3 |
| 7 | Kimi K2.7 CodeMoonshot / Fireworks | 97.6% | 100.0% | 98.8 | 9.19s | $0.95 · $4 |
| 8 | Claude Fable 5.1Anthropic | 97.6% | 100.0% | 98.8 | 11.83s | $10 · $50 |
| 9 | GLM 5.3Zhipu / Fireworks | 97.6% | 100.0% | 98.8 | 22.72s | $1.40 · $4.40 |
| 10 | GPT-5.6 TerraOpenAI | 97.6% | 95.0% | 96.3 | 3.14s | $2 · $12 |
| 11 | Grok 4.5xAI | 96.5% | 100.0% | 98.2 | 7.22s | $2 · $6 |
| 12 | GLM 5.3 FlashZhipu / Fireworks | 96.5% | 100.0% | 98.2 | 10.25s | $0.15 · $0.5 |
| 13 | Gemini 3.1 Pro ThinkingGoogle | 96.5% | 100.0% | 98.2 | 12.06s | $2 · $12 |
| 14 | Kimi K3Moonshot / Fireworks | 96.5% | 100.0% | 98.2 | 20.97s | $3 · $15 |
| 15 | GPT-6 AstraOpenAI | 96.5% | 95.0% | 95.8 | 4.93s | $10 · $50 |
| 16 | o4-miniOpenAI | 92.9% | 100.0% | 96.5 | 7.16s | $1.10 · $4.40 |
| 17 | Gemini 3.7 FlashGoogle | 91.8% | 100.0% | 95.9 | 3.02s | $0.75 · $3.75 |
| 18 | Gemini 3.6 FlashGoogle | 91.8% | 100.0% | 95.9 | 7.89s | $1.50 · $7.50 |
Scores move when we re-run the suite. Use the shape of the board (who sits high on both axes) more than any single percentage. IrisAgent routes traffic from this living board, not from a one-time bake-off.
Deeper narrative and method: Best LLMs for a customer support chatbot (2026) · How IrisAgent measures accuracy
How we keep the board honest
- Production-shaped harness: retrieval, tools, and prompts match what customers get — not a stripped chat playground.
- Two graded suites: answerable tickets (resolution) and should-decline tickets (hallucination resistance).
- One variable only: the LLM. Everything else is held constant.
- Latency and published token prices sit next to quality so speed and cost are visible tradeoffs, not footnotes.
- Living board: IrisAgent defaults and customer routing follow this eval loop as models move.
FAQ
What is the IrisAgent Support Model Arena?
A living leaderboard of LLMs scored on real customer-support work inside IrisAgent. Every model gets the same retrieval, tools, and prompts. We score resolution, hallucination resistance, latency, and price so you can see the tradeoffs that actually matter in production.
How is this different from a generic LLM leaderboard?
Public model leaderboards optimize for exams and coding puzzles. Support cares about two axes that fight each other: resolving tickets when the answer is in your knowledge base, and declining when it is not. This arena measures both, on real tickets, in the same stack we ship.
Why publish resolution and hallucination together?
A model that answers everything looks great on resolution and fails when it should hold back. A timid model looks safe and under-resolves. The useful models sit high on both. That is the Pareto frontier buyers should care about.
Do you also evaluate intent matching?
Yes. FAQ and intent-matching evals live in the same case-answer package. The public table focuses on resolution × hallucination × latency × cost because those four decide most support routing choices. Intent suites refresh on the same cadence.
How often is this updated?
This snapshot is dated September 22, 2026. We refresh when we re-run the case-answer leaderboard (typically as new frontier models land or weekly when the board moves). IrisAgent production routing follows the living board, not a one-off bake-off.
Can I use a different model with IrisAgent?
Yes. IrisAgent is model-agnostic. We pick defaults from this board and can route by customer, channel, and latency budget. The arena is how we keep that choice honest.
Want this board running on your tickets?
IrisAgent grounds answers in your KB, declines when it should, and keeps model choice on a living eval — not a launch-day screenshot.