Support Model Arena
Which LLM is best at each support job? We grade models inside the production IrisAgent stack on answering, declining, citing, routing, and tagging. Same retrieval, same tools, same prompts. Only the model changes.
Which model is the best all-round AI support agent?
Weighted blend of every category: 30% answer accuracy, 30% hallucination resistance, 15% cited sources, 15% intent routing, 10% ticket tagging. Only models scored on every category are ranked.
Overall vs speed
p50 response time on the answer suite. Top-left is fast and accurate.
Overall vs price
Blended $ per 1M tokens at a 3:1 input:output mix. Top-left is cheap and accurate.
Full leaderboard
Every model and every category in one table. Click a column to sort and a model for latency percentiles and run dates. The highlighted column is the category you picked above.
| # | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 97.0% | 100.0% | 100.0% | 90.6% | 97.5% | 87.8% | 10.3s | $4 · $20 | |
| 2 | 95.8% | 98.8% | 100.0% | 85.9% | 95.0% | 89.8% | 5.1s | $2 · $6 | |
| 3 | 95.3% | 100.0% | 100.0% | 77.6% | 97.5% | 90.8% | 4.4s | $5 · $30 | |
| 4 | 95.1% | 97.6% | 100.0% | 82.4% | 97.5% | 88.8% | 11.9s | $2 · $12 | |
| 5 | 94.6% | 97.6% | 100.0% | 83.5% | 92.5% | 88.8% | 10.1s | $0.15 · $0.5 | |
| 6 | 94.1% | 97.6% | 95.0% | 87.1% | 95.0% | 89.8% | 9.1s | $4 · $20 | |
| 7 | 93.9% | 97.6% | 100.0% | 76.5% | 95.0% | 88.8% | 4.8s | $10 · $50 | |
| 8 | 93.0% | 92.9% | 100.0% | 82.4% | 92.5% | 88.8% | 4.8s | $2 · $10 | |
| 9 | 92.8% | 94.1% | 95.0% | 84.7% | 95.0% | 90.8% | 12.4s | $0.22 · $0.66 | |
| 9 | 92.8% | 91.8% | 100.0% | 82.4% | 95.0% | 86.7% | 10.9s | $2 · $6 | |
| 11 | 92.7% | 100.0% | 90.0% | 85.9% | 95.0% | 85.7% | 9.9s | $10 · $50 | |
| 12 | 92.5% | 100.0% | 90.0% | 80.0% | 97.5% | 88.8% | 5.4s | $5 · $30 | |
| 13 | 92.4% | 91.8% | 100.0% | 80.0% | 95.0% | 85.7% | 3.2s | $0.75 · $3.75 | |
| 14 | 91.9% | 100.0% | 90.0% | 75.3% | 97.5% | 89.8% | 3.7s | $2 · $10 | |
| 15 | 91.5% | 98.8% | 90.0% | 85.9% | 92.5% | 80.6% | 19.5s | $3 · $15 | |
| 16 | 90.9% | 88.2% | 100.0% | 80.0% | 92.5% | 85.7% | 39.0s | $2 · $6 | |
| 17 | 90.7% | 94.1% | 95.0% | 77.6% | 92.5% | 84.7% | 4.6s | $0.1 · $0.5 | |
| 18 | 90.6% | 95.3% | 95.0% | 70.6% | 95.0% | 86.7% | 5.5s | $2 · $10 | |
| 19 | 90.4% | 88.2% | 100.0% | 72.9% | 95.0% | 87.8% | 7.3s | $1.50 · $7.50 | |
| 20 | 90.2% | 92.9% | 95.0% | 72.9% | 95.0% | 86.7% | 6.8s | $1.10 · $4.40 | |
| 21 | 90.0% | 98.8% | 85.0% | 82.4% | 95.0% | 82.7% | 7.3s | $1.50 · $9 | |
| 21 | 90.0% | 94.1% | 90.0% | 77.6% | 95.0% | 88.8% | 4.9s | $0.1 · $0.5 | |
| 23 | 89.9% | 97.6% | 85.0% | 76.5% | 95.0% | 93.9% | 2.6s | $2 · $12 | |
| 24 | 89.6% | 96.5% | 90.0% | 71.8% | 92.5% | 89.8% | 4.4s | $0.2 · $1.20 | |
| 25 | 89.5% | 84.7% | 100.0% | 75.3% | 95.0% | 85.7% | 3.6s | $0.75 · $3.75 | |
| 26 | 86.6% | 98.8% | 70.0% | 83.7% | 97.5% | 87.8% | 8.6s | $0.5 · $3 | |
| n/a | 98.8% | 100.0% | 87.1% | n/a | 79.6% | 12.2s | $10 · $50 | ||
| n/a | 91.8% | 100.0% | 77.6% | n/a | n/a | 7.5s | $0.1 · $0.5 | ||
| n/a | 100.0% | 85.0% | 87.1% | n/a | n/a | 8.7s | $5 · $25 | ||
| n/a | 95.3% | 95.0% | 83.5% | n/a | n/a | 5.0s | $2 · $10 | ||
| n/a | 98.8% | 95.0% | 85.9% | 95.0% | n/a | 23.0s | $1.40 · $4.40 | ||
| n/a | n/a | n/a | n/a | 95.0% | 85.7% | n/a | n/a | ||
| n/a | n/a | n/a | n/a | 95.0% | 82.7% | n/a | $0.1 · $0 |
Scores move when we re-run the suites. Read the shape of the board (who sits high across categories) more than any single percentage. IrisAgent routes production traffic from this living board, not from a one-time bake-off.
Deeper narrative and method: Best LLMs for customer support chatbots · How IrisAgent measures accuracy
Methodology
Every model runs inside the same production IrisAgent stack. Same retrieval, same tools, same prompts. Only the model changes.
Real support tickets whose answer exists in the knowledge base. A pass means the model resolved the ticket with a correct, grounded answer.
Tickets that must not be answered from the knowledge base. A pass means the model declined or handed off instead of making something up.
Same answerable tickets, stricter bar: the answer must be correct and cite the expected knowledge base article.
Multi-turn chats against configured FAQ intents and workflows. Checks cover picking the right intent, ignoring near-miss intents, continuing an active workflow, and escalating on request.
Exact-match ticket classification across three production tag catalogs from B2B customers, each with its own taxonomy.
- Production-shaped harness: retrieval, tools, and prompts match what customers get, not a stripped chat playground.
- Answer suites use anonymized real tickets, split into answerable tickets and tickets that must be declined.
- Latency and published token prices sit next to quality so speed and cost are visible tradeoffs, not footnotes.
- Exact-match grading, single runs. A gap of one or two checks is within noise, so ties share a rank.
- Living board: IrisAgent defaults and customer routing follow this eval loop as models move.
FAQ
What is the IrisAgent Support Model Arena?
A living leaderboard of LLMs scored on real customer-support work inside IrisAgent. Every model gets the same retrieval, tools, and prompts. We grade answer accuracy, hallucination resistance, cited sources, intent routing, and ticket tagging, and show latency and price next to quality.
How is this different from a generic LLM leaderboard?
Public leaderboards optimize for exams, coding puzzles, or crowd preference. Support has its own failure modes: answering when the knowledge base is silent, citing the wrong article, routing a chat to the wrong workflow, or mis-tagging a ticket. Each category here measures one of those jobs on production-shaped data in the same stack we ship.
What does each category measure?
Answer accuracy: Real support tickets whose answer exists in the knowledge base. A pass means the model resolved the ticket with a correct, grounded answer. Hallucination resistance: Tickets that must not be answered from the knowledge base. A pass means the model declined or handed off instead of making something up. Cited sources: Same answerable tickets, stricter bar: the answer must be correct and cite the expected knowledge base article. Intent routing: Multi-turn chats against configured FAQ intents and workflows. Checks cover picking the right intent, ignoring near-miss intents, continuing an active workflow, and escalating on request. Ticket tagging: Exact-match ticket classification across three production tag catalogs from B2B customers, each with its own taxonomy.
How is the Overall score calculated?
Weighted blend of every category: 30% answer accuracy, 30% hallucination resistance, 15% cited sources, 15% intent routing, 10% ticket tagging. Only models scored on every category are ranked. Ties share a rank, and small gaps (one or two tickets) are within run-to-run noise.
What are decision models?
Decision APIs such as Jev and OpenAI Decisions only choose among options you give them. They cannot write answers, so they appear in Intent routing and Ticket tagging but not in the answer categories or Overall.
How often is this updated?
This snapshot is dated October 8, 2026. The answer suites re-run weekly and whenever a new frontier model lands; routing and tagging re-run when the board moves. IrisAgent production routing follows the living board, not a one-off bake-off.
Can I use a different model with IrisAgent?
Yes. IrisAgent is model-agnostic. We pick defaults per task from this board and can route by customer, channel, and latency budget.
Want this board running on your tickets?
IrisAgent grounds answers in your KB, declines when it should, and picks the right model per task from a living eval, not a launch-day screenshot.