So, Which LLMs Are the Best for Building a Customer Support Chatbot? (2026 Refresh)

A while back we published a benchmark answering one deceptively simple question: if you are building an AI chatbot to answer support tickets, which large language model should sit at the center of it? Back then the contenders were GPT-4, GPT-3.5, Llama 2, and Mistral. (Here is that original post.)

A lot has changed. We now have GPT-5.x, Claude Opus 4.x, the Gemini 3 family, Grok 4, and a wave of open-weight models that have quietly closed the gap. (We wrote about the open-source surge here.)

So we re-ran the test on the latest models, scored against real support data. And the headline finding is not “model X won.” It is that the two things you actually care about in support pull in opposite directions, and picking a model is really about choosing where you want to sit on that line.

The one thing most LLM benchmarks get wrong

Public leaderboards measure the wrong things for support. Knowing a model can pass the bar exam or solve competition math tells you almost nothing about whether it will correctly answer “why was I double charged last month?” without inventing a refund policy that does not exist.

In customer support, two things matter, and they fight each other:

1. Resolution accuracy. When the customer’s question has a correct answer in your knowledge base, does the model find it and answer correctly?

2. Hallucination resistance. When the question cannot be answered from your knowledge base, does the model correctly hold back, or does it confidently make something up?

Here is the uncomfortable part. A model that answers everything will score brilliantly on resolution and terribly on hallucination, because it never knows when to stop. A model that is cautious will score brilliantly on hallucination and terribly on resolution, because it bails on questions it could have answered. The same trait that makes a model safe makes it unhelpful, and vice versa.

That is why a single number on a generic leaderboard is useless here. You have to measure both, then decide how you want to balance them.

How we tested

We ran each model inside the exact setup we ship to customers, not a stripped-down demo:

• Real, anonymized support tickets

  drawn from live support queues across our customer base: billing, account access, product how-to, troubleshooting, and policy questions. Across both evals that is roughly 100 real tickets, not synthetic prompts.

• Two separate graded sets:

  • A resolution set that genuinely have a correct answer in the customer’s knowledge base. We check whether the model resolved it correctly, graded against the real article content, not just the title.

  • A hallucination set of around 17 tickets that deliberately have no good answer (vague internal forwards, unrelated notifications, requests needing data the bot does not have). Here the correct behavior is to decline or hand off, not to invent an answer.

• Retrieval-augmented and agentic. Every model got the same agentic setup: it can call tools and retrieve from the customer’s real knowledge base before answering. This is how a production support bot actually works. No model was asked to answer from memory.

• Everything held constant except the model. Same retrieval, same prompt, same tools. The only variable we changed was the LLM.

We grade both and weight them equally, then look at where each model lands on the two axes together. The best model is not the one that wins either metric outright. It is the one that sits high on both at once: helpful and safe in the same breath.

The 2026 results

We are not going to hand you a table of raw scores. They move every time we re-run the suite, and frankly they say as much about our internal grading pipeline as about how a model will behave on your tickets. What holds up is the shape of the field. So here it is: every model plotted by how often it resolves answerable tickets (the horizontal axis) against how reliably it holds back when it should not answer (the vertical axis). Up and to the right is better, helpful and safe at the same time.

A few things jump out.

The leaders are not who you would expect. The model sitting furthest into the top-right corner, strong on both axes at once, is an open-weight model: DeepSeek V4 Pro, at a fraction of the price of the frontier flagships. Right behind it is Claude Opus 4.6. The “best” model for support is the one that refuses to be either reckless or timid, and increasingly that is no longer the most expensive name on the list.

Now find the dot alone in the bottom right. That model resolves almost every answerable ticket, a near-perfect score on the horizontal axis, and yet it would rank dead last overall, because it answers everything, including the questions it should have declined. On a generic leaderboard it looks like a winner. In production it is the model most likely to confidently tell a customer something false. This is the entire point of plotting both axes instead of one.

The tradeoff, made concrete

The clearest way to see the tension is to look at the two ends of the chart.

• Gemini 3 Flash

  is the eager intern, alone in the bottom right. It resolves nearly everything, but its hallucination resistance is the worst on the chart. It will answer anything you put in front of it: wonderful when the answer exists, dangerous when it does not. On a generic leaderboard its resolution score makes it look like the winner. In production it is the model most likely to make something up.

• DeepSeek V4 Pro

  sits at the balanced end, up in the top right. It resolves nearly as much as the eager models, and in exchange it almost never makes things up. That combination, the strongest on the chart, is exactly what a high-stakes support team wants, and here it comes at open-weight prices.

Same dial, two positions. The eager model maximizes one number at the cost of the other. The balanced model gives up a few points of raw resolution to stay safe, and wins overall because in support, being wrong is far more expensive than being briefly unhelpful.

The important nuance: where a model lands is not purely a fixed property of the model. It is also a tuning decision. We saw this directly. An earlier configuration of Claude Opus 4.8 was truncating its own output and looked like it resolved only a fraction of its tickets. Once we fixed it, the same model jumped back into the front of the pack. Move the dial (prompt, retrieval thresholds, output limits, when to hand off) and the same model behaves very differently. The model sets your range. Your configuration sets your position on it.

Speed is the hidden third axis

Notice that two of the best all-rounders (Claude Opus 4.6 and Kimi K2.6) are also among the slowest to respond, on the order of a minute per answer. For an email or case-answer workflow, that is invisible. For a live chat widget where a customer is staring at a typing indicator, it is a dealbreaker.

This is why the fast tier matters so much. Grok 4.3, Gemini 2.0 Flash, and Claude Haiku 4.5 all land respectably on both axes while answering in single-digit seconds. For high-volume, latency-sensitive chat, these are the models that let you serve most traffic instantly and reserve a slower, more careful model for the hard escalations.

Open source has genuinely caught up

The biggest shift since our last benchmark is at the value end of the chart. Open-weight models are no longer the budget compromise.

DeepSeek V4 Pro sits at the very top-right of the chart, matching or beating every proprietary model on the combined picture while costing a fraction as much on output tokens. Kimi K2.6 is right there with it. For teams that care about cost control, data residency, or not being locked to a single vendor’s roadmap, the open-weight options are now first-class choices, not fallbacks. (We dug into this trend in depth here.)

Beyond the scores: what each model family is actually good at

Our benchmark measures two things precisely. But choosing a model also means knowing the character of each family, the qualities that do not show up in a single accuracy column. Here is how the families behave in support workloads, based on our testing and the broader pattern of how each lab tunes its models.

Claude (Opus and Haiku). The careful one. Anthropic’s models lean toward “I would rather hand off than guess,” and within the Claude line Opus 4.6 and 4.8 post the strongest hallucination resistance, which makes the Opus family a default pick for regulated and high-stakes support. Claude is also unusually good at holding a brand voice and following multi-step instructions without drifting. The tradeoff is cost and, on the larger models, latency.

GPT and the o-series (OpenAI). The generalist. The GPT family is the safe, broadly capable default with the deepest ecosystem and the most mature tool-calling and function-calling, which matters a lot once your bot has to do things (look up an order, issue a refund, file a ticket) rather than just answer. GPT-4.1 and o4-mini are particularly strong value picks: high resolution, fast, reasonably priced. The flagship GPT-5.x models are excellent but priced at the top of the market.

Gemini (Google). The high-volume workhorse. Gemini’s strengths are very large context windows (useful when you want to stuff long knowledge-base articles or whole conversations into a single prompt), native multimodal handling (screenshots, photos of a broken product, PDFs), and an exceptionally strong cheap-and-fast tier. Gemini 2.0 Flash answers in ~3.6 seconds at roughly a dime per million input tokens. The catch, as our data shows, is that the fastest Gemini models can be over-eager and need tight grounding to keep hallucinations in check.

Grok (xAI). The balanced speedster. Grok 4.3 posted one of the best speed-to-quality ratios in the test: single-digit-second responses with solid scores on both axes. A strong candidate for live chat.

Open-weight models (DeepSeek, Kimi, GLM). The control option. These now match or beat the proprietary models on quality while costing a fraction as much, and because you can self-host them, they are the answer when data residency, privacy, or vendor independence is non-negotiable. The practical cost is operational: you (or your vendor) own the hosting, scaling, and reliability.

The checklist: what to weigh before you commit

The original version of this post laid out the dimensions that actually decide a support deployment. They still hold, so here is the 2026 version of the checklist:

1. Resolution accuracy. Can it answer correctly when the answer exists? Necessary, but not sufficient on its own.

2. Hallucination resistance. Will it shut up when it should? The single most underrated metric, and the one most likely to cause real damage if you ignore it.

3. Speed. Email and case workflows tolerate slow models. Live chat does not. Match the tier to the channel.

4. Cost. Per-token price times your volume times your average context size. A “cheap” model with a huge retrieved context can cost more than a pricier model with tight retrieval. Model the real number.

5. Instruction following and tone. Can it stay on-brand, follow your escalation rules, and return structured output (JSON, specific formats) reliably? This is where Claude and the frontier GPT models shine.

6. Proprietary vs open-weight. Closed models are turnkey. Open-weight models give you cost control, data residency, and no lock-in, at the price of running them yourself.

No single model wins all six. The right choice is the one that wins the dimensions your business cannot compromise on.

Where this is heading

Two shifts are worth planning for. First, the gap between proprietary and open-weight models has effectively closed on support tasks, so vendor lock-in is now a choice rather than a necessity. Second, the frontier is moving from “answer the question” to “take the action,” meaning tool use, multi-step agentic workflows, and multimodal inputs (a customer’s screenshot, a photo of a damaged item) are becoming table stakes. The models that win the next round will be the ones that can safely act, not just respond. That raises the stakes on hallucination resistance even further: a model that invents a fact is bad; a model that invents an action is worse.

So which one should you pick?

The honest answer is that the model is no longer the hard part. Here is how we think about matching the model to the business, the same way we framed it in the original post:

• High-stakes, low-tolerance-for-error support

  (airlines, financial services, healthcare, enterprise B2B): lean toward the high-hallucination-resistance end. The strongest there are the open-weight DeepSeek V4 Pro and Kimi K2.6, followed by Grok 4.3 and Claude Haiku 4.5; among the flagship Claude models, Opus 4.6 and 4.8 hold the line best. The cost of one confidently wrong answer about billing or eligibility dwarfs the cost of an extra human handoff.

• High-volume, cost-sensitive support

  (e-commerce, gaming, freemium SaaS): the fast value tier is your friend. Grok 4.3, Gemini 2.0 Flash, and the open-weight models give you strong accuracy at low cost and low latency, with safeguards layered on top.

• Live chat where speed is non-negotiable:

  start with a fast model (Grok 4.3, Gemini 2.0 Flash, Claude Haiku 4.5) as your default and escalate the genuinely hard tickets to a slower, more careful model.

And keep cost in perspective. A cheaper model that hallucinates is the most expensive option of all. Prices also move month to month, so treat the dollar figures above as a snapshot and run your own numbers against your real traffic mix.

The part the leaderboard does not show

Here is the truth behind every number above: the model is maybe 20% of what makes a support bot good.

Every model in our test got the same retrieval, the same grounding, the same tools, and the same carefully built agentic setup. That scaffolding is why even the middle of the table is production-viable. Swap in a naive “stuff the docs in the prompt and hope” setup and the best model on this list will hallucinate its way to the bottom.

What actually moves the needle in production:

• Retrieval quality.

  If the right article never makes it into context, no model can answer correctly. Most “the AI hallucinated” failures are really retrieval failures.

• Grounding and citations.

  Forcing answers to cite their sources, and refusing when nothing relevant is retrieved, is what keeps hallucination resistance high regardless of which model you use.

• Knowing when to hand off.

  As the Gemini-3-Flash-versus-DeepSeek-V4-Pro contrast shows, the answer-versus-decline decision is the whole ballgame. That logic lives in your orchestration, not in the raw model.

• Continuous evaluation.

  The only reason we can publish this table is that we run this benchmark on every model and prompt change. If you are not measuring resolution and hallucination on your own real tickets, you are flying blind.

This is exactly the layer IrisAgent's AI for customer support is built around. We run model-agnostic on top of frontier and open-weight LLMs, with retrieval, grounding, hallucination detection, and continuous evaluation tuned on your real support data. You get top-of-table balance without rebuilding the scaffolding yourself, and without being locked to one vendor’s model when the leaderboard reshuffles again next quarter (and it will).

The bottom line

There is no single best LLM for customer support, and anyone who hands you one number is selling you something. The latest models are all capable. What separates a support bot customers trust from one they learn to ignore is where you set the dial between answering and declining, how fast you answer, and everything wrapped around the model: retrieval, grounding, escalation, and relentless evaluation on your own data.

Pick a model that matches your tolerance for risk. Then spend your real energy on the other 80%.

Want to see how this performs on your support tickets, not our benchmark? Book a demo with IrisAgent and we will run your real cases through it.