Best Performing LLMs for Customer Support: Open Source Models Rise

The large language model (LLM) landscape has changed dramatically over the past year, reshaping how businesses choose models for customer support chatbots and automation platforms. If you read our previous evaluation on LLMs, you know we recommended GPT-4 for top-tier reliability but highlighted the promise of open models like Mistral and Llama. Fast forward to mid-2025, and the rapid evolution, new open-source projects, and benchmarks are demanding a sequel post!

The Open-Source LLM Surge: More Models, More Momentum

Open-source LLMs are now a mainstream force. Recent stats show that over 60% of AI-driven enterprises intend to evaluate open-source models by the end of 2025, and more than 67% of organizations are already using some flavor of LLM for generative tasks. Adoption is fueled by:

• Explosion of options: Many of 2025’s most exciting releases are open. Notable examples include Kimi-K2 (32B, DeepSeek R1 (685B parameters, open weights), Qwen3 from Alibaba (over 200B parameters), Llama 3.1/4 from Meta, Nemotron-4 from Nvidia, and now GPT-OSS from OpenAI (over 117B parameters).

• Competitive performance: Top open-source models like Kimi-K2 and GPT-OSS now rival or surpass many proprietary models on reasoning and instruction following.

• Community velocity: Open models are being fine-tuned by hundreds of teams, rapidly improving benchmarks and niche strengths.

• Cost savings and flexibility: Hosting your own, or choosing a vendor built on open-source weights, can reduce inference costs by up to 100x compared to closed models.

Recent Closed vs. Open Results: The Playing Field Narrows

We've evaluated GPT-5 alongside top new open-source models like Kimi-K2 and DeepSeek-R1 on our proprietary customer support-focused eval dataset. Here are the key takeaways:

• Open-source models are now outperforming many large closed-source LLMs in customer support tasks.

• Despite expectations that newer models improve over time, GPT-5 and others show limited gains specifically for CX and customer support use cases.

• The industry focus is shifting away from instruction-following enterprise agents toward coding agents.

Recent side-by-side tests (e.g., for customer support accuracy) found the best open models slightly trailed the top closed models on consistency but performed on par for many domain-specific and RAG-augmented customer support tasks. Fine-tuned open models excelled in specialized workflows and offered immense cost savings.

Open-source models are now regularly outperforming smaller proprietary models and matching the top tier in scenarios where customization and data security are critical.

What’s Changed: Key Insights from the Latest LLM Wave

Several overarching trends have helped reshape the LLM ecosystem:

• Context windows are massive: 2025’s best open and closed models feature context windows from 128,000 to 1 million tokens, breaking previous limitations and enabling long-running workflows.

• Open collaboration outpaces vendor lockdown: The open-source community’s rapid fine-tuning and benchmark sharing have democratized LLM innovation, with enterprise adoption following suit.

• Specialization wins: Fine-tuned, domain-specific open models often beat generalist closed-source models for customer support, technical troubleshooting, and code assistance.

• Cost and privacy flexibility: The open model surge lets businesses self-host for privacy or use trusted vendors at a fraction of closed model pricing—a crucial factor for startups and cost-sensitive industries.

• Benchmarks evolve: New evaluation datasets target real-world business needs including hallucination rate, cost-benefit, instruction compliance, and long-context performance—all areas where open-source options are improving fast.

Takeaways and Recommendations for 2025

• Enterprises with high compliance/security needs: Consider fine-tuned or vendor-hosted open models (Kimi-K2, GPT-OSS, DeepSeek, Qwen) with robust RAG and guardrails.

• Cost-sensitive SMBs: Open models now provide state-of-the-art support automation at a fraction of closed-source costs—don’t overlook the emerging options.

• Hybrid/federated strategies: Combining multiple LLMs—closed for edge cases, open for routine tasks—is increasingly practical.

• Constantly re-benchmark: Model quality is improving monthly, and the best fit can change as new open-source models appear and as your data or needs evolve.

• The LLM gold rush is alive and well, and for chatbot builders in 2025, the rise of high-quality, open-source models means there’s never been a better—or more dynamic—time to pick your stack.

How IrisAgent Gets Improved Performance Above Baseline

IrisAgent’s superior performance in customer support automation stems from a multi-layered approach that goes well beyond using off-the-shelf large language models:

• Fine-Tuning on Customer Data: By fine-tuning LLMs directly on specific customers’ historical support interactions and documented knowledge, IrisAgent significantly improves relevance and accuracy in responses. This customization lets the model grasp unique terminology, workflows, and tone, resulting in a more natural and precise conversational experience.

• Industry-Specific Fine-Tuning: Beyond individual customers, IrisAgent applies fine-tuning on industry-specific corpora—such as legal, travel, SaaS, or retail support data—which enhances the model’s domain expertise. This layered fine-tuning helps the AI handle niche queries and regulatory requirements better than generic LLMs.

• Guardrails Using Retrieval-Augmented Generation (RAG): IrisAgent integrates robust RAG pipelines that dynamically retrieve up-to-date and trusted information from whitelisted sources during conversations. This approach ensures the answers are grounded in verified documents and reduces hallucination risks, maintaining customer trust and compliance.

• Hallucination Removal via Models and Heuristics: Despite strong language modeling capabilities, hallucinations remain a critical challenge. IrisAgent employs dedicated hallucination detection models that flag suspect content, combined with heuristic rules tailored to customer support contexts. Suspicious answers trigger fallback strategies, such as returning verified snippets or escalating to human agents.

• Adaptive Instruction Following: IrisAgent optimizes instruction-following with specialized instruction-tuning and prompt engineering, enabling the model to reliably produce structured responses (e.g., JSON data, step-by-step guides) required by enterprise workflows.

• Multi-Model Federation: Leveraging an ensemble of foundation models, IrisAgent dynamically routes queries to the most suitable model based on task complexity, context length, and latency requirements, balancing speed with accuracy efficiently.

• Real-Time Performance Monitoring and Feedback Loops: Continuous monitoring captures performance metrics like accuracy, user satisfaction, and error rates in production. This data feeds into automated retraining pipelines as well as manual expert reviews, facilitating swift iterative improvements.

• Explainability and Transparent Responses: IrisAgent enhances user trust through transparent sourcing and explainability features that disclose answer provenance and confidence levels, critical in high-stakes customer support.

Together, these advances enable IrisAgent to outperform baseline models, delivering highly accurate, contextually aware, and trustworthy AI-driven customer support at scale.

If you’re navigating model choices or need help benchmarking the latest options for your business case, we’re always happy to share our hands-on expertise—get in touch!