Muse Spark Open for Customer Support: Benchmark Results
IrisAgent now supports Muse Spark Open, Meta's new open-weight model, served through Fireworks. We ran it through the same customer-support suites we use for every model on the Support Model Arena. The short verdict: Muse Spark Open answers support tickets as well as almost any model we have tested, and it cites the right article more often than most. But it also invented answers on tickets it should have declined, it routed chat intents less reliably than any other model we run, and it is the slowest model on our board. For a support bot that talks to customers unsupervised, that trade does not work yet. We are not routing production traffic to it.
The short version
98.8% answer accuracy, tied for 5th of 32 models and tied for the best of any model under $2 per million input tokens.
85.9% cited sources, tied for 6th of 32 models and ahead of GLM 5.3 Flash, Claude Sonnet 5.5 and o4-mini.
80% hallucination resistance, second lowest on the board. Every other open-weight model we run scores 95% or higher.
87.5% intent routing, the lowest of 32 models, but 90.8% ticket tagging, tied for 2nd.
The slowest typical response on the board, about 4x slower than GLM 5.3 Flash, the open-weight model we run in production today.
Overall 88.7%, 25th of 27 ranked models. At $1.25 / $4.25 per million tokens it costs about 8x more than GLM 5.3 Flash.
How we tested
Every model on the arena runs inside the same production IrisAgent stack. Same retrieval, same tools, same prompts. Only the model changes. We grade five support jobs:
Answer accuracy: anonymized real support tickets whose answer exists in the knowledge base. A pass is a correct, grounded resolution.
Hallucination resistance: tickets that must not be answered from the knowledge base. A pass is a decline or handoff instead of an invented answer.
Cited sources: the same answerable tickets, but the answer must also cite the right knowledge base article.
Intent routing: multi-turn chats routed against configured FAQ intents and workflows, including near-miss intents and escalation requests.
Ticket tagging: exact-match classification against production tag catalogs from B2B customers.
The Overall score weights these 30% answer accuracy, 30% hallucination resistance, 15% cited sources, 15% intent routing and 10% tagging. The full method and every other model are on the Support Model Arena.
Results
Model | Overall | Answer accuracy | Hallucination resistance | Cited sources | Intent routing | Ticket tagging | Typical speed | Slowest responses | Price per 1M tokens (in / out) |
|---|---|---|---|---|---|---|---|---|---|
Muse Spark Open | 88.7% | 98.8% | 80.0% | 85.9% | 87.5% | 90.8% | Slowest | Long tail | $1.25 / $4.25 |
GLM 5.3 Flash | 94.4% | 97.6% | 100% | 82.4% | 92.5% | 88.8% | Medium | Moderate tail | $0.15 / $0.50 |
DeepSeek V4.1 Flash | 92.8% | 94.1% | 95.0% | 84.7% | 95.0% | 90.8% | Slower | Moderate tail | $0.22 / $0.66 |
Kimi K3 | 92.8% | 98.8% | 95.0% | 84.7% | 92.5% | 80.6% | Slow | Very long tail | $3 / $15 |
Grok 4.5 | 94.1% | 97.6% | 95.0% | 87.1% | 95.0% | 89.8% | Fast | Tight tail | $2 / $6 |
o4-mini | 87.9% | 91.8% | 90.0% | 69.4% | 95.0% | 86.7% | Medium | Tight tail | $1.10 / $4.40 |
Claude Sonnet 5.5 | 93.0% | 92.9% | 100% | 82.4% | 92.5% | 88.8% | Fast | Tight tail | $2 / $10 |
Single runs, exact-match grading. The decline suite is small, so a single ticket moves hallucination resistance by several points. Treat small gaps as ties. Speed labels compare the same production stack for every model. Muse Spark Open is in a limited serverless preview on Fireworks through October 29, 2026.
Where Muse Spark Open is strong
It answers. Muse Spark Open resolved 98.8% of answerable tickets, tied with Kimi K3 and GLM 5.3 and the best of any model under $2 per million input tokens. Only four models on the board, all frontier models, did better. When the right article is in the knowledge base, it finds the answer and writes it.
It cites the right article. 85.9% of its answers pointed to the correct knowledge base article, against 82.4% for GLM 5.3 Flash and Claude Sonnet 5.5 and 69.4% for o4-mini, the closest model to it on price. Citations are how a customer checks an answer and how an agent trusts one, so this matters more than its 15% weight suggests.
Ticket tagging. Muse Spark Open tagged 90.8% of tickets correctly against production tag catalogs, tied for 2nd of 32 models. About two in three of its misses left a ticket untagged rather than applying a wrong tag, which is the safer error when tags drive routing.
Where it falls short
It invents answers it should decline. Muse Spark Open scored 80% on hallucination resistance. On tickets the knowledge base cannot answer, it wrote an answer anyway about one time in five. Every other open-weight model we run scores 95% or better, and GLM 5.3 Flash declined every one. The same eagerness that lifts its answer accuracy is what hurts it here: a model that always tries to answer will also answer when it should not.
It is slow. Its typical response is the slowest on the board: about 4x slower than GLM 5.3 Flash, about 2x slower than GLM 5.3, and more than 6x slower than Claude Sonnet 5.5. Its slowest responses hold up better than its typical ones suggest (a shorter tail than Kimi K3 or GLM 5.3), but in live chat a slow typical answer is felt on every conversation, not just the worst one.
Price for what you get. At $1.25 / $4.25 it costs roughly 8x more per token than GLM 5.3 Flash, which scores higher overall, declines every ticket it should, and answers about 4x faster. Muse Spark Open's gains over it, about one more ticket answered in a hundred and a few more correct citations, sit inside the noise of a single run.
Intent routing. It passed 87.5% of routing checks, the lowest of 32 models. Most of its misses were about agent requests: a bare request for an agent that should first prompt the customer to describe their issue went straight to handoff, and a frustrated message that should have gone to an agent did not. Several models miss the first of these, but none missed as many routing checks overall.
How IrisAgent uses it
IrisAgent does not bet on one model. Our multi-LLM engine picks a model per task from this board, and customers can choose the model they prefer. Muse Spark Open is available today. On these results we are not replacing GLM 5.3 Flash in our batch ticket path or using Muse Spark Open for live chat. Its strength in answering and citing makes it worth testing for offline work where speed matters less, such as drafting knowledge base articles, and we will rerun it when the preview ends and Meta or Fireworks update it. Every answer, whichever model writes it, goes through IrisAgent's grounding checks before it reaches a customer.
For the wider picture of how today's models compare on support work, see Best LLMs for customer support chatbots and how we measure accuracy.
See the full board
The Support Model Arena ranks every model we support by category, with latency and price next to quality. If you want to see how these models perform on your own tickets, book a demo.
Loading the form. Prefer email? Reach us at info@irisagent.com or call +1 617 249 3312.