What Are AI Tokens? A Plain-English Guide for Support Leaders
An AI token is the basic unit of text a large language model reads and writes. Before a model processes a customer's message, it splits the text into tokens: whole words, pieces of words, punctuation marks, or spaces. In English, one token averages about 4 characters, or roughly three quarters of a word, so 100 tokens is about 75 words. Tokens are also how AI providers meter usage, which makes them the unit behind both the cost and the speed of every AI-generated support reply.
How Tokenization Works
A tokenizer is a fixed vocabulary of tens of thousands of text fragments that the model learned during training. Common words like "order" or "refund" usually map to a single token. Rarer words get split into pieces: "unsubscribed" might become "un" plus "subscribed", and a product SKU, error code, or tracking number can break into many small fragments. Each token is then converted into a number the model can compute with.
Most tokenizers are trained largely on English text, which has a side effect support teams should know about: the same message in another language often uses more tokens. Depending on the model, languages such as Japanese, Hindi, Thai, or Arabic can take an estimated 1.5 to 3 times as many tokens as the English equivalent. For a global support team, that means multilingual conversations cost more and fill the model's context window faster.
Input Tokens vs Output Tokens
Input tokens: Everything sent to the model, including the system instructions, the customer's message, the conversation history, and any knowledge articles retrieved to ground the answer.
Output tokens: Everything the model writes back, including the reply and, for reasoning models, the internal "thinking" tokens generated before the answer.
The split matters because providers price the two differently. Output tokens typically cost several times more than input tokens (often 3 to 5 times as much), and they are generated one at a time, so long replies are both more expensive and slower to produce.
What Is a Context Window?
The context window is the maximum number of tokens a model can consider in a single request, counting input and output together. Current models range from roughly 128,000 tokens to more than 1 million. A bigger window lets the model see a longer ticket thread or more documentation at once, but it is not free. Every token in the window is billed, and models tend to lose precision when the one relevant fact is buried in a very long prompt. Filling the window is rarely the goal. Filling it with the right content is.
Why Token Counts Drive Cost and Speed
Almost every commercial LLM API charges per million tokens, with separate rates for input and output. Prices vary by more than 100 times across models: small, fast models cost well under a dollar per million input tokens, while frontier reasoning models cost several dollars per million input tokens and considerably more for output. A simple way to estimate the cost of one AI reply:
Cost per reply = (input tokens × input price + output tokens × output price) ÷ 1,000,000
Speed follows the same logic. The model has to process every input token before it starts writing, then generates output tokens one by one. Trimming unnecessary input shortens the wait for the first word, and keeping replies concise shortens everything after it.
What Uses Tokens in a Support Conversation
In a typical AI support interaction, the customer's question is the smallest part of the bill. Tokens add up from five places:
System prompt: Brand voice, policies, escalation rules, and tool definitions. It is sent again with every turn.
Retrieved knowledge: Help center articles, internal docs, and past resolved tickets pulled in to ground the answer. This is often the largest share of input.
Conversation history: Earlier messages in the thread are resent on each new turn, so token use climbs as the conversation gets longer.
Tool results: Order lookups, account details, and other API responses returned from connected systems.
The reply: The output tokens the customer actually reads.
Because instructions and history are resent on every turn, a long chat costs much more than the sum of its visible messages. A ticket that bounces back and forth ten times can use many times the tokens of one that is resolved in the first reply.
How to Control Token Spend
Retrieve only relevant passages. Use retrieval augmented generation (RAG) to pass the few paragraphs that answer the question, not whole articles or the entire knowledge base.
Summarize long threads. Replace older turns with a compact summary that keeps order numbers, decisions, and open issues.
Use prompt caching. Major providers discount repeated prompt prefixes such as a long system prompt, often by up to 90% on the cached input. Keep static instructions at the start of the prompt so they can be cached.
Right-size the model. Route simple intents like password resets or order status to a smaller, cheaper model, and reserve frontier models for complex troubleshooting.
Keep replies tight. Give length guidance in the prompt. Shorter answers cost less, arrive faster, and are usually easier for customers to act on.
Measure tokens per resolution. Track usage per resolved conversation, not per message, so you see the true cost of getting a customer to an answer.
How IrisAgent Keeps Tokens Working for You
The cheapest token is one that never needed to be sent, and the most expensive reply is a wrong one that creates a second ticket. IrisAgent is built around both ideas. Its retrieval layer pulls only the verified knowledge and live system data that are relevant to the question, which keeps prompts lean and answers grounded. When the verified sources do not contain an answer, the AI does not fill the gap with a guess. It hands the conversation to a human agent with the full context attached, so nobody has to start over.
As a full CX operating system, IrisAgent applies the same discipline across chat, email, and ticket workflows, and lets teams match the model to the task. You can compare how leading models perform on real support work, and what they cost, on the Support Model Arena before choosing one.
Learn More
Start with the fundamentals in Understanding LLM: Large Language Model, then see how grounding cuts both tokens and hallucinations in What Is Retrieval Augmented Generation (RAG)? and Context Management in AI Conversations.
For budgeting, read AI Chatbot Pricing: The Complete Customer Support Cost Guide and compare models side by side on the Support Model Arena.
Frequently Asked Questions
What is an AI token?
An AI token is the basic unit of text a large language model reads and writes: a whole word, a piece of a word, a punctuation mark, or a space. In English, one token averages about 4 characters, or roughly three quarters of a word, so 100 tokens is about 75 words.
What is the difference between input and output tokens?
Input tokens are everything sent to the model, including system instructions, the customer's message, conversation history, and retrieved knowledge articles. Output tokens are everything the model writes back, including reasoning tokens. Output tokens typically cost several times more than input tokens, often 3 to 5 times as much, and are generated one at a time.
Why do non-English conversations use more tokens?
Most tokenizers are trained largely on English text, so other languages get split into more pieces. Depending on the model, languages such as Japanese, Hindi, Thai, or Arabic can take an estimated 1.5 to 3 times as many tokens as the English equivalent, which raises cost and fills the context window faster.
How can support teams reduce LLM token costs?
Retrieve only the relevant passages with RAG instead of whole articles, summarize long threads, use prompt caching for repeated instructions (often discounted by up to 90%), route simple intents to smaller models, keep replies concise, and track tokens per resolved conversation rather than per message.
Loading the form. Prefer email? Reach us at info@irisagent.com or call +1 617 249 3312.