Token, the Unit AI Chatbots Bill and Think In

A token is the small chunk of text a language model actually reads and writes, usually a word, part of a word, or a punctuation mark. As a rough rule, one token is about three quarters of an English word, so 1,000 tokens is roughly 750 words. Models do not see letters or sentences. They see a sequence of tokens, and they predict the next one.

Why it matters when you are buying or building

Tokens are the meter. Chat platforms bill by tokens in and tokens out, and almost every "why is my bill higher than expected" conversation traces back to token count rather than conversation count. The traps are consistent:

• A long system prompt is re-sent on every single turn. A 2,000 token prompt across a twenty turn conversation is 40,000 input tokens before the customer has said anything interesting.
• Retrieval that pulls ten knowledge base chunks when three would do triples the input cost of every answer.
• Conversation history compounds. Turn one is cheap, turn twenty is not.

Output tokens usually cost several times more than input tokens, so a bot that writes four friendly paragraphs when one sentence would do is quietly expensive as well as annoying to read.

Voice is billed differently. Most voice platforms I run, including the Telnyx and ElevenLabs style stacks, charge per minute rather than per token, and bundle the model, speech to text and text to speech into that rate. Tokens still matter there, because they drive how long the model takes to produce a reply, which drives latency.

A production example

I run an OpenAI Realtime plus Twilio voice agent for an energy company where fixed infrastructure runs about fourteen dollars a month plus usage. The fixed part was never the risk. Usage was. Early on the agent gave long, chatty answers, which cost more and, worse, made callers interrupt it constantly. Tightening the prompt to force short spoken answers cut both the token spend and the interruptions in one change. On voice, shorter is cheaper and better at the same time, which is a rare thing.

How to sanity check a quote

Ask a vendor for a per conversation cost, not a per token cost, and ask what conversation length they assumed. Then ask what happens at three times that length. Anyone operating real bots will have that number. Anyone reselling a dashboard usually will not. For a fuller breakdown of what these builds actually run, see my chatbot cost guide.