A developer pastes a long contract into a chat assistant and is told the conversation is too long. A marketing team opens its first AI invoice and finds usage measured in millions of tokens, not words. A support team in Türkiye notices that the same prompt in Turkish uses up its budget faster than in English. All three have met the same small unit: the token.
This post explains what a token is, how text is split into tokens (with a real English and Turkish example), why some languages need more of them, how tokens set the cost and limits of a model, how to count them and how to use fewer. It ends with the word's other meanings.
What is a token in AI?
A token is the smallest unit of text an AI language model handles: the model reads a prompt as a sequence of tokens and writes its answer one token at a time. A token is often a whole word, such as "order", but it can also be part of a word, a punctuation mark or a group of digits.
Each token has an ID number in the model's vocabulary, the fixed list of pieces it knows. The model never sees letters. It receives a list of numbers, predicts the number of the next token, and the software turns the numbers back into text. A large language model is, at its core, a system trained to make that next-token prediction well.
Models that accept images, audio or video turn those inputs into tokens too, so a photo or a voice clip also uses part of a request's budget.
Why do AI models use tokens instead of words?
Tokens are a middle ground between letters and whole words. If a model read single letters, every text would become a very long sequence, which is slow and expensive to process. If it read whole words, its vocabulary would need every word, name, typo and new brand in every language, and it would still meet words it had never seen.
Subword tokens solve both problems. The vocabulary has a fixed size, and a word that is not in it is built from smaller pieces. A rare surname or a product code does not break the model; it just costs a few more tokens.
| Unit | Vocabulary | Unseen words | Sequence length |
|---|---|---|---|
| Characters | Very small | Never a problem | Very long |
| Whole words | Huge, never complete | Often unknown | Short |
| Subword tokens | Fixed, large | Built from pieces | In between |
How does tokenization work?
Tokenization is the step that cuts text into tokens before the model sees it, and many current language models, OpenAI's among them, do it with byte pair encoding (BPE). BPE began as a data compression method; Sennrich, Haddow and Birch adapted it in a 2015 paper on rare words in machine translation, encoding rare and unknown words as sequences of subword units.
A BPE tokenizer is built once, before the model is trained:
- Start from the smallest units: single characters, or raw bytes in byte-level tokenizers such as OpenAI's, so text in any script can be written.
- Count neighboring pairs in a large sample of training text, such as "t" followed by "h" in English.
- Merge the most frequent pair into a new vocabulary entry, "th", and treat it as one unit from then on.
- Repeat, so frequent pieces grow into syllables, common suffixes and finally whole words with their space attached, such as " the".
- Stop at a target size. OpenAI's
o200k_baseencoding, for example, has about 200,000 entries. - Encode new text by applying the learned merges to your prompt and replacing each piece with its ID number.
Because merges follow frequency, common pieces become single tokens and rare ones stay split. Spaces matter too: in o200k_base, price and price with a space are two different tokens.
Every model family has its own tokenizer, so the same text gives different counts on different models, even from one vendor. Anthropic states that Claude Opus 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier ones.
What does a sentence look like as tokens?
The quickest way to understand tokens is to watch a real tokenizer split a sentence. We ran two sentences with the same meaning through OpenAI's open-source tiktoken library (version 0.14.0) with the o200k_base encoding, which the library uses for GPT-4o. The bars mark where one token ends and the next begins:
English, 10 tokens:
Your| order| has| been| shipped| and| will| arrive| tomorrow|.
Turkish, 16 tokens:
Sip|ariş|iniz| k|arg|oya| ver|ildi| ve| yar|ın| el|iniz|de| olacak|.| Sentence | Characters | Words | Tokens |
|---|---|---|---|
| English | 53 | 9 | 10 |
| Turkish | 53 | 7 | 16 |
Both sentences are exactly 53 characters long, yet the Turkish one needs 60 percent more tokens. The sample shows the usual patterns:
- Spaces travel with the next word: " order" is one token, so spaces rarely cost a token of their own.
- Punctuation is often separate: the full stop is a token in both sentences.
- Common English words stay whole, while Turkish words split at suffixes and sometimes inside the root:
kargoya(to the courier) becomes " k", "arg" and "oya". - Numbers split into chunks: the date
2026-11-18takes six tokens in the same encoding.
These counts belong to this encoding only; another model's tokenizer draws the lines elsewhere.
Why do some languages use more tokens than English?
Some languages use more tokens because a tokenizer learns its pieces from its training text: what was frequent there becomes whole tokens, what was rare stays split. A tokenizer trained mostly on English text holds many complete English words and fewer complete words of other languages.
The gap can be large. In a study presented at NeurIPS 2023, Petrov and colleagues found that the same text in different languages can differ in token count by up to 15 times, and that the gap remains even in tokenizers built for many languages. The effects they name: a higher price for the same service, slower responses and less room in each request.
Turkish adds a second reason. It is an agglutinative language: it attaches a chain of suffixes to a root, so one word can carry a whole English phrase. bulamadıklarımızdan ("from those we could not find") is one word and five tokens; the six-word English phrase takes six. Per word Turkish looks expensive, per meaning the gap is smaller, though in our sentence above it was still 16 tokens against 10.
How do tokens affect the cost of AI?
AI APIs bill by the token and count input and output tokens separately. Input tokens are everything you send: system instructions, conversation history, documents, tool definitions and images. Output tokens are what the model writes back.
The price lists of OpenAI and Anthropic, checked on 18 November 2026, both quote prices per million tokens and both charge more for an output token than for an input token. Both also offer a lower rate for cached input, a repeated prefix such as a long system prompt that the provider keeps between requests.
Two details surprise first-time users. Hidden reasoning is billed as output: OpenAI's reasoning guide says reasoning tokens are not visible through the API but are billed as output tokens, and Anthropic says the same of its thinking tokens. And an API-based chat usually resends the earlier conversation with each new message, so history is paid for again on every turn.
The cost of one request is input tokens times the input rate plus output tokens times the output rate. A Turkish prompt with 60 percent more tokens than its English twin costs 60 percent more to send.
What is a token limit?
A token limit is the most tokens a model can handle in one request, and there are two. The context window is the total the model can take into account at once, its own answer included. The maximum output caps how many tokens it may write in one response, and you can set it lower through the API.
Everything counts toward the context window: instructions, history, documents, tool results, images and the output. What belongs in that window, and why a bigger one is not automatically better, is covered in context engineering vs prompt engineering.
What happens at the limit depends on where you hit it:
- The input alone is too long: the API rejects the request; Anthropic's returns the error "prompt is too long".
- The answer reaches the output cap: generation stops mid-sentence and the response is flagged as cut off (
stop_reasonismax_tokenson Anthropic's API, the status isincompleteon OpenAI's). - A chat app runs out of room: it may drop or summarize the oldest messages, so the assistant "forgets" the start of a long conversation.
How do you count tokens?
Count tokens with the tokenizer or counting tool of the model you use, because each model family splits text differently:
- OpenAI publishes the tiktoken library used above and a tokenizer web page on its developer platform.
- Anthropic offers a free token counting endpoint,
/v1/messages/count_tokens, which accepts the same input as a real request, images and PDFs included, and returns an estimate. - Google provides a
count_tokensmethod in the Gemini API, and its token documentation states how images and audio are counted. - Every API response reports the tokens actually used in a
usagefield, and that number is what you are billed for.
For a quick estimate, vendors give rules of thumb for English. OpenAI's documentation says one token is about 4 characters or 0.75 words of English text, so 1,000 tokens are roughly 750 words; Google's says 100 tokens are about 60 to 80 English words. Neither holds for other languages: our Turkish sentence averaged 3.3 characters per token, the English one 5.3.
How can you use fewer tokens?
Send only the text the model needs and ask for no more output than you will use:
- Send clean text, not raw HTML. Markup, scripts and menus can cost far more than the content. In our web scraping for RAG test, one documentation page took 21,044 tokens as raw HTML and 4,405 as Markdown.
- Trim the history: keep the last few turns in full and summarize the rest.
- Cap and shape the output: set a maximum length and ask for a format, such as three bullet points, instead of an open essay.
- Remove repeated boilerplate such as unused tool definitions and duplicated examples.
- Cache fixed prefixes where the API supports it; they still take room, but cost less.
- Resize images to what the task needs before you send them.
Web pages deserve a special mention: retrieval systems, research agents and evaluation sets are often built from public pages, and every page passed to a model is paid for in tokens, so extracting the main content first is the cheapest step. When you collect those pages from many countries at scale, a Residential Proxy helps: it routes each request through a home connection in the country you choose, so you get the page local visitors see. Collect within the site's terms and at a polite rate.
Teams that would rather not run the collection themselves can ask our sales team about managed page extraction through our Web Scraping API; the data scraping page covers our proxy pools for collecting at volume.
Where do you meet tokens?
You meet tokens wherever text goes into or out of a language model: as message limits in chat assistants, as the unit of invoices and rate limits on API dashboards, as the chunk size in retrieval systems, as a different price for the same text in each language of a translation project, and as the cost of every tool call in an AI agent's loop.
What else does "token" mean?
Outside language models, "token" has several unrelated meanings that often appear in the same search results:
- Access token: a temporary digital key a site or app issues after you sign in. Anyone holding it can act as you, so a Discord token or any other session token must never be shared.
- Security token: a device or app that produces one-time sign-in codes, or a USB key that stores a digital certificate.
- Crypto token: a digital asset recorded on a blockchain.
- Token in programming: the smallest meaningful unit a compiler reads in source code, such as a keyword or a symbol.
Common mistakes
- Treating tokens as words. A budget of 1,000 tokens is not 1,000 words, and the ratio changes with the language.
- Reusing counts across models. A count from one tokenizer can be far off on another.
- Estimating other languages with the English rule of thumb. Measure a sample in the real tokenizer.
- Sending raw HTML to a model. The markup can be most of the bill.
- Forgetting the output side. Long answers and hidden reasoning are billed at the higher output rate.
- Ignoring the cut-off flag. A response stopped at the output limit is incomplete, even if it looks finished.
Decision guide
| Need | What to do |
|---|---|
| Estimate a prompt's cost | Count with the vendor's tokenizer or endpoint |
| Rough size check in English | About 4 characters or 0.75 words per token |
| Budget for another language | Measure a sample in that language |
| Feed web pages to a model | Extract main content, send Markdown |
| Long chat costs too much | Summarize older turns, cache fixed prefixes |
| Answers stop mid-sentence | Raise the output limit or ask for less |
| Request rejected as too long | Split the input, retrieve only relevant parts |
Frequently asked questions
How many words is 1,000 tokens?
In English, about 750 words by OpenAI's rule of 0.75 words per token, or 600 to 800 words by Google's estimate. Most other languages, code and number-heavy text give fewer words per 1,000 tokens, so use the figure only for English prose.
Why does Turkish use more tokens?
Turkish words are less frequent than English ones in the text many tokenizers learn from, and Turkish attaches long chains of suffixes to a root, so words split into several pieces. In our example, a Turkish sentence with the same length and meaning as an English one needed 16 tokens instead of 10.
Do spaces and punctuation count as tokens?
Yes, though a space is usually attached to the word after it, while punctuation such as a full stop is often a separate token. Extra spaces, line breaks and indentation add tokens, which matters most in code and copied web text.
Are images counted in tokens?
Yes, and the number depends on the image size and the vendor's formula. Google's Gemini documentation, for example, counts an image of up to 384 pixels on both sides as 258 tokens and splits larger images into tiles of 258 tokens each.
What happens when you hit the token limit?
If the input is too long for the context window, the API rejects the request. If the answer reaches the output limit, it stops mid-sentence and is marked as cut off. Chat apps may instead drop or summarize older messages, which is why a long conversation can lose track of its beginning.
Are AI tokens the same as crypto tokens?
No. An AI token is a piece of text a language model reads or writes, and the unit used to measure and bill model usage. A crypto token is a digital asset recorded on a blockchain; the two only share a name.
Summary
A token is a piece of text, often a word, sometimes part of a word or a punctuation mark, that a language model handles as a number from its vocabulary. Tokenizers build that vocabulary from frequent pieces of their training text, which is why English usually needs fewer tokens than Turkish. Tokens decide what a request costs, input and output billed separately, and how much fits into it. Count them with the tokenizer of the model you use, measure each language on its own and send clean text rather than raw pages, so the tokens you pay for carry content.




