What are tokens in AI? It counts pieces, not words
To read and write text, an AI model breaks it into pieces called tokens: sometimes a whole word, sometimes part of one, sometimes a punctuation mark. So what are tokens in AI to you? The unit your limits are measured in and, often, your bill.
What are AI tokens? A publisher’s invoice
Wide Margin Press, an eight-person publisher in Minneapolis, uses an AI to draft back-cover copy. The invoice doesn’t say “texts.” It says tokens. They’re AI’s electricity meter. You don’t need to understand electricity, but you should know what makes the meter spin: everything the AI reads and everything it writes.
AI tokens explained: neither letters nor words
Before it reads, the model cuts the text into pieces taken from a fixed vocabulary. Everyday words are usually one piece; long or rare ones, several: “sun” is typically a single token, while an unusual word like “colophon” gets split into a few. As rough guidance for English, and not an exact figure, a token comes to about four characters, or around three-quarters of a word. Every model cuts text its own way.
What is a context window?
It’s how much text the AI can take in at once, measured in tokens. Everything goes inside: the instructions, your request (the prompt), the documents you hand it, the earlier conversation and the answer it writes. What doesn’t fit, it doesn’t see. And filling it up has a catch: with very long inputs, models can miss details, especially in the middle.
What AI tokens cost, and why they matter
Many services charge for tokens read and for tokens written, and the written ones usually cost more. A long conversation gets more expensive as it goes, because each new turn rereads what came before. The guide to ChatGPT business pricing brings this down to actual plans.
What tends to get mixed up
A token isn’t a word, and it isn’t a digital coin either: it has nothing to do with crypto. It isn’t an embedding: a token is a piece of text; an embedding is the point that sums up its meaning. And the one doing the work with tokens is the LLM.
How many pieces is your text?
Change the paragraph or paste one of your own. Each color is one piece. Then add documents and watch the window.
What you type doesn’t leave this page: the counting happens in your browser.
Roughly how an AI would count it, piece by piece
- Words
- 40
- Pieces (approx.)
- 52
Rule used here: a short word is one piece; a long one, a piece for every four letters or so; each punctuation mark, one more. It’s a simplification to get the idea across, not a real tokenizer: every AI splits text its own way, and the true count will differ.
What if it also has to read this?
This paragraph takes up almost no room in an example window of 100,000 pieces. A hundred-page manual, at about 60,000, already fills more than half.
The 100,000-piece window is an example: some are smaller and some are much larger. A page is counted here as 600 pieces, a round figure.
What uses more?
| Situation | What the AI reads and writes | Usage |
|---|---|---|
| A short question | Reads two lines and writes four | Low |
| A question with a long document | Reads the whole document to answer one line | Higher: what’s read drives it |
| A writing job | Reads little and writes a lot | Higher: what’s written usually costs more |
| A very long conversation | Every turn rereads everything before it | Grows with each message |
Rough guidance: it holds for any AI that measures its usage in tokens.
How it looks in Flipa: tokens and credits
Flipa doesn’t talk to you about tokens. It counts in credits, so nobody on your team has to count tokens. Every use of AI spends credits from the account’s pool. In our tests, a question used between 1 and 4 credits and a job, such as drafting a text, between 45 and 138; with more documents, the same question uses a bit more. Whoever manages the account sees it on the usage screen. The app’s interface is in Spanish today. The rest is in pricing.
AI tokens and the context window: quick questions
What are AI tokens?
They’re the units a language model divides text into: whole words, parts of words and punctuation marks. The model doesn’t see sentences. It sees a row of tokens, and both its capacity and its usage are measured in them.
How many tokens are in a word?
It depends on the word, the language and the model. Short, frequent words are usually one token; long ones, two or more. As rough guidance for English, 100 tokens come to about 75 words. That’s not an exact figure: it shifts with the text and with each model.
What is a context window?
The maximum number of tokens the model can have in front of it at one time, counting what it reads and what it writes. Whatever is left outside doesn’t exist for that answer. Its size varies a lot from one model to another.
What are AI tokens and what do they cost?
Tokens measure the model’s work: processing each one takes computing, both to read and to write. Providers that bill by usage publish a rate per quantity of tokens, usually higher for what the model writes than for what it reads, and it varies by model, so check the provider’s pricing page. It’s pay-per-use, like electricity by the kilowatt-hour.
Are Flipa credits tokens?
No. Credits are the unit Flipa counts usage in. A job uses more than a question, and a bit more when there are more documents, but we don’t publish a conversion to tokens. What each plan costs, with its credits, is on the pricing page. Uploading a document with text uses none; a scanned page uses a fraction of one credit.
Tokens: what makes the meter spin
And you see it before you run out, on the usage screen.
- AI counts pieces, not words
- Reading more costs more
- In Flipa you see it in credits