What Are Tokens in AI? Why Your Usage Is Counted in Pieces of Words

AI pricing pages say things like "$3 per million input tokens." Usage limits reset after so many tokens. Context windows hold "200K tokens." If you build with AI tools, tokens are the unit everything is measured in. Here's what they are.

What a token is

AI language models don't read letters or whole words. They read tokens: chunks of text that are often a word, part of a word, a space plus a word, or a punctuation mark.

The sentence:

Tokenization isn't hard to understand.

might be split into something like:

Token | ization |  isn | 't |  hard |  to |  understand | .

Common words are usually one token. Rare or long words get split into pieces. Each model family has its own tokenizer — its own way of chopping text — so exact counts differ between Claude, GPT, and Gemini.

Rules of thumb

For ordinary English text:

1 token ≈ 3–4 characters
100 tokens ≈ 70–75 words
1,000 tokens ≈ a page and a half of text

These are rough. Some content costs noticeably more:

Code — symbols, indentation, and variable names split into many tokens.
Non-English languages — often more tokens per word, especially non-Latin scripts.
Numbers, URLs, and random strings like IDs and hashes.
Images and PDFs — models count these in tokens too, based on size.

Input tokens and output tokens

Every request to a model has two sides:

Input tokens — everything the model reads: your message, the conversation so far, instructions, files, and tool results.
Output tokens — everything the model writes back.

APIs price them separately, and output tokens cost several times more than input tokens. (Claude Pro vs Max vs API for Claude Code covers how that plays out across plans.)

The part that surprises people: in a conversation, the whole history is sent again with every message. Your tenth message includes the previous nine exchanges as input. Long conversations get expensive, and slow, because each step re-reads everything before it. Prompt caching softens the cost of re-reading unchanged content, but the context still fills up.

Why coding agents use so many tokens

A chatbot answers a question. A coding agent like Claude Code works: it reads files, runs commands, reads the output, edits, re-reads, and tests. Every file it opens and every line of command output becomes input tokens. A single task can involve dozens of steps, each re-reading a growing context.

That's why an hour of agentic coding uses far more tokens than an hour of chatting. (How to Keep Claude Code Costs Down.)

Tokens and the context window

A model can only consider a limited number of tokens at once: its context window. Everything — instructions, conversation, files, and the answer it's writing — must fit. (What Is a Context Window?.)

Practical ways to use fewer tokens

In AI coding tools:

Start fresh sessions for new tasks. Don't carry a long conversation about the login page into work on the billing page. In Claude Code, /clear does this.
Point to specific files rather than asking the tool to "look at everything."
Keep instruction files short. A CLAUDE.md is sent with every request; a bloated one costs tokens every time. (How to Write a CLAUDE.md.)
Avoid dumping huge logs. Paste the relevant error, not 5,000 lines.
Use a smaller model for simple tasks where the tool lets you choose.

In your own AI-powered app:

Set a maximum output length on every call.
Trim conversation history — keep the recent messages and a summary of older ones.
Cap input size so users can't paste a novel into every request. (How to Stop Bots From Running Up Your AI App's Bill.)
Use prompt caching where your provider supports it, for large instructions or documents you send repeatedly.
Record token usage per request — providers return the counts in every response — so you know what each feature costs.

Counting tokens

Anthropic's API returns exact input and output token counts with every response, and offers a token-counting endpoint for checking before you send. In Claude Code, the /context command shows what's taking up space in the current session.

EasySpawn connects Claude Code to your own Claude subscription or API key — Anthropic bills your AI usage directly, and EasySpawn bills only for the workspace, so there's no markup on your tokens. See pricing or join the waitlist.