Token Counter.

A token is not a word

Tokenizers are trained, not designed. Byte Pair Encoding starts from raw bytes and repeatedly merges the most frequent adjacent pair until it has a vocabulary of somewhere between 50,000 and 200,000 entries. What comes out is a table of subword fragments: common English words survive as a single token, rarer ones come apart into stem and suffix, and anything the training corpus barely saw falls back toward individual bytes.

Two consequences catch people out. The leading space is usually part of the token, so hello and " hello" are different entries and gluing strings together can change the count. And modern tokenizers are byte-level, so there is no unknown-token failure case — everything encodes, some things just encode expensively.

Where four characters per token falls apart

The ~4 chars/token rule comes from English prose and holds up there. Everywhere else it leaks:

• ▸ Code. Identifiers are not vocabulary entries — getUserById splits into several fragments, and every bracket, operator, and dot is a token of its own. Real source usually lands nearer three characters per token.
• ▸ JSON. Braces, quotes, colons, and commas are all tokens, and every record repeats every key. A 200-row API response spends a serious share of its budget re-stating field names.
• ▸ Whitespace. Newer OpenAI vocabularies added multi-space tokens precisely because indented Python was brutal under the GPT-2 tokenizer, where a run of indentation cost roughly one token per space. Which tokenizer you are on decides what your formatting costs.
• ▸ High-entropy strings. UUIDs, hashes, base64 blobs, minified bundles. There are no useful merges to find, so they shatter — a single UUID can cost more than a dozen tokens.
• ▸ Non-Latin scripts. A Chinese or Japanese character often costs a full token on its own, against roughly four English characters per token. Arabic, Hebrew, and Devanagari usually do worse, fragmenting toward individual UTF-8 bytes. Per character, that is several times the price for the same meaning.

Counts do not port between model families

There is no universal token. Every model family ships its own tokenizer, and a bigger vocabulary means fewer tokens for the same string — which is why the same paragraph can differ by a double-digit percentage across providers, and by much more if it is not English. OpenAI publishes tiktoken (cl100k_base for the GPT-4 generation, o200k_base for newer models). Anthropic does not publish Claude's tokenizer and gives you a count_tokens endpoint on the Messages API instead. Google exposes countTokens for Gemini. Open-weight models ship their tokenizer with the checkpoint, so AutoTokenizer gives you an exact count locally.

So a count measured on one model is an estimate on every other one. Migrating a prompt across providers can quietly change both your bill and whether it still fits.

Tokens are the unit of your bill and your window

Both limits that matter are denominated in tokens, and neither one is measured in characters. Pricing is per million input and output tokens, with output typically several times the price of input. The context window is a token budget shared by the system prompt, tool definitions, retrieved chunks, conversation history, the user turn, and the reply — so every token you spend on context is one the model cannot spend on an answer. Prompt caching, embedding limits, and rate limits are all measured the same way.

Which is why a text-only count is a floor, not a total. Tool and function schemas are serialised into the request and billed as input. Images are converted to tokens based on their dimensions. Neither shows up in the box above.

When to stop estimating

Estimates are for sizing decisions: is this prompt roughly 2k tokens or roughly 200k, is this chunk in the right ballpark, is this feature going to cost cents or hundreds. The moment a number becomes load-bearing, switch to the real tokenizer — before a budget forecast at volume, before anything that can hard-fail on a context limit in a batch job, before you settle on a chunk size for a vector index, and always when the content is code, JSON, or a non-Latin script.