How Tokenization Works: Turning Text into Numbers for Language Models

A language model never sees text directly. Before any learning happens, input text is split into tokens, and each token is mapped to an integer ID. Tokenization decides what the model can easily represent, how much each prompt costs to process, and how much text fits into its context window. This article explains the common method, byte pair encoding, and the tradeoffs behind it.

Why not just use letters or words?

Two simple options come to mind. Splitting text into individual characters gives a tiny vocabulary, but sequences become very long, and the model must learn spelling from scratch before it learns meaning. Splitting text into whole words gives short sequences, but the vocabulary explodes. Every inflection, misspelling, rare name, and new coinage needs its own entry, and any word not in the list becomes an unknown token the model cannot interpret.

Subword tokenization sits between these extremes. Common words become single tokens, while rare words break into familiar pieces. A word like "unbelievably" might become a few tokens that the model has seen often, so it can still reason about the parts.

Byte pair encoding in plain steps

Byte pair encoding, often abbreviated BPE, was adapted for language models from a 1994 data compression technique. The training procedure is short:

1. Start with a vocabulary of individual characters or bytes.
2. Count every adjacent pair of symbols in a large training corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary.
4. Repeat until the vocabulary reaches a target size.

Consider a toy corpus with the words low, lower, lowest, and newer. The pair of letters "l" and "o" appears in every low word, so it is merged early. Then "lo" and "w" merge into "low". Later merges may join "e" and "r", producing "er", which appears in lower and newer. The end result is a vocabulary where "low" is one token, while rare forms are assembled from pieces.

Encoding new text reverses the process. The tokenizer applies the learned merges in the same order, so the same input always produces the same tokens.

Real vocabularies and their sizes

Production tokenizers typically have tens of thousands to a few hundred thousand entries. The choice is a tradeoff between sequence length and vocabulary size.

Tokenizer	Example model family	Vocabulary size	Notes
Byte-level BPE	GPT-2	50,257	Falls back to raw bytes, so no input is unknown
WordPiece	BERT	30,522	Merges chosen by likelihood, uses continuation markers
SentencePiece BPE or unigram	Many multilingual models	Varies, often 32,000 to 256,000	Works on raw text without whitespace pre-splitting

The GPT-2 vocabulary of 50,257 tokens is a commonly cited figure, and it illustrates the design. Its base alphabet covers all 256 possible byte values, and merges sit on top of that. Any byte sequence can therefore be encoded, even emoji or unusual scripts, by falling back to smaller pieces.

Why token counts matter

Context windows, API limits, and processing cost are all measured in tokens, not words. A common rule of thumb for English is that one token corresponds to roughly three quarters of a word, so a thousand tokens is roughly 750 words. This is only an estimate. Code, numbers, URLs, and non-English text often produce more tokens per word, because the tokenizer's merges were learned mostly from its training data.

This matters in practice. A prompt that looks short on the page may use many more tokens than expected, especially if it contains dense technical content. Counting tokens with the tokenizer that matches the model is the only reliable method.

Whitespace and capitalization are part of the token

Many tokenizers attach leading spaces to words. The string " hello" and "hello" can be different tokens, with the first usually the more common form in running text. Capitalization also changes tokenization, so "Model" and "model" may be separate entries. This means small formatting changes can alter the token sequence, even when the meaning is the same.

These details are not cosmetic. A model may behave slightly differently when the spacing of a prompt changes, because the input IDs have changed.

What tokenization cannot do

Tokenization is a fixed preprocessing step. The model does not decide how to split text while reasoning. Consequently, tasks that require seeing individual characters can be harder than they appear. Counting the letters in a word is difficult when the word is stored as two or three multi-letter tokens, because the model never receives the letters as separate inputs.

Tokenization also shapes multilingual performance. A tokenizer trained mostly on English may use several tokens for a single word in a less represented language. The same content then costs more context and more processing, and the model has fewer learned statistics for those pieces.

Designing a tokenizer

Designers choose the vocabulary size, the training corpus, and whether to normalize text before splitting. A larger vocabulary shortens sequences and speeds up processing, but each token needs its own embedding vector, which increases the size of the model's input and output layers. A smaller vocabulary keeps those layers compact, but produces longer sequences and more work per sentence.

The training corpus matters as much as the size. Merges reflect whatever text the tokenizer saw, so a corpus heavy in code produces code-friendly tokens, and one heavy in a single language favors that language. Tokenizers are therefore part of a model's design, not a generic utility.

Special tokens and chat formatting

Beyond ordinary text, tokenizers define special tokens that carry structure. Examples include a beginning-of-sequence marker, an end-of-sequence marker that tells the model when to stop, and padding tokens that make batches the same length. Chat models add role markers that separate system instructions, user messages and assistant replies. These are reserved vocabulary entries, and ordinary text splitting never produces them.

Chat formatting is part of how a model is trained and prompted, so the template must match the one used during training. Sending the same words with a different template can degrade behavior, which is why chat interfaces hide the template from the user and apply it automatically.

Key takeaways

Language models operate on tokens, which are subword pieces learned from a corpus. Byte pair encoding builds them by repeatedly merging the most frequent adjacent pairs.

Token counts determine context limits and cost. English prose averages roughly three quarters of a word per token, while code and other languages often use more.

Tokenization shapes what models find easy or hard, including character-level tasks and multilingual text. Use the tokenizer that belongs to the model when you need accurate counts or want to understand its quirks.