What is tokenization in LLMs?

In this question, we will learn what tokenization is in LLMs, why it matters, and how different tokenizers change the model input and behavior.

We will cover the following:

• The intuition (an analogy that makes it click)
• How it actually works (the real mechanics, with a worked example)
• Tokenization strategies compared
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: Tokenization is the step that converts raw text into a sequence of discrete tokens and corresponding numeric ids that an LLM can process. Token choices (character, subword, word, or byte) determine the vocabulary size $V$, the number of tokens $T$ used to represent input, and how the model handles rare or out-of-vocabulary text. Good tokenization balances compactness, generalization, and robustness.

The intuition (an analogy that makes it click)

Think of text as a spoken sentence and tokenization as choosing the words you will hand to a translator. If you hand single letters the translator must stitch many pieces together but can handle any spelling. If you hand full words the translator works faster but fails on unknown words. Subword tokenization sits in the middle, like handing common roots and suffixes so the translator recognizes patterns while still splitting novel words into known pieces.

How it actually works (the real mechanics, with one concrete worked example)

At a high level, tokenization has two phases: preprocessing and segmentation. Preprocessing normalizes text: trimming, optional lowercasing, and handling of whitespace or punctuation. Segmentation maps text to tokens based on the tokenizer rules and then to numeric ids via a vocabulary.

We can write the total tokens for a sentence with $W$ words as

where $t_i$ is the number of tokens used to represent word $i$. Different tokenizers change the distribution of $t_i$.

Worked example: tokenize the sentence "I can't believe it's 2025!" using three strategies.

Notes on this table: whitespace tokenization keeps whole words but fails on punctuation splitting choices; subword tokenization often splits contractions and numbers into meaningful subpieces; byte-level BPE can represent any input exactly but produces more tokens on average.

A common trained tokenizer builds a vocabulary of size $V$ and learns merges so that frequent subwords are single tokens. During inference, the tokenizer greedily or using longest-match rules segments text into tokens and then maps each token to its id in $\{0,\dots,V-1\}$.

Tokenization strategies compared

We commonly see these families of tokenizers:

• Character-level: each character is a token. Robust but long sequences; $t_i$ equals the length of the word.
• Word-level: tokens are whole words or punctuation. Compact when vocabulary covers the domain but cannot handle unknown words without an unknown token.
• Subword (BPE, WordPiece, Unigram): split words into frequent subword units. Good tradeoff between vocabulary size and token length.
• Byte-level BPE: operates on raw bytes and learns merges. Handles any Unicode text and keeps vocabulary manageable.

Compare properties in a short table.

Practical issues and tips

• For multilingual models choose byte-level or large subword vocabularies so scripts are handled consistently. We often prefer byte-level BPE for robustness across languages.
• Numbers, dates, and rare proper nouns can fragment into many tokens. If you must optimize for cost, consider a tokenizer trained on your domain so frequent tokens are single pieces.
• Tokenizers are deterministic at inference. If you change tokenizers or vocabularies, the model input ids change and the model may no longer behave as intended.

Tradeoffs and failure modes

Tokenization decisions affect model size, training speed, and inference cost. Small $V$ means the model embeds fewer tokens but may produce longer token sequences and slower inference per input. Large $V$ reduces sequence length but increases embedding matrix size and memory.

Common failure modes:

• Poor handling of rare orthography or emojis if the tokenizer was not trained on similar data.
• Excessive tokenization of structured data, such as long hex strings or base64, leading to very long $T$ and higher cost.

Questions the interviewer might ask

Some follow-up questions you might get:

How does tokenizer vocabulary size affect model parameters? A larger vocabulary $V$ increases the embedding matrix size roughly by $V$ times embedding dimension, which raises memory and parameter count linearly.

Why do subword tokenizers help with rare words? Subword tokenizers break rare words into known morphemes or pieces so the model can reuse embeddings and generalize compositionally rather than treating each rare word as an unknown token.

What are byte-level tokenizers good for? Byte-level tokenizers can represent any byte sequence, so they are robust to unusual Unicode, mixed scripts, or adversarial input, at the cost of more tokens for common words.

How do tokenizers handle spaces and punctuation? Many tokenizers encode whitespace as explicit tokens or as part of tokens using prefixes. This preserves word boundary information which helps the model learn where tokens join.

When might you retrain a tokenizer? Retrain when your domain contains frequent tokens not well represented by the base tokenizer, for example technical logs, DNA sequences, or a low-resource language with unique morphology.

Some things to note:

• Tokenization is not neutral; it encodes inductive biases into the model input.
• Always log token counts during dataset preparation to estimate training and inference cost.

What the interviewer is really testing

They want to know you understand how raw text becomes model input, and why design choices like vocabulary $V$ and token granularity matter for model capacity, cost, and robustness. They are checking for practical awareness: how tokenization affects sequence length, embedding size, multilingual handling, and failure modes in real systems.