Explain WordPiece and SentencePiece.

In this question, we will learn the practical and theoretical differences between WordPiece and SentencePiece so you can explain why a model uses one or the other and how they affect tokenization and training.

We will cover the following:

• The direct answer and short summary
• The intuition that makes subword tokenization sensible
• How each algorithm actually works with a concrete example
• Practical differences, deployment implications, and a comparison table
• Tradeoffs and failure modes

WordPiece and SentencePiece are both subword tokenization methods that reduce out of vocabulary issues and control vocabulary size; WordPiece uses a greedy likelihood-based expansion on words with explicit continuation markers, while SentencePiece can train either BPE merges or a probabilistic unigram model and tokenizes raw text without pretokenization. In practice, WordPiece is common in BERT-style pipelines and SentencePiece is flexible for multilingual and byte-level needs.

The intuition (an analogy that makes it click)

Think of building a toolbox for words. If you start with only single letters, many long words are expensive to represent. Subword tokenizers pick useful pieces to store in the toolbox so you can assemble words cheaply. WordPiece picks pieces by testing which additions best explain your training set in a greedy way. SentencePiece either repeatedly merges frequent pairs like a craftsman combining parts, or keeps a large set and prunes unlikely pieces based on a probabilistic model. Both reduce the need to store every full word and both let models generalize to rare or new words.

How it actually works

We will show a compact worked example with the word "playing" and a tiny toy corpus: "I am playing" and "She was playing". Start with characters and space markers.

WordPiece (conceptual steps)

1. Initialize vocabulary with characters and a special unknown token.
2. Consider candidate merges or subwords and pick the one that most increases corpus likelihood under a simple model.
3. Repeat until you reach desired vocabulary size.

SentencePiece has two modes. The BPE-like mode merges frequent adjacent symbol pairs greedily. The Unigram mode starts with a large candidate vocabulary and removes tokens that reduce the marginal likelihood the least. The Unigram model optimizes the sum of log marginal probabilities over training sentences. If we denote a sentence as $s$ and a segmentation as $z$ with tokens $w_t$, the marginal sentence probability is

Training maximizes the sum of $\log p(s)$ over the corpus by pruning or updating $p(w)$.

Worked example tokens for "playing":

Here $\_play$ denotes a token that includes a leading space marker. WordPiece typically relies on white-space-tokenized words first then subword split while SentencePiece treats raw text uniformly.

Practical differences and deployment implications

Vocabulary training

• WordPiece: often trained on pretokenized words. It greedily adds subwords by increasing a likelihood or frequency objective.
• SentencePiece: trains on raw text. It can use BPE merges or the Unigram language model. Unigram is probabilistic and can produce alternate segmentations.

Markers and byte handling

• WordPiece commonly uses continuation markers like "##" to show a subword is not a word start.
• SentencePiece often represents spaces as an explicit symbol like leading underscore, or uses byte-level encoding so any input is representable without unknown tokens.

Determinism and sampling

• WordPiece tokenization is deterministic given the vocabulary and greedy longest-match rules.
• SentencePiece Unigram can provide probabilistic sampling of segmentations which can help data augmentation during pretraining.

Performance

• Inference speed is similar when using greedy longest-match tokenization. Unigram sampling adds overhead if used at runtime.
• Training complexity: Unigram optimization is heavier than greedy merges but produces a compact vocabulary with a principled likelihood objective.

Tradeoffs and failure modes

• If you need robust byte-level handling for many scripts or noisy text, SentencePiece byte-level mode is safer.
• If you prefer simple deterministic splits and reuse of existing white-space tokenization, WordPiece is straightforward.
• Probabilistic segmentation helps cover unseen morphological variants but can complicate deterministic decoding.

Questions the interviewer might ask

Some follow-up questions you might get:

How does Byte-Pair Encoding differ from WordPiece? BPE is a greedy pair-merging algorithm that merges the most frequent adjacent symbol pairs. WordPiece is similar conceptually but the original WordPiece uses a likelihood-based or frequency-informed greedy expansion tailored to words and often uses continuation markers.

Why use a leading underscore in SentencePiece? The leading underscore encodes a space or word boundary so tokenization preserves where words start. That helps models learn when a token is a word-initial piece versus a continuation piece.

What is the Unigram model objective in SentencePiece? It maximizes corpus log-likelihood by marginalizing over possible segmentations: sum over sentences of $\log p(s)$ with $p(s)=\sum_z \prod_t p(w_t)$. This probabilistic view allows pruning less useful tokens.

When does probabilistic segmentation help? During pretraining, sampling alternative segmentations can act as data augmentation and help models generalize to morphological variation. At inference, deterministic splits are typical for stable behavior.

How do these choices affect model size and performance? Smaller vocabularies reduce embedding parameters but increase sequence lengths. Larger vocabularies save sequence length but increase embedding table size. Choose a balance based on compute, memory, and language morphology.

Some things to note:

• Vocabulary markers differ across toolkits; make sure token-to-text conversion matches training.
• Byte-level handling reduces unknowns but can increase token count for languages with long multi-byte characters.

What the interviewer is really testing

They want to see that you understand subword tokenization tradeoffs: algorithmic differences, how vocabularies are trained, and the impact on model inputs and performance. Showing familiarity with Unigram marginal likelihood, BPE greedy merging, and practical issues like markers and byte handling demonstrates both theoretical and deployment-ready knowledge.