Tokenization Summary

Tokenization is the step that splits raw text into smaller units called tokens, which are then mapped to numbers and fed to a model. The size of these units is a design choice: whole words, pieces of words, or single characters. Each choice has trade-offs in vocabulary size, sequence length, and how well the model handles rare or new words. A word that is missing from the model's vocabulary is called OOV (out-of-vocabulary), and it is usually replaced by a generic unknown token.

Word-level tokenization splits text on spaces and punctuation, so every word becomes one token. It is the most straightforward approach and gives one embedding per word.

Input : "I love running"
Tokens: ["I", "love", "running"]
    

Simple: No special algorithm or training is needed, because splitting on whitespace and punctuation is usually enough. Interpretable: Every token is a real word, so it is easy to inspect what the model sees and what each embedding stands for.

Risk of OOV: The vocabulary is fixed, so any new word, name, typo, or rare term becomes an unknown token and its meaning is lost. For example, "blockchainify" would be replaced by [UNK]. Does not leverage knowledge of root: Words such as "run", "running", and "runner" are treated as three unrelated tokens, so the model cannot reuse what it learned about the shared root.

Subword tokenization keeps common words whole and breaks rare words into smaller, reusable pieces. Algorithms such as Byte Pair Encoding (BPE) and WordPiece build this vocabulary from the training data. This is the standard choice in modern models such as BERT and GPT.

Input : "unhappiness"
Tokens: ["un", "##happi", "##ness"]    (WordPiece style)
    

Leverages common prefixes and suffixes: Pieces such as "un-", "-ness", and "-ing" are shared across many words, so the model can understand a new word from parts it already knows. Learned from the data: The vocabulary is not hand-written. BPE repeatedly merges the most frequent pair of symbols, so frequent patterns get their own tokens automatically.

Risk of OOV, though less than word-level: A rare word is split into smaller known pieces rather than being replaced by an unknown token, so OOV is much less likely. It can still happen if a character or symbol never appeared in the training data, such as a rare emoji or script.

Character-level tokenization treats every individual character as a token. The vocabulary is just the alphabet, digits, and symbols, but each sentence turns into a much longer sequence.

Input : "cat"
Tokens: ["c", "a", "t"]
    

Small chance of OOV: Any word is built from a small, fixed set of characters, so nearly any input can be represented, including new words and names. Robust to casing and misspellings: A noisy input like "RoBUsT tO CASinG anD MIspeliNGs" or "helo wrld" still breaks down into familiar characters, so the model can cope with it instead of failing on an unseen word.

Makes computations slower: A sentence becomes many more tokens than with words, and models such as Transformers get more expensive as the sequence grows. Embeddings not interpretable: A single character like "t" carries almost no meaning on its own, so its embedding is hard to interpret, and the model must learn meaning from long character combinations.