Token Representations and Word2vec

After text is split into tokens, each token must be converted into numbers before a model can use it. The simplest option is one-hot encoding, but it carries no information about meaning. Learned embeddings fix this by placing similar tokens close together in a vector space. Word2vec is a classic method that learns such embeddings from raw text, using two training tasks called CBOW and Skip-gram.

In one-hot encoding, every token gets its own axis: the vector is all zeros except for a single 1 at the token's position. Because the axes are perpendicular, the dot product of any two different tokens is 0, so "teddy bear" looks as unrelated to "soft" as it does to "book". In a learned embedding, each token is a short dense vector whose values are learned from data, so related tokens point in similar directions. Now the dot product of "teddy bear" and "soft" is close to 1, while "teddy bear" and "book" stay close to 0.

Vocabulary: [soft, teddy bear, book]

One-hot  : soft = [1, 0, 0]
           <teddy bear, soft> = 0     (no similarity captured)

Embedding: soft = [0.95, 0.32, 0.01]
           <teddy bear, soft> ~ 1     (similar meaning, similar vectors)
           <teddy bear, book> ~ 0     (unrelated meaning)
    

Word2vec trains a neural network on a proxy task, which is a prediction problem that is not the real goal but forces the model to learn useful representations. The network reads billions of words of raw text, so no manual labels are required. The proxy task is only a means to an end: once training is done, the prediction part is discarded and the learned embedding layer (one vector per word) is kept. Words that appear in similar contexts end up with similar vectors.

Training text : "A cute teddy bear is reading a book"
Proxy task    : predict words from their neighbors (or the reverse)
Result        : an embedding vector for every word in the vocabulary,
                e.g. "teddy bear" ends up near "toy" and "soft"
    

CBOW looks at the words around a position and tries to predict the missing word in the middle. The context word vectors are combined (averaged), which is why it is called a "bag" of words: the order of the context words is ignored. It is fast to train and works well for frequent words. The training signal teaches the model that words which fill the same gap should have similar embeddings.

Sentence : "A cute teddy bear is reading"
Input    : [A, cute, ___, is, reading]    (context words)
Output   : teddy bear                     (word to predict)
    

Skip-gram reverses CBOW: it takes one word and tries to predict the words that appear around it. Each (center word, context word) pair becomes a separate training example, so a single sentence gives many examples. This makes training slower, but it usually represents rare words better. The model learns that a word should have an embedding that is useful for guessing its neighbors.

Sentence : "A cute teddy bear is reading"
Input    : teddy bear                     (center word)
Output   : A, cute, is, reading           (context words)

Training pairs: (teddy bear, A), (teddy bear, cute),
                (teddy bear, is), (teddy bear, reading)
    

Both methods use the same idea, that a word is defined by the words around it, but they predict in opposite directions. CBOW is quicker and suits large amounts of text with common words, while Skip-gram handles smaller datasets and rare words better.

To view tokenization done in various llms try using tiktokenizer