Recurrent Neural Networks (RNNs)

Word2vec gives every word a fixed vector but ignores the order of words in a sentence. A Recurrent Neural Network (RNN) fixes this by reading text one token at a time and keeping a memory of what it has seen so far. The same network step is reused at every position, and its memory (the hidden state) is passed from one step to the next. This page covers how an RNN works, how it is arranged for classification, multi-classification and generation, and how it compares with Word2vec.

An RNN processes a sentence from left to right. At each step it takes the current token and the hidden state from the previous step, and from them it produces a new hidden state and, if needed, an output. Because the hidden state carries information from earlier words, the prediction at each position depends on everything read before it. Drawn out over time ("unrolled"), the RNN looks like a chain of identical blocks that share the same weights.

Sentence: "A cute teddy bear is reading"

Step 1: input "A"           -> predict "cute"
Step 2: input "cute"        -> predict "teddy bear"
Step 3: input "teddy bear"  -> predict "is"
Step 4: input "is"          -> predict "reading"

At step 4 the hidden state still carries information about
"A cute teddy bear", so the model can guess a word like "reading".
    

The three NLP task families from the earlier overview can all be built from the same RNN cell. What changes is where the outputs are read: only at the end, at every step, or after the whole input has been read. In the diagram, green boxes are inputs and the starting state, blue boxes are RNN cells, and red boxes are outputs.

The RNN reads the whole input text token by token and ignores the outputs of the earlier steps. Only the final hidden state, which now summarizes the entire text, is used to produce one label. This fits tasks such as sentiment analysis, where a whole opinion is mapped to a single class.

Input : "The food was great but the service was slow"
        x(1)  x(2) ...                              x(Tx)
Output: one label from the last step -> Mixed / Neutral
    

Here the RNN produces an output at every time step, so the output sequence has the same length as the input. Each tag is predicted using the current token and the hidden state built from all earlier tokens. This is the layout used for part-of-speech tagging and named entity recognition.

Input : Sundar   Pichai   works   at   Google
Output: PERSON   PERSON   O       O    ORG
    

For generation, the RNN first reads the entire source text without producing any output, which builds up a summary in its hidden state. It then continues from that state and writes the output tokens one by one. Because the reading part and the writing part are separate, the output length does not have to match the input length. This layout is used for machine translation and summarization.

Source (read first): "How are you?"        (English, 3 tokens)
Output (written)   : "¿Cómo estás?"        (Spanish, 2 tokens)
    

Very simple, yet powerful: The model is a small shallow network trained on an easy prediction task, but the vectors it learns work well for many downstream tasks. Intuitive embeddings: Similar words get similar vectors, so distances and directions in the space have a clear meaning (for example, "teddy bear" is close to "soft").

Word order does not count: CBOW treats the context as a bag of words, so "dog bites man" and "man bites dog" look alike to it. Embeddings not context aware: Each word has one fixed vector, so a word with several meanings, such as "bank" in "river bank" and "bank account", gets the same vector in both sentences.

Word order matters: The hidden state is updated token by token, so the model can tell "dog bites man" from "man bites dog", and the representation of a word depends on its context. State-of-the-art results: RNNs, especially LSTMs, gave the best results on many NLP tasks such as translation and tagging before newer architectures such as Transformers became popular.

Vanishing gradient problem: During training, the learning signal is passed backward through every time step and tends to shrink along the way. As a result, a plain RNN struggles to learn links between words that are far apart, and LSTMs were designed with gates to ease this problem. Slow computations: Each step needs the result of the previous one, so tokens cannot be processed in parallel, which makes training and inference slow on long texts.