Making RNNs Actually Work: LSTMs, Bidirectionality, and the Encoder-Decoder

Stacking, Bidirectionality, the Encoder-Decoder, and LSTMs Last post ended with a simple RNN and three promises: LSTMs, bidirectional RNNs, and attention. This post delivers the first two, plus the refinements that turn a working-on-paper RNN into something you'd actually deploy. By the end, you'll know how to stack RNNs for depth, why reading a sentence backward as well as forward (bidirectionality) makes representations sharper, how the encoder-decoder turns one sequence into a different one for machine translation, and exactly what breaks in a simple RNN that LSTMs and GRUs were invented to fix. Attention, the fix for the last problem we'll hit, gets its own home in the transformer post, so we'll stop right at the edge of it. The simple RNN was the idea. This post is the engineering. A vanilla RNN carries a thread of hidden state through time, but in practice, that thread frays on long sequences, only sees the past, and bottlenecks everything through one final vector. Each section here is a fix for one of those problems. Put them together, and the path from "RNN" to "transformer" looks less like a leap and more like a series of obvious next steps. Stacking RNNs for Depth The first refinement is the easy one. Nothing says an RNN's output has to go straight to a prediction. You can feed the entire output sequence of one RNN as the input sequence to another. Then another. These are stacked RNNs (also called deep RNNs), and they usually outperform a single layer. Why does depth help? The same reason it helps in vision. Each layer learns representations at a different level of abstraction. The lower layers pick up fundamental, local properties; in language, that's roughly the level of parts of speech and named entities. The higher layers compose those into bigger groupings: descriptive phrases, "this is the answer to a question," and so on. We can't point at layer 3 and say "this one does coreference." But the theory holds up well enough that researchers have probed m