6. Sequence Models › 6.3 Long Short-Term Memory Model (LSTM)

Section 6.3
Long Short-Term Memory Model (LSTM)

To address the vanishing and exploding gradient problem, Hochreiter and Schmidhuber [46] introduced the long short-term memory model (LSTM). An architecture diagram for the LSTM, known as a cell, is shown in Figure 85. Like blocks in a residual network, the LSTM cell attempts to induce better flow of information by short-circuiting some of the operations that may lead to suppressed gradients. It does this by introducing a cell memory, \(c_t\) and gating functions in addition to the cell output/latent state \(h_t\). Three gating functions are defined, \(f_t\), \(i_t\) and \(o_t\), as follows,

\begin{align} f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) & \text{forget activations} \tag{214}\\ i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) & \text{input activations} \tag{215}\\ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) & \text{output activations} \tag{216}\end{align}

The cell memory \(c_t\) and output \(h_t\) are then updated as

\begin{align} c_t &= f_t \circ c_{t-1} + i_t \circ \tanh(W_c x_t + U_c h_{t-1} + b_c) & \text{cell memory} \tag{217}\\ h_t &= o_t \circ \tanh(c_t) & \text{cell output} \tag{218}\end{align}

where \(\circ\) denotes elementwise multiplication.

In words, the cell memory is updated as a combination of the previous memory and input signal, regulated by the forget and input gates, respectively. If the forget gate is low, then the cell forgets its previous value and is just updated with the input. If the input gate is low, then the cell is retains its previous memory (still modulated by the forget gate). The cell outputs its memory if the output gate is high.

Architecture of the long short-term memory (LSTM) model [46]
Figure 85: Architecture of the long short-term memory (LSTM) model [46].

As can be seen in the architecture diagram (and the equations defining the cell), there is a relatively simply path through from the cell memory at the previous time step \(c_{t-1}\) to the cell memory at the current time step \(c_t\). This mitigates the problem of vanishing gradients for long sequences.

The LSTM was the architecture of choice for processing sequential data for many years, but has recently been replaced by the transformer (which we will see in the next lecture).