6. Sequence Models › 6.1 Recurrent Neural Networks

Section 6.1
Recurrent Neural Networks

A recurrent neural network (RNN) is a model that processes an input sequence one element at a time. It also takes part of the output and feeds it back into itself as an additional input. This allows it to keep state information based on previously processed inputs. A schematic diagram of an RNN is shown in Figure 81. Let \(x_t\) be the \(t\)-th element in the input sequence and \(h_{t-1}\) be the partial output from the previous time-step. Then the output of the RNN is computed as,

\begin{align} (y_t, h_t) &= f(x_t, h_{t-1}; \theta) \tag{203}\end{align}

where the same function \(f\) with same parameters \(\theta\) is applied at each time-step. Variable \(h_t\) is called the hidden or latent state. It is initialized to some nominal value, \(h_0\), at the start of the sequence. Note that as defined, the RNN produces one output for every input, and is therefore a many-to-many model. It can be made into a many-to-one model by waiting until the entire sequence is processed and discarding all but the last output.

A recurrent neural network (RNN) is used to process sequence data
Figure 81: A recurrent neural network (RNN) is used to process sequence data.

A concrete example of an early recurrent neural network proposed by cognitive scientist Elman [25] is shown in Figure 82. Here the latent state and output are computed as,

\begin{align} h_t &= \textbf{tanh}(W_h h_{t-1} + U_h x_{t} + b_h) \tag{204}\\ y_t &= \textbf{sigmoid}(W_y h_{t} + b_y), \tag{205}\end{align}

respectively. This is just a variant of the multi-layer perceptron where the hidden state \(h_t\) and output \(y_t\) are elementwise non-linear transformations of linear functions of the current input \(x_t\) and previous hidden state \(h_{t-1}\). The learnable parameters are \(W_h\), \(U_h\), \(b_h\), \(W_y\) and \(b_y\).

Concrete example of an RNN model
Figure 82: Concrete example of an RNN model.

We can view convolution over an arbitrarily long input signal as a very simple recurrent neural network. For brevity let us consider the one-dimensional case. Let \(a \in \reals^p\) be a \(p\)-length filter kernel, and let \(x_t \in \reals\) be the value of the input signal at time \(t\). Initialize \(h_0 = (0, \ldots, 0) \in \reals^{p}\). Then we can implement convolution as a recurrent network with

\begin{align} h_t &= (h_{t-1}[2:p], x_t) \tag{206}\\ &= (x_{t-p+1}, \ldots, x_{t-1}, x_t) \tag{207}\\ y_t &= a^T h_t \tag{208}\end{align}

where the output is delayed by \(p-1\) time steps, which is how long it takes for the model to process the receptive field of the filter kernel. However, general RNNs are much more expressive.