6. Sequence Models

Chapter 6
Sequence Models

In this lecture we go beyond models that process fixed-size vectors and tensors, and produce fixed-size outputs, to models that can handle input and output of arbitrary length. These are known as sequence models, and they come in a variety of flavours. We will introduce the idea of a recurrent neural network (RNN) for processing sequence data, and then show how these can be used to solve several tasks involving language. In the next lecture we will discuss the transformer model, which is another approach to processing sequences.

So far in this course we have been concerned with processing fixed-size objects, such as \(n\)-dimensional feature vectors and \(H\)-by-\(W\) pixel images, which has been useful for tasks such as image classification and object detection. Being able to process sequential data is also very important. There are many examples of tasks involving such data, including image captioning [5, 52], visual question answering [7], video classification [14, 93], natural language translation [96], and vision-and-language navigation [6, 47]. These can be categorized as being one-to-many, many-to-one or many-to-many based on the number of elements in the input and output sequences, respectively, as illustrated in Figure 80.

Unlike images and labels, a sequence has arbitrary length making it difficult to process with the models that we studied in previous lectures.1 We need a network architecture that can handle data of any length.

Unlike an image or label, a sequence model allows inputs and outputs to have arbitrary length. Different combinations are suited for different applications, inc
Figure 80: Unlike an image or label, a sequence model allows inputs and outputs to have arbitrary length. Different combinations are suited for different applications, including (a) image classification, (b) image captioning, (c) language-prompted image generation, and (d) language translation. Again the encoder-decoder paradigm (e) is useful when thinking about these applications.
  1. 1. We did seen examples of things like bounding box proposals, which are in principle sets or arbitrary size, but these were always padded or truncated to fixed-length lists stored in fixed-size tensors. We also saw fully convolutional networks for image segmentation, which can be viewed as a very special case of the types of models that we will see in this lecture.