Section 6.6
Applications

We now discuss three applications for sequence models using recurrent neural networks in computer vision.

6.6.1 Image Captioning

One of the first applications of recurrent neural networks in computer vision was to automatically describe the contents of images in natural language. In other words, image captioning. Conceptually, an image captioning model starts by encoding the image into some latent space followed by decoding the latent representation, by a sequence model, into a sentence that describes the image. Indeed, this was exactly the recipe followed by the early deep learning captioning model of Karpathy and Fei-Fei [52]. Here a convolutional neural network (CNN) is used to encode the image, which provides the initial hidden state for a recurrent neural network (RNN) language model decoder. The CNN is typically pre-trained on the ImageNet dataset¹ and the RNN trained on paired image and caption data to minimize cross-entropy loss of predicted captions. An illustration of the model is shown in Figure 90.

Anderson et al. [5] significantly improved over the earlier captioning models by incorporating a bottom-up and top-down process, and introducing the notion of attention when generating each word.² See Figure 90(b). This model is still the de facto standard against which new captioning models are compared.

Briefly, the idea of bottom-up top-down (BUTD) attention is to focus on different parts of the image as the caption is generated so as to condition the next-word probability distribution. A set of candidate salient regions \(V = \{v_1, \ldots, v_n\}\) are generated from two sources: a standard \(m \times m\) CNN feature map and bounding boxes from an object detector, e.g., Faster-RCNN. See Figure 91. Features on the salient regions from this bottom-up process are combined using a top-down process that weights them based on current context supplied by the RNN latent state, \(h_t\), as

where \(g\) is a learnable attention function, e.g., multi-layer perceptron. The method then follows the standard language model approach to generate the next word,

As discussed above, beam search can be used to improve the probability of the generated sequence, and hence the quality of the captions.

Several automatic metrics have been proposed to evaluate captioning models. Most of these (BLEU, ROUGE, METEOR, and CIDer) are based on comparing \(n\)-grams in the generated caption with those from a reference set of human provided captions. An alternative approach called SPICE [3] is to construct a scene graph [59], which encodes entities and their relationships in an image, and then evaluates how well the nouns, verbs and adjectives in generated caption covers the scene graph. However, caption evaluation is notoriously difficult because images can be described in numerous different ways. Ultimately, human assessment is the best judge, albeit very expensive on a large scale.

6.6.2 Visual Question Answering

Like image captioning, visual question answering (VQA) [7] combines image and language modalities, in this case to answer an arbitrary question about an image. The question and answer are phrased in natural language, or sometimes answers can be selected from a discrete set of options. Unlike image captioning, the input to the model is both an image and a text sequence, i.e., the question. Examples are shown in Figure 92.

Typical models for VQA involve encoding the image and question into latent representations using a pre-trained CNN and RNN, respectively. The combination of image and question latent representations is then decoded into an answer. For open-ended answers a language decoder model is used, whereas for multiple-choice the task is reduced to a multi-label classification problem and an MLP with softmax output layer is used.

Early VQA datasets and models were plagued with biases, e.g., answering “two” whenever it was asked how many of anything is in the image. This has been addressed with bigger datasets and more careful collection to avoid simple biases. More advanced models also include specialized modules for recognising logos and reading text so that they can answer more sophisticated questions about the image.

6.6.3 Vision and Language Navigation

Vision and language navigation (VLN) is a very interesting application that requires reasoning over both visual and language modalities. It was first introduced by Anderson et al. [6]. Here the task is for an agent to navigate through an environment following a natural language instruction, where the instruction refers to visual features of the environment (e.g., “exit the room through the glass doors”). Initial research focused on virtual environments with the agent only allowed to visit a discrete set of locations in the environment. At each location the agent would get to observe the environment from that location and then have a small set of choices of where to go next (or to stop).

The field of VLN has since exploded to entertain many variations, including the ability to pre-explore and map out an environment, operate in a continuous environments, and enhancing instructions with actions to take when reaching the destination (e.g., “bring me a spoon”). Hong [47] has created a visual summary of the history and current state of VLN research. See Figure 93. His PhD thesis provides comprehensive details.

The VLN task can be viewed as an example of an AI planning or sequential decision making problem, wherein an agent is required to take a sequence of actions to achieve some goal (e.g., navigating from a starting location to a destination location). We will explore this in more detail later in the course when we introduce Markov decision processes (MDPs) and deep reinforcement learning (RL) [97].