Section 1.1
Tour of Application Domains

In this section we provide a very quick tour of problems studied in computer vision, natural language processing and other application domains. These fields are huge so this tour are just the tip of the iceberg to give you a flavour of the types of tasks being solved, and the language used to describe them. Many tasks are related and there there are many variants that differ in underlying assumptions, categories to be recognized, availability and quality of training data, additional or different imaging modalities, etc. All have been advanced by deep learning in one way or another.

1.1.1 Computer Vision

Perhaps the most basic task in computer vision requiring machine learning is image classification.¹ Here the goal is to assign a single label to the entire image—the label can be binary (e.g., yes/no to a particular question) or multi-class where the system returns a label from a pre-defined set, typically of categories of objects (e.g., dog, cat, etc.). Algorithms often also provide a confidence score in addition to the label. These tasks are somewhat ill-defined and a critical inbuilt assumption is that the image neatly fits into one of the categories and the object (for object label sets) is dominant and well-framed within the image.

Going beyond a pre-defined set of categories, open vocabulary image classification allows images to be labeled with any noun, even those not seen during training of the model. A typical task of this type might be what object is this? This is a very challenging task and methods are usually augmented with an external knowledge base or pre-trained large vision-and-language model.

Structured prediction is a more refined machine learning approach outputs structured data rather than a single categorical label or regression value. In the context of computer vision the canonical example is object detection where the algorithm is expected to output a set of object labels and bounding box pairs corresponding to objects found in the image. Another example is the (human) pose estimation task, where we are asking to locate all body joints such as hands, elbows, shoulders, etc. Here a skeleton model defines the joints and how they are connected in the model, which constrains the location of one joint with respect to another.

There is a very large and specialized research area on text recognition, which aims to locate and read text within the image. This has applications from number plate recognition for automating parking lot ticketing, to interpreting receipts for financial systems, to reading text in-the-wild for place recognition or language translation (e.g., of menus at foreign restaurants).

Pixel labeling tasks also go beyond giving a single label to the entire image and instead assign one or more class labels to every pixel in the image. These labels can represent semantics, such as a sky pixel, or indicate membership such as all pixels having the same (integer) label belong to the same object. The most basic example of the latter type is (unsupervised) image segmentation or over-segmentation, which breaks images up into meaningful regions sometimes called superpixels. This is often a pre-processing building block for some more sophisticated downstream algorithm.

Panoptic segmentation, for example, labels each pixel with a semantic category (form a pre-defined set) and an instance identifier. This allows separation of multiple instances of the same object, e.g., multiple people in a crowd. Background categories are often considered to not have different instances. Sometimes researchers will use the terms things and stuff to distinguish between categories with distinct well-defined boundaries (and hence countable instances) versus those with amorphous shape and extent such as sky and grass.

Pixel labeling tasks are not restricted to semantic categories. Monocular depth estimation is a task that labels every pixel with its depth to the camera. If we have two cameras (or two views from the same camera) with known relative pose then the problem becomes stereo depth estimation, which can be solved by understanding the geometry of image formation and using traditional computer vision techniques [39].

A large area of computer vision involves searching and sorting operations on which many downstream tasks are built. In feature matching we are interested in finding corresponding pixels between two (or more) images, i.e., pixels that represent the same location in 3D space. This helps with applications such as camera calibration, object tracking, and 3D reconstruction. In visual search we are interested in finding information based on a query image such as the name of a famous landmark. In attribute ranking a set of images is sorted according to some visual attribute, which is useful for applications like online shopping.

There are several tasks in computer vision that require reasoning over temporally evolving scenes. In tracking we are asked to detect and track objects as they move through an environment where they may sometimes be occluded by other objects in the scene. The task of activity recognition is akin to image classification but for video data where we ask what activity is happening in a (short) video clip. Just like image classification this can be extended to activity detection and activity segmentation for longer video clips.

Several computer vision tasks are concerned with editing or synthesizing images as the output. There has been a long history of work focused on image denoising, which aims to remove noise and restore corrupted images, image colorization, which aims to convert a black and white image to colour, and image inpainting, which aims to remove an object from an image and fill in the background with a plausible reconstruction. In video stabilization, post-processing is used to remove jitter caused by a handheld camera and motion blur caused by fast moving objects.

More recently, deep learning has enabled tasks such as novel view synthesis, which depicts a scene from a viewpoint not previously seen by the model, i.e., different to where the image (or set of images) was originally taken. Neural radiance fields (NeRFs) are a popular class of deep learning model for novel view synthesis. Another task is restyling, which changes the style but not the viewpoint or content of a scene, e.g., depict an adult as a child. We can also create completely novel (fake) images and videos using generative AI, which involves both vision and language understanding.

1.1.2 Natural Language Processing

One of the core areas of natural language processing (NLP) is understanding the structure and meaning of language, including morphology, syntax and semantics. Morphology characterises the formation of words, e.g., “running”, “ran” and “runs” are all related to the dictionary word “run.” Syntax is about how words are combined into grammatically valid sentences. It involves identifying parts-of-speech such as nouns and verbs, and parsing sentences to understand the relationship between words. Semantics is concerned with the meaning of words, sentences and larger pieces of text and spoken language. For example, recognising names and associating pronouns with a particular person or people (named entity recognition) and distinguishing between the meanings of words that have the same spelling or sound (word sense disambiguation), e.g., “bank” as a financial institution or the side of a river. The ability to automatically parse text and perform information extraction comes from our ability to understand the syntax and semantics of language.

The task of speech recognition is a long standing AI problem that involves transcribing a speech signal into written language, and is now largely solved by deep learning methods. One of the key challenges is that different words can sound the same, as alluded to above, that different speakers pronounce words differently, and that background noise and context all play an important role in us understanding speech. The inverse problem of speech synthesis—generating a speech signal from text—is much easier although still a challenge for natural sounding speech.

Another well-studied problem is NLP is machine translation, e.g., translating from English to French, or from Chinese to German. Here the problem is that different languages have different rules and constructs so we cannot simply build a one-to-one mapping of words. Methods initially invented for solving machine translation are now at the heart of large language models (LLMs) that enable tasks such as dialogue (i.e., chatbots), question answering, text generation and summarization. Some claim that these models are capable of commonsense reasoning. Indeed, large language models exhibit remarkable fluency when applied to these tasks, even if their accuracy is sometimes lacking.

1.1.3 Multimodal Problems (Vision and Language)

One of the more impressive feats that deep learning has been able to achieve is unified reasoning over multiple data modalities such as vision and language. This manifests itself in many different tasks. In image captioning the goal is to describe an image in words. This is refined in visual question answering (VQA) where we ask specific free-form questions (i.e., not necessarily seen during training) of an image. For example, given an appropriate image where such questions make sense, what color are her eyes? what is the mustache made of?

Visual and language navigation (VLN) is a relatively new area of research where we ask a robot (or simulated agent) to follow a natural language navigation instruction using visual cues, e.g., exit the dining room via the door next at the far end of the table, walk to the kitchen and stop in front of the fridge.

Another popular vision and language task is to locate a specific object in an image, such as to find the light blue truck. Here the algorithm is expected to return a bounding box or segmentation mask of the object being localized. The task can be combined with VLN to navigate to a particular location and then identify a target object in the scene. Continuing from the example above we might ask the robot to open the fridge and locate the bottle of milk.

As mentioned above, in addition to basic image editing tasks, we can combine vision and language to create completely novel (fake) images and videos using generative AI from text prompts such as an image of an astronaut riding a horse and a video of a woolly mammoth.

1.1.4 Robotics and Sequential Decision Making

Game play has been a long-standing test of AI systems. Research in this area includes state representation, search and planning under uncertainty, learning of policies, opponent modeling, and long-horizon credit assignment. Games can be deterministic with perfect information (e.g., chess or Go) or stochastic and partially observable (e.g., poker). Reinforcement learning is a classic machine learning approach to develop policies for sequential decision making that occurs in complex games. Deep learning variants [73, 101] have demonstrated amazing results in this area.

Planning tasks generalise beyond simple games and focus choosing a sequence of actions to achieve some goal while respecting constraints and optimizing objectives. Markov decision processes (MDPs) and partially-observable MDPs are classic models that solve the problem of planning under uncertainty and usually involve reinforcement learning to solve. Planning underpins many high-level AI applications, from logistics and scheduling to decision-making in robotics and autonomous systems.

In robotic movement and manipulation, perception (e.g., object detection and pose estimation), planning, and control are combined to navigate and interact with objects in a physical environment. Unlike games or abstract planning, manipulation must contend with noisy sensors, imperfect actuation, deformable objects, and complex physical interactions. Autonomous driving [98] is a special type of robotic task with the goal of developing vehicles that can drive themselves either in controlled situations or in open-ended environments with many other moving vehicles and pedestrians. Research into autonomous driving has brought many safety and driver assist technologies into regular vehicles (such as lane tracking and collision detection).

1.1.5 Scientific Discovery and Medicine

Science and medicine has seen an increasing application of deep learning methods over the past several years. One significant area of research is in understanding biological processes and novel drug discovery. The 2024 Nobel Prize in Chemistry awarded to David Baker, Demis Hassabis and John Jumper, was in acknowledgment of their work in computational protein design and protein structure prediction, enabling the creation of entirely new proteins. Hassabis is the founder of DeepMind where work initially focusing on game play (e.g., AlphaGo) has been applied to numerous scientific endevours including AlphaFold for predicting the structure of proteins.

Other areas of deep learning in science includes weather forecasting, medical image analysis, clinical decision support, and surrogate models for complex physical processes such as fluid dynamics. In the not-too-distant future we may seen brain-computer interfaces driven by deep learning models that interpret brain activity and link your thoughts directly to controlling your phone and other devices.