Chapter 8
Transformers
This lecture discusses the transformer model, which is the current state-of-the-art model for deep natural language understanding and generative AI. We begin by revisiting the idea of representing words as vectors. We then introduce the dot-product attention mechanism at the heart of the transformer. After covering language models, we show how images can also be represented as sets of vectors and processed by the transformer. The lecture ends with a review of applications that use transformers in applications beyond language and some practical considerations in speeding up the attention mechanism.
Recurrent neural networks (RNNs) have been successfully applied to many sequence classification and generation tasks. However, they suffer from a couple of major drawbacks, specifically the vanishing gradient problem and the fact that all information, including long-range correlations, needs to be captured succinctly by the latent state vector. Transformers [103] overcome these drawbacks by processing all elements of a sequence concurrently—although this introduces some computational challenges of its own, namely, quadratic compute and memory. But despite their high computational cost, transformer models are at the cutting-edge of artificial intelligence, powering technologies like large language models (LLMs) and generative AI.