Chapter 9
Contrastive Learning

Data is processed by deep learning algorithms as vectors (or more generally tensors). One important decision in designing algorithms is how to represent the data. This can have a significant impact on the performance of the models. We already saw this when considering how to represent semantic labels such as “dog” or “cat” for multi-class classification. Here typical representations are integers or one-hot vector encodings. The topic of representation learning is concerned with automatically learning data embeddings, or latent representations, which are useful for general inference tasks. The desire is that the embeddings somehow capture relevant semantic information about the data. Embeddings are often learned from massive amounts of unlabelled data, which is cheap to collect, using a self-supervised objective.

Learning data embeddings or representations has a long history that pre-dates the current deep learning era. For example, in classical machine learning the task of metric learning [104] is about finding meaningful distance measures between \(n\)-dimensional vectors, which are not captured well by their Euclidean distance. The simplest approach is to learn a warping of the space and then applying the Euclidean distance on the transformed features. If the warping is affine then the resulting metric is a Mahalanobis distance, \(d_M(x, y) = \sqrt{(x - y)^T M (x - y)}\) for some positive semi-definite matrix \(M\) where \(x\) and \(y\) are vectors in the original (unwarped) space.¹ Setting \(M\) to the identity matrix recovers the Euclidean distance in the original space.

Another early excursion into representation learning involved restricted Boltzmann machines (RBMs), which aim to capture probability distributions over complex data, in this case via an energy function over a bi-partite graph between a latent representation and observed features (units). Stacked RBMs or deep belief networks are a multi-layer variant that can be successfully trained in a layerwise fashion using a contrastive divergence technique [44] that approximates a gradient to bring the estimated model distribution inline with the true data distribution. Although they share the adjective contrastive, the techniques discussed in this lecture are distinct from that of contrastive divergence, which relates more to generative models that we discuss in Lecture 11. The RBM and its variants have been superseded by more recent techniques, but remains an important model to study for its historical value and theoretical insights.