Section 9.1
Self-supervised Objectives

Learning representations of data is sometimes based on a self-supervised learning objective. This can be thought of as an unsupervised machine learning setting which is transformed into a supervised setting by constructing labels from the data itself. The simplest (and earliest) form of representation learning is is auto-encoding where the goal is to find a low-dimensional encoding of the data, \(z = E(x)\), that admits decoding back to the original signal, \(D(z) \approx x\). Other objectives can be derived from various pretext tasks such as predicting image rotations, colourization of greyscale images, or completion from masked input (language, image or video). Note that in all these cases the dataset of input-output pairs \(\cD = \{(x_i, y_i)\}\) can be derived from the data itself.