9. Contrastive Learning › 9.3 Applications

Section 9.3
Applications

9.3.1 Word2Vec (Skip-grams)

Word2Vec [71] was an early attempt at representing words \(w\) from some large vocabulary \(\cV\) in a high-dimensional embedding space \(\reals^n\). We saw this in Lecture 8 when discussing transformers. The key idea is for word \(w_i\) to be a good predictor of other words \(w_j\) that are nearby in a sentence. That is, for a given training corpus \(\cT\), we want to maximize

\begin{align} \sum_{w_i \in \cT} \sum_{w_j \in \cN_i} \log P(w_j \mid w_i) \tag{281}\end{align}

where \(P\) is modelled as the softmax over embedding similarities and \(\cN_i\) is the set of words nearby \(w_i\), i.e., it’s neighbourhood. Let \(z_i\) and \(z_j\) be the learned embeddings for words \(w_i\) and \(w_j\), respectively. Then

\begin{align} P(w_j \mid w_i) \propto \exp(z_i^T z_j) \tag{282}\end{align}

Note that this is exactly an InfoSNE-style contrastive loss if the normalizing term to make probability distribution sum to one is constructed from a subset of negatives (instead of the full vocabulary).

As discussed in Lecture 8, each word in the vocabulary \(w_i \in \cV\) is mapped to a learned latent representation \(z_i \in \reals^n\). This is equivalent to encoding \(w_i\) using a one-hot vector and passing through a single linear layer, \(z_i \gets A w_i + b\) where \(A \in \reals^{n \times |\cV|}\). The one-hot vector acts to extract the \(i\)-th row from \(A\).

In some cases embeddings \(z_i\) and \(z_j\) come from different encoders resembling MoCo from above.

9.3.2 BERT: Bidirectional Encoder Representations from Transformer

Unlike Word2Vec, BERT [22] encodes and entire sequence at time. The result is that the representation for individual token depends on the context, i.e., the same token can have a different embedding in different sentences. This is important if we want to disambiguate between homonyms—words with the same spelling that have different meaning.

BERT uses an encoder-only transformer with so-called bi-directional attention. In other words, it does not use a causal mask so that words earlier in the sentence can attend to words that appear later on. The model is trained using two losses: masked token prediction and next sentence prediction. The latter is a binary loss where the model must classify whether the second sentence follows the first (positive) or not (negative). The <cls> token from the BERT transformer can be used as a sentence-level summary. An architecture diagram for the BERT model is shown in Figure 125.

The BERT [22] model for learning language embeddings
Figure 125: The BERT [22] model for learning language embeddings.

Later language embedding methods such as SimCSE [28] augment BERT with a contrastive loss to bring semantically similar sentences together and push different ones apart.

9.3.3 CLIP: Contrastive Language-Image Pretraining

The idea of learning semantically useful representations for data naturally extends to multimodal learning where the goal is to encode objects of different forms, such as images and text, into the same latent space such that related objects have close embeddings. Contrastive language-image pretraining (CLIP) [83] is an early example of this idea, using a contrastive objective to align text and images.

Let \(z^{(I)}_i\) and \(z^{(T)}_i\) be a pair of related image and text embeddings obtained from a learnable vision encoder \(f^{(I)}\) and learnable language encoder \(f^{(T)}\), respectively. Then the goal of CLIP is to minimize the loss

\begin{align} \ell &= -\log \frac{\exp \text{sim}(z^{(I)}_i, z^{(T)}_i)}{\sum_{j} \exp \text{sim}(z^{(I)}_i, z^{(T)}_j)} \tag{283}\end{align}

summed over all image-text pairs in the dataset. In CLIP the dataset is constructed by collecting a large amount of images and their captions scraped from the Internet. An illustration of the CLIP framework is shown in Figure 126.

The CLIP framework was improved by BLIP [65], which introduces some architectural changes and includes a mechanism to filter out noisy captions. It performs better at generative tasks such as image captioning, scene understanding, and visual question answering (VQA) whereas CLIP performs better at image retrieval from a text prompt.

More recently SigLIP [106] was developed for efficient and robust training on very large datasets but with small batches. The main difference is to replace the softmax cross-entropy loss of CLIP with a binary sigmoid loss

\begin{align} \ell_{ij} &= \begin{cases} -\log \sigma\!\left(\text{sim}(z^{(I)}_i, z^{(T)}_j) + b\right) & i = j \\ -\log \sigma\!\left(-\text{sim}(z^{(I)}_i, z^{(I)}_j) - b\right) & i \neq j \end{cases} \tag{284}\end{align}

where \(\sigma\) is the logistic function. SigLIP is currently the representation learning model of choice for large vision-language models (LVLMs).

Illustration of the contrastive loss used in CLIP [83] for aligning text and images in a unified embedding space
Figure 126: Illustration of the contrastive loss used in CLIP [83] for aligning text and images in a unified embedding space.