6. Sequence Models › 6.7 Joint Vision and Language Embedding

Section 6.7
Joint Vision and Language Embedding

The importance of models that can reason over both visual and language modalities has prompted researchers to investigate whether it is possible to embed images and text in the same latent space such that the visual and language representations of the same concept appear close in the embedding space. This joint embedding ability is the key to enabling prompt-based image generation technologies. It also facilitates so-called zero-shot tasks such as prediction where the model can label images with categories for which it has not been explicitly trained, image captioning, and prompt-based image retrieval.

A common approach introduced by Radford et al. [83] known as CLIP and later improved by BLIP [65], uses a metric learning method called contrastive learning. Here the method is given a training set of (image, text)-pairs, \(\cD = \{(I_i, T_i)\}_{i=1}^{N}\). These could be collected, for example, by scraping the Internet for captioned photographs. We assume that images are mapped into the latent space using an image encoder network \(f^{\text{img}}\) and text is mapped into the latent space using a text encoder network \(f^{\text{txt}}\) with parameters \(\theta^{\text{img}}\) and \(\theta^{\text{txt}}\), respectively. Then given the training set we can define a contrastive loss between any image \(I_i\) and any text \(T_j\),

\begin{align} \ell_{i,j}(\theta^{\text{img}}, \theta^{\text{txt}}) &= -\log \left(\frac{\exp \phi(I_i, T_j)}{\sum_{k \neq i} \exp \phi(I_i, T_k)}\right) \tag{231}\end{align}

where \(\phi\) is a temperature scaled similarity measure. Let \(z^{\text{img}}_i = f^{\text{img}}(I_i; \theta^{\text{img}})\) and \(z^{\text{txt}}_j = f^{\text{txt}}(T_j; \theta^{\text{txt}})\) be the embeddings for image \(I_i\) and text \(T_j\), respectively. Then a typical choice for \(\phi\) is cosine similarity,

\begin{align} \phi(I_i, T_j) &= \frac{1}{\tau} \left(\frac{z^{\text{img}}_i}{\|z^{\text{img}}_i\|_2}\right)^{\!T} \! \left(\frac{z^{\text{txt}}_j}{\|z^{\text{txt}}_j\|_2}\right) \tag{232}\end{align}

Here \(\tau > 0\) is the temperature hyper-parameter.

Minimizing this loss on the dataset (i.e., summing over \(\ell_{i,i}\) for \(i = 1, \ldots, N\)) brings embeddings for image \(I_i\) and text \(T_j\) close to each other for \(i=j\) and far apart for \(i \neq j\). The method is illustrated in Figure 94 where we want high similarity for pairs along the diagonal of the image-text matrix and low similarity for off-diagonal pairs.

Contrastive language-image pre-training (CLIP) for jointly embedding images and text [83]
Figure 94: Contrastive language-image pre-training (CLIP) for jointly embedding images and text [83].

One may naturally ask about the validity of the loss function. Couldn’t it be the case that some pairs \((I_i, T_j)\) should be close in feature space even if \(i \neq j\), for example two different images of a dog, \(I_i\) and \(I_j\)? Yes, off-diagonal terms in the comparison matrix depicted in Figure 94 may indeed refer to the same concept and so should not be driven apart. It turns out, that the relative occurrence of such examples is overshadowed by the number of times that the concepts are different coupled with the fact that the loss is only applied to batches of data, and so the method still works as a very good approximation to what we really want (without having to exhaustively label all possible \(N^2\) combinations).

Contrastive learning will be covered in more depth in Lecture 9.