11. Generative AI: VAEs, Diffusion and Flows › 11.1 Background and Revision

Section 11.1
Background and Revision

Before embarking on deep learning models for learning distributions from data (or more precisely how to sample from such distributions), we first revise some important background material.

11.1.1 Gaussian Distributions

A one-dimensional Gaussian distribution characterized by mean and variance
Figure 132: A one-dimensional Gaussian distribution characterized by mean \(\mu\) and variance \(\sigma^2\).

Gaussian distributions are ubiquitous in statistics, science and engineering. You should easily recognize the classic bell shape of the one-dimensional Gaussian shown in Figure 132.

A multivariate Gaussian distribution with mean \(\mu \in \reals^n\) and covariance \(\Sigma \in \reals^{n \times n}\) has form

\begin{align} p(x; \mu, \Sigma) &= \frac{1}{(2 \pi)^{\frac{n}{2}} |\Sigma|^{\frac{1}{2}}} \exp \left(-\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu) \right) \tag{322}\end{align}

which we often write as \(\cN(x; \mu, \Sigma)\) or \(\cN(\mu, \Sigma)\) for short. Here \(|\Sigma|\) denotes the determinant of the covariance matrix and together with the other terms in the scaling factor in front of the expression (called the partition function), acts to normalize the distribution so that it integrates to one.

The Gaussian distribution has some useful properties:

A special case is the zero mean, unit covariance Gaussian, \(\cN(0, I)\). We can sample from an arbitrary Gaussian by transforming a sample from the unit Gaussian. That is, we can sample \(x \sim \cN(\mu, \Sigma)\) as

\begin{align} x &= \mu + \Sigma^{1/2} \epsilon \tag{323}\end{align}

where \(\epsilon \sim \cN(0, I)\). This is sometimes called the reparameterization trick, which we will use it often in this lecture.

11.1.2 Jensen’s Inequality

Jensen’s inequality is a result from convex analysis that states that for any convex function \(f(x)\) and any distribution \(p(x)\) we have

\begin{align} f\!\left( E_{x \sim p(x)} \left[ x \right] \right) \leq E_{x \sim p(x)} \left[ f(x) \right] \tag{324}\end{align}

For concave functions the inequality is reversed. Specifically \(\log x\) is a concave function of \(x\), so

\begin{align} \log E \left[ x \right] &\geq E \left[ \log x \right] \tag{325}\end{align}

A good way to remember the direction of the inequality is through the visualization in Figure 133. Here the function is concave and the line between any two points, representing the expected value of the function \(E \left[ \log x \right]\), lies below the function itself \(\log E \left[ x \right]\).

An illustration of Jensen’s inequality for . A concave function evaluated at a convex combination of points sits above the convex combination of the function ev
Figure 133: An illustration of Jensen’s inequality for \(\log x\). A concave function evaluated at a convex combination of points sits above the convex combination of the function evaluated at those points.

11.1.3 Variational Lower Bound

Graphical model for
Figure 134: Graphical model for \(P(X, Z) = P(Z) P(X \mid Z)\).

Let \(X\) and \(Z\) be two random variables modeled by some joint distribution \(p_{\theta}\). A graphical model [58] illustrating the factored distribution \(p_\theta(Z) p_\theta(X \mid Z)\) is shown Figure 134. Suppose we only get to observe \(X\). We want \(p_\theta(X)\) to be a good probabilistic model of our observed data, \(\cD\). We can try doing this by choosing \(\theta\) that maximizes the probability of the observed data as per the maximum likelihood principle,

\begin{align} \max \; \mathbb{E}_{x \sim \cD} \left[ \log p_\theta(x) \right] \tag{326}\end{align}

Unfortunately this is difficult to compute in general as it requires integrating out \(Z\),

\begin{align} p_\theta(x) &= \int p_\theta(z)\, p_\theta(x \mid z) \,\text{d}z \tag{327}\end{align}

Later we will see how to get around this difficulty by introducing a lower bound on \(\log p_\theta(x)\).