3. Back-propagation and Learning › 3.1 Notation

Section 3.1
Notation

Before proceeding it is worth clarifying notation for gradients. For scalar-valued functions \(f: \reals \to \reals\) we denote by

\begin{align} \fracdd[f]{x} \tag{81}\end{align}

the total derivative of function \(f\) with respect to argument \(x\). If a function takes more than one argument we can differentiate with respect to each argument separately by taking partial derivatives denoted by, for example,

\begin{align} \fracpp[f(x, y)]{x}. \tag{82}\end{align}

What is often confusing is when \(y\) is also a function of \(x\). Then calculus dictates that the total derivative with respect to \(x\) is the sum of direct and indirect terms,

\begin{align} \fracdd[f(x, y)]{x} &= \fracpp[f(x, y)]{x} + \fracpp[f(x, y)]{y} \fracdd[y]{x}. \label{eqn:total_derivative} \tag{83}\end{align}

We already saw in the previous lecture that for multi-variate functions \(f: \reals^n \to \reals\) we denote the gradient by

\begin{align} \nabla f(x) &= \left( \frac{\textrm{d} f}{\textrm{d} x_1}, \ldots, \frac{\textrm{d} f}{\textrm{d} x_n} \right) \tag{84}\end{align}

which is an \(n\)-dimensional (column) vector. More generally, for multi-variate vector-valued functions, \(f: \reals^n \to \reals^m\) we define

\begin{align} \fracdd{x} f(x) &= \begin{bmatrix} \frac{\textrm{d} f_1}{\textrm{d} x_1} & \ldots & \frac{\textrm{d} f_1}{\textrm{d} x_n} \\ \vdots & \ddots & \vdots \\ \frac{\textrm{d} f_m}{\textrm{d} x_1} & \ldots & \frac{\textrm{d} f_m}{\textrm{d} x_n} \end{bmatrix} = \begin{bmatrix} \frac{\textrm{d} f_i}{\textrm{d} x_j} \end{bmatrix}_{ij} \tag{85}\end{align}

as the \(m\)-by-\(n\) matrix of total derivatives, sometimes called the Jacobian matrix. Note that this is the transpose of \(\nabla f\) for scalar-valued functions. For functions with signature \(f: \reals^{n} \times \reals^{\tilde{n}} \to \reals^m\) we can also define the matrix of partial derivatives,

\begin{align} \fracpp{x} f(x, y) &= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \ldots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \ldots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}. \tag{86}\end{align}

This convention makes the chain rule particularly easy to express by, for example, replacing the scalar products in Equation \(\ref{eqn:total_derivative}\) with matrix multiplications and following the same ordering of expressions. We also have that the affine function \(y = Ax + b\) has derivative \(\fracdd[y]{x} = A \in \reals^{m \times n}\) without having to introduce transposes, which is nice.1

We demonstrate the use of matrix multiplication for simplifying the chain rule for gradient calculations using a function \(f: \reals^2 \to \reals\) of two variables \(x_1\) and \(x_2\), each a function of parameter \(\theta\). We wish to calculate \(\ifracdd[f]{\theta}\), which we can do via the chain rule of differentiation as

\begin{align} \fracdd[f(x_1(\theta), x_2(\theta))]{\theta} &= \fracpp[f]{x_1} \fracdd[x_1]{\theta} + \fracpp[f]{x_2} \fracdd[x_2]{\theta} \tag{87}\\ &= \begin{bmatrix} \fracpp[f]{x_1} & \fracpp[f]{x_2} \end{bmatrix} \begin{bmatrix} \fracdd[x_1]{\theta} \\ \fracdd[x_2]{\theta} \end{bmatrix} \tag{88}\\ &= \fracdd[f]{x} \fracdd[x]{\theta} \tag{89}\end{align}

where the derivatives in the last line are understood to be Jacobian matrices.

Symbols \(\dd\) and \(\dd[X]\) are also used to denote the derivative operators, which is cleaner that writing \(\fracdd{x}\) and \(\fracpp{x}\), especially with inline text, but is less familiar to computer science students so won’t be used here. Some authors use \(D_1\) for partial derivatives with respect to the first argument, \(D_2\) for the second, etc.

Last, and perhaps most importantly, many authors (including ourselves) are sometimes sloppy with notation and the reader should carefully check the intent from the context.

  1. 1. One drawback of this convention is that the gradients propagated through deep learning networks are with respect to a scalar-valued loss function \(L\) and, in all frameworks, stored transposed so that they have the same dimensionality/shape as the variable with respect to which the derivative is taken.