2. Linear Classification and Multilayer Perceptrons › 2.1 Logistic Regression

Section 2.1
Logistic Regression

Roughly speaking, logistic regression turns an affine function of the input into a probability distribution over outputs. Let \(y \in \{0, 1\}\) be a random variable with distribution defined by

\begin{align} P(y = 1 \mid x) &= \frac{\exp(a^T x + b)}{1 + \exp(a^T x + b)} \tag{18}\end{align}

where \(a\) and \(b\) are parameters and \(x \in \reals^n\) is the observed feature vector. The maximum likelihood principle states that we should choose parameters \(a\) and \(b\) that maximizes the probability of the observed data under the model,

\begin{align} P(\cD) &= \prod_{i=1}^{N} P\left(y^{(i)} \mid x^{(i)}\right) \tag{19}\end{align}

In practice, we (equivalently) minimize the negative log-likelihood function instead,

\begin{align} \ell(a, b; \cD) &= -\log\!\left(\prod_{i : y^{(i)} = 1} \frac{\exp(a^T \! x^{(i)} + b)}{1 + \exp(a^T \! x^{(i)} + b)} \prod_{i : y^{(i)} = 0} \frac{1}{1 + \exp(a^T \! x^{(i)} + b)} \right) \tag{20}\\ &= -\!\sum_{i : y^{(i)} = 1} \! (a^T \! x^{(i)} + b) + \sum_{i=1}^{N} \log\!\left(1 + \exp(a^T \! x^{(i)} + b) \right) \tag{21}\end{align}

which is convex in variables \(a \in \reals^n\) and \(b \in \reals\).

The logistic function is an s-shaped curve (sigmoid) that maps from real numbers to the interval \([0,1]\). It is therefore useful for modeling binary probability distributions. An extension of the logistic function is the so-called softmax function, which we will see later. The definition of the logistic function for scalar variable \(z \in \reals\) is,

\begin{align} \sigma(z) &= \frac{\exp(z)}{1 + \exp(z)} \tag{22}\\ &= \frac{1}{1 + \exp(-z)} \label{eqn:logistic_fcn} \tag{23}\end{align}

Replacing \(z\) with \(a^Tx + b\) allows us to learn a probability distribution conditioned on vector-valued features \(x \in \reals^n\),

\begin{align} P(y = 1 \mid x; a, b) &= \sigma(a^T x + b) \tag{24}\end{align}

as we have done above. A property of the logistic function is that \(\sigma(z) \geq 0.5\) if, and only if, \(z \geq 0\). So a classification rule \(a^Tx + b \geq 0\) corresponds to \(P(y \mid x) \geq 0.5\), i.e., predict positive on an input \(x\) if the probability is above 50%.

An illustration of the logistic function for scalar \(z = a^Tx + b\) and two-dimensional features \((x_1, x_2)\) is shown in Figure 8. Observe that the function is bound between zero and one. Note also that for the two-dimensional case (and higher-dimensions) when we look along the \(a\) direction we get a one-dimensional logistic curve.

The logistic function, , for 1D and 2D features
Figure 8: The logistic function, \(y = (1 + \exp(-a^T x - b))^{-1}\), for 1D and 2D features.

Revisiting our example of binary classification over two-dimensional data, we can now see what a logistic function fitted to the data might look like as illustrated in Figure 9. Notice that in the separable case the logsitic curve is much sharper than in the non-separable case. In fact, in the separable case the function can be made arbitrarily sharp as we continue to train the model. In the non-separable case, the value of the logistic function gives us an estimate of the probability of an example being labeled positive.

Illustration of logistic functions fitted to two-dimensional separable and non-separable data. The logistic curves are drawn aligned to the direction of
Figure 9: Illustration of logistic functions fitted to two-dimensional separable and non-separable data. The logistic curves are drawn aligned to the direction of \(a\).

Having our classifier too confident around the decision boundary, even in the separable case, is undesirable. Remember we are fitting to a set of training data, and what we really want is for our model to generalise to unseen test data. One way to reduce the confidence of the classifier on positive and negative training examples is through regularization. Here we add a penalty on the magnitude of the model’s parameters, and minimize the resulting combination of loss function and regularizer,

\begin{align} \text{minimize}_{a,b} \quad \ell(a, b; {\cal D}) + \frac{\lambda}{2} \|a\|^2 \tag{25}\end{align}

Note that we have not put any regularization on the offset, or bias, parameter \(b\).

This type of regularization is also called weight decay because of it resulting in pushing the parameter values towards zero as evident when considering the gradient of the regularized loss,

\begin{align} \nabla_a \left( \ell(a, b; {\cal D}) + \frac{\lambda}{2} \|a\|^2 \right) &= \nabla_a \ell + \lambda a \tag{26}\end{align}

Ignoring the \(\nabla_a \ell\) term for the moment, then taking a step in the negative gradient direction will change parameter \(a\) to \((1 - \eta \lambda) a\), where \(\eta\) is the gradient step size (or learning rate), to be discussed later in the lecture. That is, regularization tends to push \(a\) to zero. From the definition of the logistic function in Equation \(\ref{eqn:logistic_fcn}\) it should be clear that as parameter \(a\) approaches zero, the value of the logistic \(\sigma(a^Tx + b)\) approaches a constant, i.e., \(\frac{1}{1 + e^{-b}}\). Hence, regularization tends to flatten the fitted logistic curve as show in Figure 10.

Regularization can be used to flatten a logistic function to reduce confidence around the decision boundary. We have rotated the feature space compared to Figur
Figure 10: Regularization can be used to flatten a logistic function to reduce confidence around the decision boundary. We have rotated the feature space compared to Figure 9 for easier comparison of the different logistic curves.

There are several other techniques for preventing over fitting that we’ll see throughout the course, e.g., data augmentation. One such technique is temperature scaling that can be applied post-hoc. Here we introduce a temperature parameter \(\tau > 0\) to the logistic function,

\begin{align} y &= \sigma\left(\frac{z}{\tau}\right) = \frac{1}{1 + \exp\left(-z / \tau\right)} \tag{27}\end{align}

The effect of \(\tau\) is shown in Figure 11. A high value of \(\tau\) has a similar affect to regularization, tending to flatten the curve. A value of \(\tau\) less than one will sharpen the curve.

Temperature scaling. The parameter controls the sharpness of the logistic curve
Figure 11: Temperature scaling. The parameter \(\tau > 0\) controls the sharpness of the logistic curve.

2.1.1 Logistic Regression as a Single Neuron

A very naive model of a biological neuron in the brain is a cell that fires its output if its accumulated input exceeds some threshold. We can very roughly view logistic regression as a single artificial neuron, as shown in Figure 12(left). The output \(y\) will be high if the weighted sum of the inputs \(z = a^Tx + b\) is greater than zero, and low otherwise. The arrows represent scalar signals. This is the classical depiction of a neuron in an artificial neural network, and is somewhat biologically inspired. The more modern view in deep learning is as a computation graph, illustrated more compactly in Figure 12(right). Here, the arrows can represent vector (or higher-order tensor) signals.

Classic view (left) and modern view (right) of a single neuron as either the explicit expression or more compact expression
Figure 12: Classic view (left) and modern view (right) of a single neuron as either the explicit expression \(y = \sigma\left(\sum_{i=1}^{n} a_i x_i + b\right)\) or more compact expression \(y = \sigma(a^Tx + b)\).

2.1.2 The XOR Problem and Multi-layer Perceptrons

As we saw earlier, a logistic regression classifier is only able to correctly classify linearly separable data. A famous example of a problem where the data cannot be separated by a single linear decision boundary is the XOR problem. See Figure 13(left). Here examples from the positive class (blue circles) are distributed in two clusters, one close to the point \((0, 1)\) and the other close to the point \((1,0)\). Likewise, examples from the negative class (red diamonds) are distributed in two clusters, one close to the point \((1,1)\) and the other close to the point \((0,0)\). No straight line separates the two positive clusters from the two negative clusters.

The positive and negative examples can, however, be separated if we allow an additional layer of processing, as illustrated in Figure 13(right). In the first layer we construct a (linear) decision boundary \(a_1^T x + b_1\) to separate the negative cluster close to \((1,1)\) from the remaining three clusters. This can be thought of as implementing a NAND logic gate, \(\xi_1 = \sigma(a_1^T x + b_1)\), where values close to 0 map to False and values close to 1 map to True. Still in the first layer, we construct another (linear) decision boundary \(a_2^T x + b_2\) to separate the negative cluster close to \((0,0)\) from the remaining three clusters. Similar to the first decision boundary, this can be thought of as implementing an OR logic gate, \(\xi_2 = \sigma(a_2^T x + b_2)\). If we plotted \(\xi_1\) versus \(\xi_2\) we would find that the positive examples cluster around \((\xi_1, \xi_2) = (1, 1)\), whereas the negative examples cluster around either \((\xi_1, \xi_2) = (0, 1)\) or \((\xi_1, \xi_2) = (1, 0)\). As such, we can construct a second layer (linear) decision boundary \(c^T z + d\), which acts as an AND logic gate on the first layer’s output \(z = (\xi_1, \xi_2)\) to solve the XOR problem.

The XOR problem (left) can be solved with a two-layer network (right)
Figure 13: The XOR problem (left) can be solved with a two-layer network (right).

In the preceding example the first layer of processing defines new features for the second layer. This motivates the multi-layer perceptron [88, 105]. More formally, a multi-layer perceptron is defined by composing multiple layers of logistic functions. For example, a two-layer perceptron is the composed function

\begin{align} y = \sigma(c^T \sigma(Ax + b) + d) \tag{28}\end{align}

with parameters \(A \in \reals^{m \times n}\), \(b \in \reals^m\), \(c \in \reals^m\), and \(d \in \reals\), and where the logistic function \(\sigma\) is applied elementwise on \(z_1 = Ax + b \in \reals^m\). Parameters of multi-layer perceptrons (and other linear layers in deep learning) are often called weights. Figure 14 shows a graphical depiction of the two-layer perceptron.

To see concretely how the two-dimensional XOR problem fits the multi-layer perceptron formulation, observe that we can stack the parameters from the two neurons from the first layer as

\begin{align} A &= \begin{bmatrix} a_1^T \\ a_2^T \end{bmatrix} \quad \text{and} \quad b = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} \tag{29}\end{align}

giving

\begin{align} \begin{bmatrix} \xi_1 \\ \xi_2 \end{bmatrix} &= \sigma \left(A x + b \right) = \sigma \left( \begin{bmatrix} a_1^T x + b_1 \\ a_2^T x + b_2 \end{bmatrix} \right) = \begin{bmatrix} \sigma(a_1^T x + b_1) \\ \sigma(a_2^T x + b_2) \end{bmatrix}. \tag{30}\end{align}
A two-layer percepton, implementing . Signal shape is indicated on the arrows
Figure 14: A two-layer percepton, implementing \(y = \sigma(c^T \sigma(Ax + b) + d)\). Signal shape is indicated on the arrows.

2.1.3 Activation Functions

Common activation functions
Figure 15: Common activation functions.

Thus far we have only considered the logistic function as operating on linear combinations of the input features. The effect of the logistic function is to produce an elementwise non-linear transformation of the features. Transformations other than the logistic function are also possible. These go under the collective name of activation functions. Other common activation functions include the hyperbolic tangent function,

\begin{align} \sigma(z) &= \tanh(z) \tag{31}\\ &= \frac{\exp(2z) - 1}{\exp(2z) + 1} \tag{32}\end{align}

and the rectified linear unit (ReLU),

\begin{align} \sigma(z) &= \max\{0, z\} \tag{33}\end{align}

These activation functions are plotted in Figure 15. For obvious reasons quantities \(z = Ax + b\) and \(y = \sigma(z)\) are often termed pre-activations and post-activations, respectively, on input \(x\). Many other activation functions have been proposed in the literature, e.g., leaky ReLU, gated linear units (GLU), GELU, SELU, radial basis, sinusoidal, etc.