Chapter 2
Linear Classification and Multilayer Perceptrons
In this lecture we introduce the notion of logistic regression in the context of binary classification. We discuss the limitations of logistic regression for non-linearly separated data. This motivates the multi-layer perceptron (MLP), which can be shown (under mild assumptions) to be a universal function approximator. Building on the basic framework of MLPs, we discuss multi-class classification and gradient descent for fitting model parameters.
Binary classification involves data instances that can be divided into two distinct categories. For example, photographs of faces and photographs not containing faces. Typically we are given a set of training examples \({\cal D} = \left\{\left(x^{(i)}, y^{(i)}\right)\right\}_{i=1}^{N}\) composed of \(n\)-dimensional input features \(x^{(i)} \in \reals^n\) representing each data instance, and target labels \(y^{(i)} \in \{0, 1\}\). Instances with label 1 are called positive (e.g., faces), while those with label 0 are called negative (e.g., not faces).
In this setting we can think of two different tasks. First, the regression task is to learn a density function \(P(y \mid x)\) from the training examples, which estimates the probability of 0 or 1 on a new input after getting to observe its features, \(x\). Second is the classifier task, which learns a classifier function \(f: \reals^n \to \reals\) from the training examples that predicts 1 on an input \(x\) if \(f(x) \geq 0\) and 0 otherwise. Often the classifier function is a linear, or more correctly, affine, function of the features, \(f(x) = a^T x + b\). These tasks are related as we will later see.
Figure 7 depicts two examples of binary classification in a two-dimensional features space. In the first example (left), a linear function can be found to correctly separate the positive examples from the negative examples. In the second example (right), no such linear function exists, and the data is said to be inseparable. Note that the affine function \(a^T x + b\) measures (signed) distance from the hyperplane that divides the space.