7. Developing, Debugging and Diagnosing › 7.1 Numerical Issues

Section 7.1
Numerical Issues

Numerical calculations on a computer are always subject to errors and deep learning is no exception—deep learning algorithms are full of numerical calculations. One of the biggest sources of numerical error is due to limited precision representations and arithmetic. For example, what is the result of computing 255 + 1? Surprising to people not familiar with how computers work, the answer can vary depending on how the numbers are represented. If we use an 8-bit unsigned integer representation then the answer is zero! You can verify this for yourself by running the following Python code, which adds one to numbers zero, one, forty-one, and two hundred and fifty-five.

import numpy as np

x = np.array([0, 1, 41, 255], dtype='uint8')
x += 1
print(x)

Of course moving to floating-point representations does not remove the problem of numerical precision, it simply shifts it elsewhere as the following example shows—what is 16,777,216 + 1? The answer, if we’re using 32-bit IEEE floating-point format, is 16,777,216. We have not performed additional at all!

import numpy as np

x = np.array([16777216], dtype='float32')
x += 1
print(x)

You may think that such examples are pathological and never really occur in practice. You would be wrong. Consider the very common task in deep learning of applying a convolutional filter to an image. If the image is represented in 8-bit format (as is typical when loaded from storage), then the filtering calculations are susceptible to these types of numerical overflow and underflow.

In deep learning there is often a trade-off between memory, compute and accuracy. In many cases we can save memory (and compute) by representing numbers with less precision and not adversely affecting results as long as we have sufficient dynamic range to avoid overflow and are careful about underflow. Operating on IEEE 16-bit floating point 'fp16' reduces memory by a factor of two compared to 32-bit floating point 'fp32' but significantly reduces dynamic range. A new 16-bit format, called 'bf16', assigns more bit to the exponent and fewer to the mantissa thereby increasing dynamic range at the cost of precision. Modern hardware also supports mixed-precision arithmetic where some tensors are stored as 16-bit (e.g., weight matrices) and others in 32-bit (e.g., optimizer state).

Other numerical issues can arise from algorithmic limitations such as not being able to generate true random numbers, which to be honest is hardly an issue in practice for deep learning models.1 How an algorithm or calculation is implemented can also cause unexpected issues. Consider, for example, computing the empirical standard deviation of a set of scalar samples \(\{x_i \in \reals\}_{i=1}^{n}\), defined as,

\begin{align} \hat{\sigma} &= \sqrt{\frac{\sum_{i=1}^{n} \left(x_i - \mu\right)^2}{n - 1}} \tag{233}\end{align}

where \(\mu = \frac{1}{n} \sum_{i=1}^{n} x_i\) is the empirical mean. This calculation requires two passes through the data—once to compute the mean, and then again to compute the standard deviation using the mean.

A seemingly better approach is to perform equivalent calculation,

\begin{align} \hat{\sigma} &= \sqrt{\frac{n \sum_{i=1}^{n} x_i^2 - \left(\sum_{i=1}^{n} x_i\right)^2}{n(n - 1)}} \tag{234}\end{align}

which only requires one pass through the data. However, the one-pass approach is fraught with danger. If the \(x_i\) are all large or the sum of the \(x_i\) is large, then we risk overflowing the numerator, since the square of a large number is a much larger number. The two-pass approach mitigates against this type of numerical overflow by centering the data about the mean before squaring, and is hence more numerically stable.

A naive implementation of softmax for a vector \(z \in \reals^K\),

\begin{align} \textbf{softmax}(z) &= \left(\frac{\exp z_1}{Z}, \ldots, \frac{\exp z_K}{Z}\right) \tag{235}\end{align}

where \(Z = \sum_{k=1}^{K} \exp z_k\), has the same problem of being susceptible to numerical underflow and overflow. Fortunately a simple trick can make the calculation numerically stable. Let \(z_{\text{max}} = \max \{z_1, \ldots, z_K\}\). Then implementing softmax as

\begin{align} \textbf{softmax}(z) &= \left(\frac{\exp (z_1 - z_{\text{max}})}{Z}, \ldots, \frac{\exp (z_K - z_{\text{max}})}{Z}\right) \tag{236}\end{align}

where \(Z = \sum_{k=1}^{K} \exp (z_k - z_{\text{max}})\) gives the same result mathematically (i.e., under infinite precision) but in a much more robust way. It should be easy to convince yourself that the calculations are mathematically equivalent since the quantity \(\exp(-z_{\text{max}})\) cancels in the numerator and denominator, i.e.,

\begin{align} \frac{\exp (z_i - z_{\text{max}})}{Z} &= \frac{\exp (z_i) \exp(-z_{\text{max}})}{\sum_{k=1}^{K} \exp(z_k)\exp(-z_{\text{max}})} = \frac{\exp (z_i) \exp(-z_{\text{max}})}{\exp(-z_{\text{max}}) \sum_{k=1}^{K} \exp(z_k)} = \frac{\exp (z_i)}{\sum_{k=1}^{K} \exp(z_k)} \tag{237}\end{align}

Stability comes from the fact that terms \(\exp(z_i - z_{\text{max}})\) are always less than or equal to one, with equality holding for at least one or them (i.e., for all \(i\) such that \(z_i\) is a maximizer, \(z_i = z_{\text{max}}\)).

The above examples have demonstrated that mathematical equivalent calculations does not necessarily mean computational equivalence. Calculating the same thing in different ways can result in very different results due to numerical issues. But numerical issues can also simply be caused by bugs in the code, i.e., erroneous implementations. If you’re lucky this will result in divide-by-zero exceptions or not-a-number values, which are easy to detect. Unfortunately, however, bugs in machine learning algorithms can be very difficult to detect as we will show later in the lecture.

  1. 1. This is more of an issue in the field of cryptography.