Section 7.4
Dataset Issues
Data is the fuel for deep learning algorithms—the cleaner the data, the more efficient the engine and the better the results will be; the more data we have, the bigger the engine you can drive (model you can train) and the better the results will be. Unfortunately, all datasets are biased and almost all datasets contain errors. Figure 101 shows examples of bias from two very popular, albeit aging, datasets. The left panel shows near duplicate images from a semantic segmentation dataset called MSRC [17]. The problem with such examples is that if one image from each pair appears in the training set and the other appears in the test set, then for a model to do well it simply needs to memorize the images, and we get a false sense of the ability for the model to generalize, which is the whole point of the test set.
The right panel in Figure 101 shows the average image from each class of the Caltech-101 dataset [63]. This dataset was a (small) precursor to the ImageNet dataset [21] for image classification. Here the average image for a class is obtained by simply taking the mean pixel values at each location for all images of that class. Clearly some classes, like face, airplane, stop sign, ying yang, etc., exhibit very little variation and would therefore be very easy to classify. This is despite the fact that if we looked at a broader collection of images (from outside the dataset) we would see significantly more variability, pointing to an unintentional bias in how the dataset was collected.
Datasets can also be labeled incorrectly. Figure 102 shows examples from several popular datasets where the given label in the dataset for a particular example is wrong. Such erroneous labels can be very difficult to find and correct without laboriously examining every image (and label) in the dataset. One may think it possible to train and an image classification model and then examine only the images that it predicts incorrectly to determined whether those images are really incorrect or whether they have been labeled wrong in the dataset. However, this only corrects false-positive labels not false-negatives (image where both the model and ground-truth are incorrect). Inevitably we have to develop algorithms that can handle noisy labels and accept the fact that our performance estimates will be slightly tainted by incorrect ground-truth.
