5. Object Detection and Image Segmentation › 5.7 Instance Segmentation

Section 5.7
Instance Segmentation

We now return to the problem highlighted earlier that semantic segmentation does not distinguish between different instances of the same object type. See, for example, Figure 77. Various solutions of similar flavour have been proposed, but the main idea is to jointly perform object detection and semantic segmentation. Gould et al. [35] proposed an early version of this predating deep learning object detection and semantic segmentation models. A more modern approach builds on the R-CNN architecture with a model called Mask R-CNN [42], which for each bounding box proposal adds a binary pixel segmentation branch to mask out pixels not belonging to the object. This is in addition to the class prediction and bounding box refinement multi-layer perceptrons. The complete architecture is shown in Figure 78.

Some researchers like to distinguish between instance segmentation—as the task of labeling unique instances—from panoptic segmentation [56], which labels both instances and class types. The main difference from Mask R-CNN is that the background regions (such as sky, trees, grass, road, buildings, and water) are also labeled.

One problem with semantic segmentation is that it does not distinguish between instances of the same class (left). A proposed solution is instance segmentation—
Figure 77: One problem with semantic segmentation is that it does not distinguish between instances of the same class (left). A proposed solution is instance segmentation—perform object detection and semantic segmentation jointly (right).
Mask R-CNN [42] architecture
Figure 78: Mask R-CNN [42] architecture.