Section 5.8
Segment Anything

Recently Kirillov et al. [57] introduced a model for various image segmentation tasks called Segment Anything. The model can run off-the-shelf or be provided with different types of prompts, including foreground/background hints and text queries (see Figure 79). It was trained on a very large set of images and based on the ViT architecture, which will be discussed in later lectures. Training took approximately three full days on a cluster of 256 A100 GPUs. The fact that it can address a host of segmentation tasks and amount of training data put in the class of foundation models—i.e., a model that is trained at scale and can be adapted to a wide range of tasks.

The training set used by Segment Anything includes 1B pseudo-ground-truth masks and 11M images. This set was bootstrapped from a smaller set of 4.3M human annotated masks and 120k images followed by a further automatically generated 5.9M masks on 180k images that were carefully filtered to eliminate poor pseudo-labels.