Asymmetric Cuts: Joint Image Labeling and Partitioning (preprint) Thorben Kroeger 1 , J¨org H. Kappes 2 , Thorsten Beier 1 , Ullrich Koethe 1 and Fred A. Hamprecht 1,2 1 Multidimensional Image Processing Group, Heidelberg University 2 Heidelberg Collaboratory for Image Processing, Heidelberg University Abstract For image segmentation, recent advances in optimization make it pos- sible to combine noisy region appearance terms with pairwise terms which can not only discourage, but also encourage label transitions, depending on boundary evidence. These models have the potential to overcome prob- lems such as the shrinking bias. However, with the ability to encourage label transitions comes a different problem: strong boundary evidence can overrule weak region appearance terms to create new regions out of nowhere. While some label classes exhibit strong internal boundaries, such as the background class which is the pool of objects. Other label classes, meanwhile, should be modeled as a single region, even if some internal boundaries are visible. We therefore propose in this work to treat label classes asymmetrically: for some classes, we allow a further partitioning into their constituent ob- jects as supported by boundary evidence; for other classes, further parti- tioning is forbidden. In our experiments, we show where such a model can be useful for both 2D and 3D segmentation. 1 Introduction Image segmentation methods typically rely on two complementary sources of in- formation: object appearance and boundary evidence. For example, in semantic labeling tasks [14] a set of object classes of interest is given. Each image can contain one or more of these instances, but might also contain many objects of unknown classes (“background”). One approach for semantic segmentation is to make use of (noisy) local object class probabilities – as obtained from learned appearance models – which can be regularized using local boundary cues. On the other hand, pure partitioning problems, as in the Berkeley Segmenta- tion Dataset [30], do not specify any object classes but rely on boundary evidence alone [5, 3, 37]. In this work, we propose a combined semantic labeling and partitioning, called Asymmetric Segmentation, which can naturally deal with object classes which are known to have strong internal boundaries and jointly optimizes the re- gion labeling, the boundaries between classes and the boundaries within classes. Furthermore we present a novel algorithm called Asymmetric Multi-Way Cut (AMWC) for solving those problems. 1