An Automatic Jigsaw Puzzle Solver zy David A. Kosiba, Pierre M. Devaux, Sanjay Balasubramanian, Tar& L. Gandhi, and Rangachar Kasturi Department of Computer Science and Engineering, The Pennsylvania State University, University Park, Pennsylvania 16802 U.S.A. Abstract zyxwvut A computer vision system to automatically analyze and assemble an image of the pieces of a jigsaw puule is presented. The system, called Automatic Puule Solver (APS), derives a new set of features based on the shape and color characteristics of the puzzle pieces. A combination of the shape dependent features and color cues is used to match the puzzle pieces. Matching is pe~ormed using a modified iterative labeling procedure in order to reconstruct the original picture represented by the jigsaw puzzle. Algorithms for obtaining shape description and matching are explained with experimental results. 1: Introduction Jigsaw puzzles contain a number of problems endemic to machine vision, like shape description, partial boundary matching, pattern recognition, feature extraction, and heuristic matching. Most of the past research involving jigsaw puzzles ignores the color information contained on the individual pieces. Freeman and Gardner [l], Hirota and Ohto [2], Nagura, et al. [3], and Radack and Badler zyxwvut [4] use critical points extracted from piece boundaries as their basis for matching. Webster, et al. [5] use a method based on identifying isthmus points on the pieces and compute the distance function from the isthmus point to the edge of the puzzle pieces, while Wolfson et al. [6] employ curve matching techniques combined with an optimal search algorithm. While neglecting color may seem attractive at first, it is simply a waste of valuable information. Humans make use of all possible information to constrain their search for matching pieces, and when dealing with large search spaces, there is no reason for computers to attack the problem any differently. Therefore, the APS system uses a combination of shape features along with the color information contained on the surface of the individual jigsaw puzzle pieces as its matching criteria. 2: Problem definition The problem of solving a cardboard puzzle consists of reconstructing a known image from a set of interlocking pieces for which a unique solution exists. It is necessary not only to match the pieces, but also to rotate, translate, and assemble them into the original image that is usually found on the box cover of the puzzle. To this end, several assumptions have been made as part of an understood set of "puzzlerules": A jigsaw puzzle is defined as a set of pieces that, when properly assembled, fit together into one region. A puzzle piece is defined as a connected planar region. Two pieces that mate share a common border segment. The corners of two adjacent matching pieces approximately coincide when assembled. The contours can be segmented into individual sides separated by the corners. The individual sides are either flat, or contain a protrusion or a hole. (This can be generalized to conform to any piece shape.) There are no gaps between correctly matching pieces of the puzzle, and the solution to the problem is unique (i.e., no two pieces can fit perfectly with the same side of a third piece). o The matching process which is described in this paper makes use of several important features of puzzle pieces. The features which are extracted consist of color samples along the edges of the pieces, parameters which describe the curvature of the edges of the pieces, convexity or concavity of the pieces, and a measure of the goodness of fit between pieces. 3: Image acquisition and preprocessing The initial image is obtained using a vision system consisting of a Mustek 24-bit color flatbed scanner, set to a scanning resolution of 150 dpi. All the puzzle pieces are scanned at once. Processing of the image, shape analysis, matching and assembling are all done on a SUN Sparcstation-IPX. The image is segmented into disjoint pieces, and a connected component analysis is performed on the image. A simple boundary tracking algorithm [7] is applied for each piece in the labeled image. 4: Feature extraction zyx / shape representation Following the segmentation procedure, the zy APS system isolates the boundaries of the idividual puzzle pieces so that neighboring edges can be correctly matched. In order to do this, relevant features are extractedfrom the individualpuzzle pieces. Since this is a problem in partial boundary matching, the first goal is to separate the individual sides of each puzzle piece. In order to accomplish this task, the locations of the corners of each puzzle piece must be obtained. Once the corners are located, the piece contours are separated into their constituent sides. 616 1051-465U94 $04.00 zyxwvutsrqp 0 1994 IEEE