Synthesizing Visual and Action Routines Using Constraint Programming ⋆ Bonny Banerjee and B. Chandrasekaran Laboratory for AI Research, Department of Computer Science and Engineering The Ohio State University, Columbus, OH 43210, USA {Banerjee, Chandra}@cse.ohio-state.edu Abstract. A diagrammatic problem-solver requires a library of visual routines (VRs) and action routines (ARs) - the VRs are used to obtain information of speciﬁed types from the diagram and ARs to modify the diagram in speciﬁed ways. The VRs/ARs required are unbounded - a new domain may call for new perceptions and actions. We report on progress on our research in building an automated VR/AR synthesis system that would take as input the deﬁnition of a new routine in terms of existing routines in the library and well-deﬁned mathematical/logical constraints and synthesize the program for the desired routine using constraint programming. We illustrate the ideas by means of an example. 1 Introduction Ullman [1] suggested that in order for the human visual system to work as eﬃ- ciently as it does, there has to exist a ﬁnite set of elementary operations that can be combined to create an unbounded set of visual routines (VRs). Researchers have proposed diﬀerent elementary operations and developed systems that can compose task-speciﬁc VRs from them, using a language of attention or genetic programming. Reasoning and problem-solving with diagrams requires a large repertoire of VRs and action routines (ARs) - VRs help perceive information from a diagram while ARs act on a diagram to modify it, where the diagram might be external, as on a piece of paper or on a computer screen, or internal, as a representation in the mind or in an artiﬁcial agent. A diagrammatic reasoning (DR) system [2] consists of a central problem-solver that calls VRs and ARs to interact with and modify the diagram, guided by the problem-solving goals. Researchers have proposed diﬀerent VRs/ARs for diﬀerent tasks, such as left-of, inside, computing a path between two points avoiding obstacles, computing the point of intersection of two curves, and so on (see [1, 2] for more examples). Since DR requires essentially an unbounded set of VRs/ARs, it is not feasible to have a library of all VRs/ARs. Moreover, many of the VRs/ARs might be required only once, such as inside a region and left of a point, so there is no good reason to keep in store such VRs/ARs permanently. A feasible solution to this problem is ⋆ This research was supported by participation in the Advanced Decision Architectures Collaborative Technology Alliance sponsored by the U.S. Army Research Laboratory under Cooperative Agreement DAAD19-01-2-0009. D. Barker-Plummer et al. (Eds.): Diagrams 2006, LNAI 4045, pp. 196–198, 2006. c  Springer-Verlag Berlin Heidelberg 2006