CAAD Futures 2011 : Designing Together, ULg, 2011 © P. Leclercq, A. Heylighen and G. Martin (eds) 785 Combined Digital & Physical Modeling with Vision-Based Tangible User Interfaces : Opportunities and Challenges HSIAO Chih-Pin 1 and JOHNSON Brian 2 1 College of Architecture, Georgia Institute of Technology, USA chsiao9@gatech.edu 2 College of Built Environments, University of Washington, USA brj@uw.edu Abstract. Designers in architectural studios, both in education and practice, have worked to integrate digital and physical media ever since they began to utilize digital tools in the design process [1]. There are benefits of working in the digital domain as well as benefits of working physically; confronting, or seeming to confront, architects with a difficult choice. Emerging strategies for human- computer interaction such as tangible user interfaces and computer vision techniques present new possibilities for manipulating architectural designs. These technologies can help bridge between the digital and physical worlds. In this paper we discuss some of these technologies, analyze several current design challenges and present a prototype that illustrates ways in which a broader approach to human computer interaction might resolve the problems. The ultimate goal of breaking down the boundary between the digital and physical design platforms is to create a unified domain of "continuous thought" for all design activities. 1. Introduction Architectural design is a challenging cognitive activity, involving tactile, spatial, and auditory perception and "wicked problems" [2]. Buildings are almost invariably unique, inappropriate on another site, in another climate, or with other functionality and requirements. Architects must often address conflicting goals using limited information, and seeking a solution that harmonizes the parts in a process sometimes referred to as "puzzle making" [3]. The process is complex