David Wolf Graduate Research Assistant Jennifer Hyland NSF REU (Research Experiences for Undergraduates) Student Timothy W. Simpson 2 Professor ASME Fellow Department of Mechanical and Nuclear Engineering, The Pennsylvania State University, University Park, PA 16802 e-mail: tws8@psu.edu Xiaolong (Luke) Zhang Assistant Professor Department of Information Sciences and Technology, The Pennsylvania State University, University Park, PA 16802 The Importance of Training for Interactive Trade Space Exploration: A Study of Novice and Expert Users 1 Thanks to recent advances in computing power and speed, engineers can now generate a wealth of data on demand to support design decision-making. These advances have enabled new approaches to search multidimensional trade spaces through interactive data visualization and exploration. In this paper, we investigate the effectiveness and effi- ciency of interactive trade space exploration strategies by conducting human subject experiments with novice and expert users. A single objective, constrained design optimi- zation problem involving the sizing of an engine combustion chamber is used for this study. Effectiveness is measured by comparing the best feasible design obtained by each user, and efficiency is assessed based on the percentage of feasible designs generated by each user. Results indicate that novices who watch a 5-min training video before the experiment obtain results that are not significantly different from those obtained by expert users, and both groups are statistically better than the novices without the training video in terms of effectiveness and efficiency. Frequency and ordering of the visualization and exploration tools are also compared to understand the differences in each group’s search strategy. The implications of the results are discussed along with future work. [DOI: 10.1115/1.3615685] Keywords: multidimensional data visualization, design optimization, assessment, user training 1 Introduction Engineers routinely use computer-based simulation and analy- sis models to support design decision-making [1], particularly during the parametric and detailed stages of design when optimi- zation tools can be employed. Optimization tools provide one means to explore multidimensional trade spaces to find the design solution that maximizes (or minimizes) one (or more) objective while satisfying relevant constraints [2]. Recent advances in com- puting power and speed have enabled new interactive approaches to search trade spaces using multidimensional data visualization and exploration tools [3,4]. Such approaches allow designers to “steer” the optimization process while searching for the best (or Pareto optimal) design(s) [5,6], and recent studies have shown sig- nificant gains in the computational efficiency by putting designers back “in-the-loop” during the trade space exploration process [7]. To support interactive trade space exploration, researchers at Penn State University and the Applied Research Laboratory (ARL) have been developing the ARL trade space visualizer (ATSV) since the early 2000s [8,9]. ATSV has evolved into a platform for conducting research into human-computer interac- tions (HCIs) by allowing us to study how designers use multidi- mensional data visualization tools to display and navigate complex trade spaces to find design solutions [10]. What has become increasingly apparent in these studies is the importance of user training, not only in terms of using the software and its capa- bilities but also in terms of interpreting visual displays that involve different representations of multidimensional data. The issue of training is not particular to the capabilities in our software (see the comparison offered in Ref. [10]), yet it provides a unique opportunity for us to study it in the context of engineering design. In this paper, we investigate the effectiveness and efficiency of interactive trade space exploration strategies by conducting human subject experiments with novice and expert users solving a single objective, constrained design problem. Our distinction between novices and experts derives from their experience with the visualization and exploration tools available in our software and not with the problem domain. The capabilities of our software are summarized in Sec. 3 following a review of related literature in Sec. 2. The experimental setup, test problem, and user trials are described in Sec. 4, and the results and their implications are dis- cussed in Sec. 5. Section 6 provides closing remarks and avenues for future work. 2 Review of Related Work Since our software (i.e., ATSV) uses data visualization as the main form of user feedback from the system, it is important to understand the differences between novices and experts with respect to using data visualization tools. Seo and Shneiderman [11] find that interactive exploration of multidimensional datasets can be challenging because it is difficult to see patterns in more than three dimensions. Klein [12] states that expertise is based on a person’s ability to recognize and match patterns. The ability to perceive patterns and then to match patterns to actions in decision-making is 2 Corresponding author. 1 An earlier version of this paper appeared in the 2009 ASME Design Engineering Technical Conferences, ASME, San Diego, CA, Paper No. DETC2009=DAC-87294. Contributed by the Simulation & Visualization Committee of ASME for publica- tion in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manu- script received March 5, 2010; final manuscript received April 11, 2011; published online September 2, 2011. Assoc. Editor: Ian Grosse. Journal of Computing and Information Science in Engineering SEPTEMBER 2011, Vol. 11 / 031009-1 Copyright V C 2011 by ASME Downloaded 02 Nov 2011 to 146.186.238.53. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm