AUTOMATICALLY IDENTIFYING USABILITY PROBLEMS IN WEB SITES THAT SUPPORT INTERNET-BASED SELLING Jungpil Hahn Assistant Professor of Management Krannert School of Management, Purdue University West Lafayette, IN 47907 Email: jphahn@mgmt.purdue.edu Robert J. Kauffman Director, MIS Research Center, and Professor and Chair, Information and Decision Sciences Carlson School of Management, University of Minnesota Minneapolis, MN 55455 Email: rkauffman@csom.umn.edu Last revised: November 27, 2005 ______________________________________________________________________________ ABSTRACT Managers at e-commerce firms are in need of proven methods for evaluating the usability of their Web sites. So, one of the most pressing issues is whether the design of their online storefronts is effective, and if not, which areas require attention and improvements. However, current usability evaluation methods (e.g., user testing, inspection and inquiry) are not well suited to the task at hand. This paper proposes a new Web site evaluation approach, which is grounded in the economic theory of production. We conceptualize human-computer interaction during online shopping as an economic production process in which customers are making use of various functionalities provided by the e-commerce Web site in order to complete a purchase transaction. This view enables us to formulate a novel perspective on Web site usability — the ability to transform inputs (i.e., use of Web site functionalities) into outputs (i.e., completed purchase transactions). We use data envelopment analysis (DEA) as the methodological vehicle for evaluating e-commerce Web sites, and propose two new metrics, Inefficiency Breadth and Unit Inefficiency, to help automatically identify potentially problematic Web site functionalities. The value of the proposed method is illustrated by applying it to the evaluation of a real-world e- commerce Web site. Index Terms: 1. Software/Software Engineering/Usability Testing (D.2.5.t) 2. Software/Software Engineering/Metrics/Performance Measures (D.2.8.b) 3. Information Systems/Information Interfaces and Presentation/User Interfaces/Evaluation Methodology (H.5.2.e) 4. Computing Milieux/Computers and Society/Electronic Commerce (K.4.4) Keywords – Data envelopment analysis, e-commerce Web site evaluation, Internet-based selling, performance measurement, software metrics, usability assessment ______________________________________________________________________________