Short, Simple, and Specific: The Influence of Item Design Characteristics in Multi-Source Assessment Contexts Ste ´phane Brutus John Molson School of Business Concordia University and Jeffrey Facteau Organizational Consultant Georgia The present study investigates the impact of item characteristics on multi-source performance assessment. Three item characteristics (syntax, double-barreledness, & behavioral specificity) were linked to the psychometric properties of items used by self, subordinates, peers, and supervisors as operationalized by the relationship between the item and the performance dimension it is intended to measure. Results show that syntax, a linguistic index that pertains to the length of items, is related to the psychometric properties of all rating sources except subordinates. The implications of this effect for the design of multi-source assessment instrument are discussed. Introduction E very year, a countless number of individuals are presented with similar statements in order to guide their evaluation of a coworker. Without question, the use of performance items in combination with a rating scale represents the most common methodology to assess individual performance in organizations (Murphy & Cleveland, 1995). Accordingly research on the topic is abundant (e.g., Austin & Villanova, 1992). Typically, performance evaluation surveys are composed of items pertaining to on-the-job behaviors (e.g., Austin & Villa- nova, 1992; Kavanaugh, 1971; Murphy & Cleveland, 1995). These items represent written stimuli, connoting relevant behaviors for the organization, that cue raters and enable them to judge the frequency or the quality of those particular behaviors for a ratee. In the past decade, multi-source assessment (MSA) has brought significant changes to the manner by which performance items are used (Brutus & Derayeh, 2002). The traditional performance appraisal processes calls for supervisors to evaluate the performance of their subordi- nates. In MSA, those same items are also used to rate one’s own performance, the performance of a supervisor, a peer, and even that of someone from outside the organization (London & Smither, 1995). Of interest in this paper is the impact of linguistic properties of performance items on their psychometric quality when completed by different types of raters. From an organizational standpoint, there exist many reasons behind the adoption of MSA such as breaking down organizational barriers and increasing communica- tions between employees (Waldman & Atwater, 1998). However, most of the appeal of MSA stems from the expected gains in reliability and validity that ensue from using multiple raters (Borman, 1997; Murphy, Cleveland, & Mohler, 2001). To date, the presence of these gains is still debated as research on the reliability and validity of MSA Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree ‘‘[D O N A L D ] A C T S D E C I S I V E L Y I N T I M E S O F C R I S I S ’’ Funding for this paper was provided by the Social Sciences and Humanities Research Council of Canada (grant #410-2000-0818) whose support is gratefully acknowledged. The authors thank John Fleenor, Cindy McCauley, Manuel London and Jean Leslie for their support of this research project. INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT VOLUME 11 NUMBER 4 DECEMBER 2003 313 r Blackwell Publishing Ltd 2003, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.