The Psychologist-Manager Journal 2001, Vol. 5, No. I, 59-72 Copyright 2001 by the Society of Psychologists in Management ISSN 1088-7156 Contrasting a 360-Degree Feedback Measure With Behaviorally-Based Assessment Tools: An Application of Generalizability Theory1 Calvin C. Hoffman2 California School of Organizational Studies Alliant International University Deborah Olson Hay-McBer Stacy L. Haase Southern California Gas Company This article describes an application of generalizability theory (GT) to data from three different groups of supervisors/managers evalu- ated for developmental purposes using a brief, internally developed 360-degree feedback measure. Major goals of this research were to examine the generalizability of 360-degree feedback ratings, and to compare the 360-feedback measure to operational selection tools previously used in this organization (structured behavioral inter- view and writing assessment). GT results are presented separately for three groups of assessees (first-line supervisors, second-line supervisors, and middle managers). In all cases, the 360-degree measure demonstrated disappointingly low levels of reliability (range of 0.27 to 0.51 depending on assumptions made), while the interview and writing assessment provided high levels of reliability (0.87 to 0.91 under the same assumptions). The results presented here, in combination with other generalizability analyses of multi- rater feedback, suggest that practitioners should carefully consid- er the appropriateness of applications of 360-degree feedback measures, and demonstrate that reliability of such measures is insufficient for applications such as selection or compensation. M ulti-rater or 360-degree feedback systems have enjoyed a tremen- dous growth in U.S. organizations during the past decade. Performance evaluation and feedback data are increasingly being gathered from what could be considered “nontraditional” sources (London & Smither, 1995). Edwards & Ewen (1996) estimated that up to 90% of Fortune 1000 firms use some form of multi-source feedback, and approximately 73% of the members of the ASTD Benchmarking Forum reported using peer review or 360-degree feedback (Bassie & Cheney, 1997). Graddick & Lane (1998) noted that multi-rater feedback is a now com - 59 This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.