CIRCULAR DEPENDENCY: TREATMENT EFFECTIVENESS EVALUATION, INSTRUMENT VALIDATION AND PERFORMANCE APPRAISAL Chong Ho Yu, Samuel A. DiGangi, Angel Jannasch, & Barbara Ohlund Arizona State University Chong Ho Yu, Instruction and Research Support 0101, Arizona State University, Tempe AZ 85287 Keywords: Treatment effectiveness, instrument validation, performance appraisal, power analysis Introduction Psychological and educational research often is focused on treatment outcomes. Appropriate interpretation of treatment outcomes is based primarily on: (a) studying the efficacy of treatment, (b) the validity and reliability of instrumentation, and (c) obtaining subjects who represent a normal cross-section of the desired population to ensure accurate performance appraisal. However, in many quasi-experimental and non- experimental settings where experimental control is lacking, often each aspect mentioned above is studied simultaneously. This convolutes the focus of research as well as comprises the interpretation of results. For example, in research on Web-based instruction, test results are often used as an indicator of both the quality of instruction and the knowledge level of students. In addition, pilot studies are implemented to examine the usefulness of the treatment program and to validate the instrument based upon the same pilot results. To address this misapplication of methods, this article discusses the differentiation of the objectives of treatment effectiveness evaluation, instrument validation, and performance appraisal. In addition, it points out the pitfalls of circular dependency, proposes the rectification of the problems inherent in circular dependency, and presents recommendations for the application of appropriate research methods. Further, a survey result was reported to substantiate these widespread misconceptions. The data were collected via the Internet by announcing the online survey to six different ListServ groups and Newsgroups. The survey consists of five multiple- choice questions, which are related to the functions of treatment effectiveness evaluation, instrument validation, and performance appraisal (see Appendix). Thirty-four graduate students, who had taken on average 4.9 undergraduate and graduate statistics courses, responded to the survey. The respondents span across twenty-five institutions in six continents/regions (North America, South America, Europe, Asia, Africa, New Zealand), twenty-two different undergraduate majors (e.g. chemistry, English, electrical engineering, history, psychology) and eighteen different graduate majors (e.g. biology, communication, computer science, education, finance, industrial engineering, statistics). Within the United States, the respondents came from eight different states. On the average, the participants answered 1.6 questions correctly. Consequences of the confusion Researchers may be confused by the function of specific aspects in studies of treatment outcomes, instrument reliability and validity, as well as subjects’ ability. Table 1 summarizes the differences of the three functions. Confusing these differences among a study's treatment, instruments and subjects may lead to grave consequences. Table 1. Differences of treatment effectiveness evaluation, instrument validation, and performance appraisal. Target Type Objective Source of variation Sample size determination Treatment Treatment effectiveness evaluation To evaluate the effectiveness of a treatment such as an instructional module Treatment effect and experimental errors Power, beta, effect size, alpha Instrument Instrument validation To evaluate the reliability and validity of an instrument such as a test or a survey Instrument items Stability Subjects Performance/ Attitude appraisal To evaluate the performance or/and the attitude of respondents Subjects’ abilities and attributes None