CONSENSUS ANALYSIS: RELIABILITY, VALIDITY, AND INFORMANT ACCURACY IN USE OF AMERICAN AND MANDARIN CHINESE PAIN DESCRIPTORS 1,2 Rod Moore, D.D.S., Ph.D., Inger Brcdsgaard, M.D., and Marc L. Miller, Ph.D., University of Washington Tai-Kum Mao, M.D., National Defense Medical Center, Taipei, Taiwan Samuel F. Dworldn, D.D.S., Ph.D. University of Washington ABSTRACT A quantitative method for validating qualitative interview results and checking sample parameters is described and illus- trated using common pain descriptions among a sample of Anglo-American and mandarin Chinese patients and dentists matched by age and gender. Assumptions were that subjects were members of a sociocultural group (e.g. ethnic or professional~lay) and answered questions independently about a monotonic domain (e.g. pain). Subjects answered 18 true~false items derived from and selected to reflect pain perceptions consistent with published and unpublished interview data. Estimates of consistency in use of descriptors within groups, validity of description, accuracy of individuals compared with others in their group, and minimum required sample size were calculated using Cronbach's o~,factor analysis, and Bayesian probability. Ethnic and professional differ- ences within and across groups were also tested using multidimen- sional scaling (MDS) and hypothesis testing. Consensus (consis- tency of subject response by group) was .99 among Anglo- Americans and .97 among Chinese. Mean subject accuracy was .81 for Americans and .57 for Chinese, indicating the need for larger numbers of Chinese to supplement each others' statements. However, larger numbers of subjects were recruited than actually required for each ethnic group at .95 confidence limits. MDS showed similarities in use of descriptors within ethnic groups, while there were differences (p < .001) between Chinese and American groups. Use of covalidating questionnaires that reflect results of qualitative interviews are recommended in order to estimate sample parameters such as intersubject agreement, individual subject accuracy, and minimum required sample sizes. (AnnBehavMed 1997, 19(3):295-300) Preparation of this manuscript was supported in part by Grants 5 R29 DE09945-03 NIH/National Institutes of Dental Research, Bethesda, Maryland, USA and Regional Center for Dental Research Clinic, Univer- sity of Washington NIDR P50-DE-08229-08. 2 The authors thank Shuang-chiu Wang and Devon Brewer for technical assistance and Susan Weller at the University of Texas for comments on the manuscript. Reprint Address: R. Moore, Department of Oral Medicine, University of Washington, Box 356370, Seattle, WA 98195-6370. 91997 by The Society of Behavioral Medicine. INTRODUCTION Subjective verbal reports from qualitative health research interviews are sometimes met with skepticism as to their scientific rigor or clinical generalizability. However, recent examples of qualitative research methods in behavioral medicine have demon- strated their usefulness in research on diabetes (1), human immuno- deficiency virus (HIV) (2,3), and cardiovascular diseases (4), as well as treatment processes such as medical (5) or dental (6) decision-making. Such studies have required the sensitivity of qualitative methods to obtain meaningful results unconstrained by limitations of a priori selection of questionnaire items. However, qualitative methods are often painstaking and time consuming, thus the need to confine efforts to small numbers of informants. Researchers also usually want to be able to generalize to similar populations, so questions may arise such as: "How can we be certain that the answers or observations from these few subjects are reliable or valid?" and "How many interview subjects are enough?" The overall aim of this study is to present a model for qualitative interview validation with concomitant estimation of sample characteristics that can be used to address these methodo- logic issues in qualitative health research. The model is described and illustrated using an example of how pain descriptions for common pains vary with the sociocultural context of ethnic samples. Only a few scientific studies have dealt with how pains attain meaning by nature of the social or cultural context from within which one is experiencing them (7-16). Beecher (7), an army surgeon, observed that pain in wounded World War II soldiers was for many a relieving reminder that they were alive and were going home. A comparable group of civilians with lesser wounds were found to complain more and requested more pain medication (8). Zborowski (9) observed hospital patients with similar postopera- tive pains and found differences between Irish, Anglo-American, Italian-American, and Jewish patients in pain expressivity and coping style. These pioneer studies and other later studies on ethnicity (10-14), laboratory versus clinic contexts (15), and patient versus doctor/dentist contexts (16,17) illustrate that the meaning of pain within social and cultural contexts weighs heavily in the description and measurement of pain. Review of these studies, however, leaves uncertain the generalizability of the results, even though they appear to be high in face validity. Cultural Consensus Theory and Reliability The qualitative research tradition (interviews, observations, focus groups), which is committed to validity, has in recent years 295