Social validation: Examining its sensitivity and the factors that influence raters’ judgments Jennifer Cupit a, * , Elizabeth Rochon a,b , Carol Leonard a,c , Laura Laird a a Department of Speech-Language Pathology, University of Toronto, 160 – 500 University Avenue, Toronto, Ont., Canada M5G 1V7 b Toronto Rehabilitation Institute, Canada M5R 1Y8 c Audiology and Speech-Language Pathology Program, University of Ottawa, Canada K1N 6N5 Introduction The aphasiology literature is replete with studies aimed at improving the communication skills of individuals with aphasia. Objective language assessments are commonly used to measure changes in a person’s communi- cative competence after treatment. However, many contend that success on a language assessment does not necessarily translate into improvement in a person’s overall communicative ability (e.g., Lapointe, Katz, & Braden, 1999), which is presumably the goal of language therapy. One solution, hav- ing a family member complete rating assessments before and after treatment, is flawed, open as it is to a placebo effect. Social validation on the other hand, can provide an evaluation of a person’s overall ability to communicate, while avoiding reliance on someone invested in the treatment outcome (e.g., a spouse). This is a process in which people who are naive to the treatment pro- cess judge pre- and post-treatment language samples on specific parameters. The development of social validation measures is currently undergoing examination, as researchers investigate how best to obtain an accurate subjective assessment of a person’s communicative abilities. The main goal of the current research was to evaluate the sensitivity of a specific method of social validation, by having raters judge narratives produced by two groups of individuals with aphasia: one group that received treatment for a naming impairment and a group that did not. It was predicted that across Time 1 and Time 2, raters would detect a positive change in the nar- ratives of the treated group, but not in the untreated group. The current study also aimed to further the examination of social validation by inves- tigating the influence of specific factors, such as knowledge of aphasia (Hickey & Rondeau, 2005) and length and/or completeness of the lan- guage sample (Lustig & Tompkins, 2002) on raters’ judgments. Methods Participants Ten healthy older adults (mean age: 66.6 years) and eight speech-lan- guage pathologists (mean age: 36.5 years) participated in the study as rat- ers. All were native English speakers, with no history of brain injury. Materials As part of a larger treatment study, 11 participants with aphasia were asked to retell the story of Cinderella before and after treatment. Seven participants received treatment for anomia, and four received no treat- ment. These pre- and post-narratives were presented to the two groups of raters in two forms: full narrative, and 30-s segment. Narratives were presented in a fixed randomized order in two separate sessions, with the condition that the pre- and post-narratives for each person with aphasia were heard together. Raters were blind to the purpose of the study. Procedure Each rater listened to each narrative in both the full and 30-s segment formats, and rated it on four parameters (amount of information pro- vided, ability to transmit the message, word-finding ability, and ease of speaking), using a 7-point scale (1 = extremely poor; 7 = extremely well). Ratings from the two rater groups (older adults, and speech-language pathologists), were analyzed using a four-way ANOVA, crossing rater group (older adult vs. speech-language pathologist), patient group (treated vs. untreated), length of narrative (full vs. 30-s segment) and time point of narrative elicitation (pre-treatment/Time 1 vs. post-treatment/Time 2). Results The analysis of the overall score (all four questions combined) showed three main effects: patient group [F(1, 16) = 226.724; p < .00], with the untreated group being rated higher overall than the treated group; time point of sample elicitation [F(1, 16) = 62.784, p < .00], with the post-treat- ment (or Time 2) samples being rated more highly; and length of sample [F(1, 16) = 27.118; p < .00], with the full segments being rated more highly than the 30-s segments. An interaction of patient group · time point of narrative elicitation [F1(1, 16) = 39.928; p < .00] was observed, which indi- cated that the difference between pre-treatment/Time 1 and post-treat- ment/Time 2 ratings was significant for the treated but not the untreated group. There was also a significant rater group · patient group interaction [F1(1, 16) = 14.008; p < .00], which was due to differences that were not of interest to this study. * Corresponding author. Fax: +1 416 978 1596. doi:10.1016/j.bandl.2007.07.026 www.elsevier.com/locate/b&l Brain and Language 103 (2007) 8–249