2. Sze FK, Wong E, Or KK, Lau J, Woo J. Does acupuncture improve motor recovery after stroke? A meta-analysis of randomized con- trolled trials. Stroke 2002;33:2604-19. 3. Park J, Hopwood V, White AR, Ernst E. Effectiveness of acupunc- ture for stroke: a systematic review. J Neurol 2001;248:558-63. 4. Schnyer RN, Allen JJ. Bridging the gap in complementary and alternative medicine research: manualization as a means of promot- ing standardization and flexibility of treatment in clinical trials of acupuncture. J Altern Complement Med 2002;8:623-34. 5. Lewith GT, White PJ, Pariente J. Investigating acupuncture using brain imaging techniques: The current state of play. Evid Based Complement Alternat Med 2005:2:315-9. 6. Lee JD, Chon JS, Jeong HK, et al. The cerebrovascular response to traditional acupuncture after stroke. Neuroradiology 2003;45: 780-4. 7. Lo YL, Cui SL, Fook-Chong S. The effect of acupuncture on motor cortex excitability and plasticity. Neurosci Lett 2005;384:145-9. 8. Jeun SS, Kim JS, Kim BS, et al. Acupuncture stimulation for motor cortex activities: a 3T fMRI study. Am J Chin Med 2005;33:573-8. doi:10.1016/j.apmr.2006.02.011 The Post-Stroke Rehabilitation Outcomes Project Revisited In December 2005, Archives published a supplement on the Post-Stroke Rehabilitation Outcomes Project (PSROP). 1 This project was a prospective observational cohort study that ex- amined rehabilitation outcomes among 1291 consecutively en- rolled stroke rehabilitation patients from 7 rehabilitation cen- ters. A strength of the study was its detailed and clinically relevant characterization of the stroke rehabilitation process and how it related to outcomes at discharge. One finding was that stroke patients who received earlier and more aggressive therapy did better on certain dimensions than those who re- ceived less aggressive therapy. We recommended that these findings be further validated through predictive validity studies and possibly controlled trials. Two commentaries 2,3 on the PSROP spoke to its strengths and limitations. As study investigators, we want to respond to the comments because they address fundamental issues of rehabilitation research design and epistemology. Jette 2 and Ottenbacher 3 organize their commentaries around 4 sets of issues: selection bias, observation bias, confounds, and inference or interpretation. We use these issues to frame our discussion, and add a fifth, the relevance of evidence hierarchies. Selection Bias “Selection bias,” says Ottenbacher, “occurs when there is a preferential inclusion of subjects with certain treatment outcomes.” 3(pS121) Researchers can guard against selection bias in several ways. We guarded against it by using data on all consecutively admitted patients. This minimized, if not elimi- nated, selection bias due to preferential inclusion. Moreover, the study was not plagued by selection bias due to refusals during informed consent process or to drop-outs because the nature of the study did not require an informed consent process. Jette and Ottenbacher are correct that, at the subgroup level, there may have been subtle patient differences unknown to us and not captured in the statistical analyses. The exhaustiveness of the patient descriptors used in the study, however, helped to minimize such differences. Even in randomized controlled trials (RCTs), there is the possibility of patient differences at the subgroup level if randomization is used for the entire study group but not at the subgroup level. Observation Bias Both Jette and Ottenbacher note that a “strength of the PSROP is the involvement of front-line clinicians in the de- velopment of the data collection instruments and actual data gathering and recording.” 3(pS121) Such involvement ensures clinical relevance, but the downside is that the clinicians’ awareness of the study’s objectives may affect how they con- sciously or unconsciously treat patients and document patient outcomes. In the PSROP, recording of interventions and documenting outcomes were done by different clinicians. Moreover, some outcomes are not as prone to observer bias as others. For example, recording discharge destination is not as prone to rater bias as recording a FIM score. It should be noted that there were no interventions pitted against another that would lead a therapist to favor 1 intervention over another. If thera- pists were prone to put their best foot forward, then the bias would be systematic across all interventions. Therapists did not know the outcomes or how the data would be analyzed. Our assessment is that after a brief self-conscious learning curve, data collection became routine and thus diminished the risk of observational bias. One way to diminish observational bias is to blind treating therapists to the intervention. Blinding the treating clinician is easy in a drug study using a placebo, but it is impossible with most hands-on rehabilitation interventions. This is why some “more-pure” research designs do not work well in rehabilita- tion research. Jette and Ottenbacher state that there is a lack of information about the accuracy or consistency of data in, and abstracted from, the medical chart. The supplement’s methods article, by Gassaway et al, 4 describes both the medical record abstraction process and the abstraction reliability process. Each medical records abstractor underwent a 4-day training session and a reliability testing process to ensure complete and accurate data collection. The study team checked for reliability at 4 points throughout the study and established a 95% agreement level between each chart abstractor and the study’s lead trainer and reliability checker. Also, inaccurate data collection is likely to be more noisy and hence would bias against finding significant predictors. Potential Confounders An abiding concern about observational studies is that the relation between an intervention and an outcome may be con- founded by other variables, one or more of which might not have been considered by the investigators. Researchers usually control for confounders through research design or statistical analyses, as in the case of the PSROP, that allowed researchers to control statistically for potential confounders. A strength of the PSROP, noted by the commentators, is that it captures a large number of potential confounders. Moreover, PSROP re- searchers relied heavily on the input of the study’s clinical practice team to identify potential confounders. One way the PSROP controlled for potential confounders was to use the Comprehensive Severity Index (CSI), an age- and disease-specific physiologic measure that encompasses more than 2000 signs, symptoms, and physical findings. The 595 LETTERS TO THE EDITOR Arch Phys Med Rehabil Vol 87, April 2006