Change in Dynamic Items and Risk Estimates over Time: Findings from a Pilot Implementation of the START-AV Brian G. Sellers 1 , Sarah L. Desmarais 1 , Joel Dvoskin 2 , Jodi Viljoen 3 , Keith Cruise 4 , & Tonia L. Nicholls 5,6 1 University of South Florida; 2 University of Arizona ; 3 Simon Fraser University; 3 Fordham University ; 5 BC Mental Health & Addition Services; 6 University of British Columbia The Present Study The purpose of this presentation is to explore whether START-AV assessments are indeed dynamic in nature and sensitive to change over time. We compared initial START-AV assessments to follow-up assessments conducted approximately three months later at three juvenile justice residential facilities in a Southern state. Change in Item Ratings Strength ratings for seven items demonstrated significant changes in the expected direction except for Medication Adherence : School/Work, External Triggers, Peer Social Support, Medication Adherence, Rule Adherence, and Insight (see Table 1). Many, but not all, of the same items evidenced significant changes in the expected direction in the Vulnerability ratings: School/Work, Substance Use, Rule Adherence, and Plan (see Table 1). Changes in Strength ratings on Substance Use and Vulnerability ratings on Peer Social Support approached significance. Methods Measure The START-AV is a SPJ instrument that assesses short- term risks in adolescents. The instrument is comprised of 23 dynamic START-AV items for both Strength and Vulnerability that assessors score as 0 (minimally present) to 2 (maximally present). Strength and Vulnerability ratings are intended to be scored independently so that a youth may score high (or low) on both Strength and Vulnerability for any specific item. Given item ratings, identification of key and critical items, and historical factors, assessors estimate risk as low, moderate, or high on eight domains (violence, self- harm, suicide, unauthorized absence, substance abuse, self-neglect, victimization, general offending). Training All case managers (N = 28) completed a full-day training workshop, which educated them on the principles and theory of violence risk assessment utilizing the SPJ approach. START-AV constructs were operationalized and a practice case was completed. Refresher training was delivered at 3-6 month intervals. Assessments START-AV assessments are completed by case managers every three months following the initial interview and file review conducted within 21 days of admission. In this presentation, we report data from the initial START-AV assessments completed at admission (Time 1) and the first follow-up assessment (Time 2). Time 2 assessments were completed for 59 youths (88% male; 12% female). The average interval time between initial assessment and follow-up was 104 days (Range=22–364 days). Age of the youth ranged from 14 to 19 years (M = 16.13, SD = 1.18). Strength and Vulnerability total scores were calculated for research purposes by summing the item ratings (possible range=0-46). When there were missing item ratings, we followed the instructions for prorating the Strength and Vulnerability total scores provided in the original (adult) START manual, version 1.1 (see p. 34, Webster et al., 2009). We removed any START assessment with five or more missing Strength or Vulnerability ratings (final n = 53 youths). Results Change in Total Scores Repeated measures ANOVAs yielded significant within- individual differences in total scores. Specifically, mean Strength total scores increased from 18.72 (SD=7.56) to 20.13 (SD=8.34) (F[1,54]=3.42, p=0.07, η p 2 =0.06) and mean Vulnerability total scores decreased from 20.19 (SD=7.37) to 18.31 (SD=8.21) (F[1,52]=5.78, p=0.02, η p 2 =0.10) . Introduction Historical approaches to risk assessment emphasized static risk factors and overall risk status (Douglas & Kropp, 2002; Douglas & Skeem, 2005; Van Der Put et al., 2010). Refocusing the aims of risk assessment from prediction to management (Heilbrun, 1997) encourages assessors to consider dynamic risk factors that can change over time in response to interventions (Douglas & Kropp, 2002; Douglas & Skeem, 2005). Several risk assessment instruments now exist that contain dynamic risk (and protective) factors that may enable assessors to better tailor and modify case management plans to reduce risk of adverse outcomes (Douglas & Kropp, 2002; Van Der Put et al., 2010). Adolescence is an important transitional developmental period during which risk and protective factors may vacillate over time within individuals (Randolph, 2004; Van Der Put et al., 2010). The Short-Term Assessment of Risk and Treatability-Adolescent Version (START-AV; Nicholls, Viljoen, Cruise, Webster, & Desmarais, 2010) is a structured professional judgment (SPJ) guide that includes greater emphasis on dynamic risk and protective factors compared to pre-existing measures. The START-AV guides a multi-faceted, idiographic short-term assessment of diverse risks through consideration of 23 factors that are dynamic in nature. To date, there have been few empirical investigations of dynamic factors (Douglas & Skeem, 2005), especially in regard to youth (see Lodewijks et al, 2010; Randolph, 2004). Hanson and Harris (2000) proposed a distinction between stable and acute dynamic factors. Stable dynamic factors (e.g., impulsivity) that may change gradually over time, and acute dynamic factors (e.g., substance use) that may change daily or hourly (Hanson & Harris, 2000). However, there is scant information regarding how often dynamic factors must be re-assessed in order to measure change (Douglas & Skeem, 2005; Webster et al., 2006). Thus, there remains a knowledge gap regarding the most appropriate means of integrating dynamic factors into clinical practice Discussion Both the overall Strength and Vulnerability total scores exhibited significant changes, which corresponded with changes at the individual item level, providing evidence of the dynamic nature of the START-AV items. Most importantly, all changes were in the anticipated direction, providing evidence of improved functioning over time in custody. Analyses failed to identify significant changes in the risk estimates in the 3-month interval. Though youth may evidence change on dynamic factors within a few months, it may take longer to see changes in associated levels of risk to self and others. Future research should examine whether changes in dynamic factors affect changes in overall risk at longer follow-up intervals. Item Ratings Time 1 Time 2 M SD M SD p 1. Social Skills S 1.08 0.60 1.08 0.60 1.00 V 0.71 0.56 0.71 0.53 1.00 2. Relationships S 0.86 0.44 0.81 0.48 0.32 V 0.88 0.47 0.82 0.57 0.41 2a. Adult Relationships S 0.79 0.56 0.86 0.52 0.25 V 0.75 0.65 0.83 0.47 0.32 2b. Peer Relationships S 0.81 0.48 0.72 0.49 0.13 V 0.98 0.55 1.04 0.60 0.41 3. School/Work S 0.71 0.68 0.90 0.67 0.01 V 1.11 0.65 0.91 0.64 0.02 4. Recreational S 0.93 0.57 1.00 0.57 0.35 V 0.75 0.64 0.66 0.61 0.13 5. Self-Care S 1.19 0.55 1.16 0.59 0.64 V 0.49 0.50 0.42 0.50 0.10 6. Mental State S 0.95 0.63 1.05 0.57 0.16 V 0.54 0.57 0.44 0.54 0.14 7. Emotional State S 0.91 0.57 1.02 0.51 0.14 V 0.75 0.55 0.64 0.62 0.11 8. Substance Use S 0.44 0.60 0.56 0.63 0.07 V 1.41 0.69 1.09 0.76 0.001 9. Impulse Control S 0.65 0.59 0.69 0.51 0.57 V 1.16 0.58 1.12 0.69 0.66 10. External Triggers S 0.55 0.54 0.73 0.45 0.02 V 1.24 0.62 1.12 0.62 0.20 11. Social Support S 0.80 0.56 0.84 0.46 0.62 V 1.02 0.61 0.92 0.52 0.17 11a. Adult Social Support S 0.87 0.54 0.87 0.47 1.00 V 0.87 0.67 0.85 0.63 0.80 11b. Peer Social Support S 0.56 0.54 0.69 0.51 0.03 V 1.21 0.57 1.08 0.65 0.07 12. Material Resources S 0.84 0.54 0.87 0.55 0.53 V 0.68 0.58 0.68 0.58 1.00 13. Attitudes S 0.91 0.54 0.98 0.55 0.25 V 0.77 0.64 0.74 0.63 0.69 14. Medication Adherence S 1.07 0.53 0.83 0.54 0.05 V 0.52 0.51 0.55 0.51 0.57 15. Rule Adherence S 0.72 0.52 0.90 0.48 0.02 V 1.05 0.62 0.84 0.57 0.02 16. Conduct S 0.81 0.60 0.90 0.61 0.17 V 0.96 0.54 0.82 0.58 0.16 17. Insight S 0.67 0.54 0.81 0.48 0.03 V 0.96 0.54 0.86 0.48 0.14 18. Plans S 0.75 0.60 0.86 0.51 0.11 V 0.93 0.54 0.75 0.55 0.02 19. Coping S 0.86 0.48 0.88 0.54 0.78 V 0.93 0.46 0.84 0.50 0.17 20. Treatability S 1.14 0.57 1.12 0.67 0.80 V 0.75 0.55 0.65 0.62 0.28 21. Parent/Home Environment S 0.80 0.56 0.85 0.56 0.50 V 0.96 0.62 0.85 0.57 0.11 Notes. S = Strength; V = Vulnerability. p values for within-subjects repeated measures ANOVAs comparing item ratings between T1 and T2. Strength Total Score Vulnerability Total Score Increase (%) Decrease (%) Increase (%) 23.1 43.6 Decrease (%) 10.3 23.1 Notes. % = valid percent of youths whose scores changed in the described directions. χ 2 (1, N = 39) = 8.90, p < 0.05. Table 1. Item Ratings at Time 1 and Time 2 Direction of Change in Item Ratings We additionally examined the relationship between changes in Strength total scores and Vulnerability total scores. For both total scores, the majority changed in the expected direction. Specifically, Strength total scores increased for 54.5% of youths, decreased for 27.3%, and did not change for 18.3%. Vulnerability total scores decreased for 60.4% of youths, increased for 32.1%, and did not change for 7.5%. As Shown in Table 2, Almost half showed improvements in both strengths and vulnerabilities. Table 2. Relationship between Change in Total Scores Change in Risk Estimates Our final set of analyses examined changes in risk estimates, if any, between Time 1 and Time 2. As may be seen in Table 3, we found no significant differences. Table 3. Risk Estimates at Time 1 and Time 2 Specific Risk Estimates Time 1 Time 2 L (%) M (%) H (%) L (%) M (%) H (%) p Violence 71.4 25.0 3.6 70.0 26.7 3.3 .492 Self-Harm 94.4 5.6 0.0 92.3 3.8 3.8 -- Suicide 100 0.0 0.0 96.0 0.0 4.0 .336 Unauthorized Leave 84.1 9.1 6.8 81.8 9.1 9.1 .711 Substance Abuse 32.7 34.6 32.7 28.6 51.8 19.6 1.00 Self-Neglect 100 0.0 0.0 100 0.0 0.0 -- Victimization 93.0 4.7 2.3 91.3 6.5 2.2 .160 General Offending 61.0 34.1 4.9 59.1 38.6 2.3 .786 Notes. S = Strength; V = Vulnerability. % = valid percent. p values for within-subjects repeated measures ANOVAs comparing M T1 and T2 risk estimates. –Statistic could not be computed due to missing data. Funding: This study was funded by the National Institute on Drug Abuse (P30DA028807). The content is solely the responsibility of the authors and does not necessarily represent the official views of the agency.