i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 761–772 j ourna l homepage: www.ijmijournal.com Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support Alice C. Li a , Joseph L. Kannry a , Andre Kushniruk b , Dillon Chrimes b , Thomas G. McGinn c , Daniel Edonyabo a , Devin M. Mann d,* a Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA b School of Health Information Science, University of Victoria, British Columbia, Canada c Department of Medicine, Hofstra North Shore-LIJ Medical School, Manhasset, NY, USA d Department of Medicine, Section of Preventive Medicine and Epidemiology, Boston University School of Medicine, Boston, MA, USA a r t i c l e i n f o Article history: Received 30 October 2011 Received in revised form 12 January 2012 Accepted 21 February 2012 Keywords: Usability Clinical decision support Clinical prediction rules Evidence-based medicine Electronic health records Clinical simulations a b s t r a c t Purpose: Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think- aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. “Near-live” clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow. Methods: This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a com- mercial electronic health record. Phase I applied usability testing involving “think-aloud” protocol analysis of 8 primary care providers encountering several scripted clinical scenar- ios. Phase II used “near-live” clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all ses- sions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods. Results: In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most pos- itive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12 min with 71% of the clinical pre- diction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its ele- ments in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS. Conclusions: This study successfully combined “think-aloud” protocol analysis with “near- live” clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase ∗ Corresponding author at: Harrison Ave, Boston, MA, USA. Tel.: +1 617 638 8021. E-mail address: dmann@bu.edu (D.M. Mann). 1386-5056/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2012.02.009