Clinical Speech to Text Evaluation Setting Hanna Suominen 1 , Jim Basilakis 2 , Maree Johnson 3 , Linda Dawson 4 , Leif Hanlen 1 , Barbara Kelly 5 , Anthony Yeo 2 , Paula Sanchez 3 1 NICTA, National ICT Australia and The Australian National University, Locked Bag 8001, 2601 Canberra, ACT, Australia 2 University of Western Sydney, Locked Bag 1797, 2751 Penrith, NSW, Australia 3 Centre for Applied Nursing Research (a joint facility of the South Western Sydney Local Health District & the University of Western Sydney), Locked Bag 7103, 1871 Liverpool BC, NSW, Australia 4 University of Wollongong, 2522 Wollongong, NSW, Australia 5 The University of Melbourne, 2010 Melbourne, VIC, Australia hanna.suominen@nicta.com.au, J.Basilakis@uws.edu.au, Maree.Johnson@sswahs.nsw.gov.au, lindad@uow.edu.au, leif.hanlen@nicta.com.au, b.kelly@unimelb.edu.au, anthony.yeo@uni.sydney.edu.au, Paula.Sanchez@sswahs.nsw.gov.au Abstract. Failures in information flow from clinical handover are the leading cause of sentinel events in the USA and associated with nearly half of all ad- verse events and over a tenth of preventable adverse events in Australia. Verbal clinical handover provides a good picture of the background clinical history and current state of clinical management of a group of patients cared for by a nurs- ing team. However, all this valuable verbal information is lost after three con- secutive shifts if no notes are taken during handover. When traditional note- taking by hand occurs, less than a third of data is transferred correctly after five shifts. We propose using an automated approach of cascading speech-to-text con- version, standardisation with respect to controlled thesauri, and structuring in accordance with documentation standards. This transcribes verbal handover in- formation into written drafts for subsequent clinical review, editing, and addi- tion to electronic health records. In this paper, we introduce the evaluation setting for this technology devel- opment in a laboratory environment. It ranks a wide range of recording devices used alone or in combination with headsets and lapel microphones based on cli- nicians’ preferences and their accuracy in speech-to-text conversion. The sam- ple consists of four student nurses and four experienced academics from diverse clinical specialties and speaking styles. To simulate realistic nursing clinical handovers, twenty handover scenarios have been scripted. The subsequent eval- uation in a clinical environment will address speech-to-text conversion, stand- ardisation, and structuring with the short-listed devices in six hospitals with the sample of thirty authentic handover situations per hospital.