A Study of Communication in the Cardiac Surgery Intensive Care Unit and its Implications for Automated Briefing Katleen McKeown, Ph.D.*, Desmond Jordan, M.D.t, Steven Feiner, Ph.D.*, James Shaw*, Elizabeth Chent, Shabina Ahmad, M.D.t, Andre Kushniruk, Ph.D.t, Vimla Patel, Ph.D.t tDepartments of Anesthesiology tCognitive Studies in Medicine *Department of Computer Science and Medical Informatics Centre for Medical Education Columbia University Columbia University McGill University New York, NY 10027, USA College of Physicians and Surgeons Montreal, CANADA New York, NY 10032, USA We present a study of the information transferred arrives. The report's quality varies with the experi- among caregivers in the context ofcardiac surgery and ence and preferences of the reporting physician and use the study to evaluate a system, MAGIC, that we the questions asked by the recipient. Personnel cur- are developing for automated generation of briefings. rently provide some critical information about the pa- Our framework integrates cognitive and quantitative tient by telephone during the operation, but this infor- evaluation methods and features three standards that mation is cursory and they are rushed. We believe that reflect current practice in the Cardiothoracic Inten- communicating an interim postoperative status report sive Care Unit (CTICU). Using experimental design to the CTICU personnel prior to the patient's arrival to compare human-generated and machine-generated could have a beneficial effect on patient care by giving briefings, we show that MAGIC's current level ofper- CTICU personnel more lead time to address otherwise formance is useful. Moreover, MAGIC could help im- unanticipated patient needs, improving the quality of prove information flow in the CTICU by providing a information transmission, and minimizing drug admin- consistent set of information earlier than in current istration errors. Our goal in developing MAGIC is to practice. The separate standards are also consistent produce a full interim report automatically, eliminating in suggesting specific modifications that may be nec- the need for the interim telephone call. essaryfor iterative design andfurther system develop- While MAGIC's language generation component ment. [2] is robust, the components for graphics generation [3], speech generation [4] and coordination among me- INTRODUCTION dia [5] are not yet ready for production use. Since When responsibilit for apatientishandedofffroMAGIC is designed to spread its information across When crespveronsiity for anpatientspecific ishndfofrom complementary media, we decided that evaluation of onstbe caregieriotea to anothe, scifi qin tion the language alone in a purely textual report would not mustcbe comu. nicate .t enure c tion quali of be particularly useful for informing our design. There- mransfedlre.and Ethelyvef w tat i atwhio should be fore, we developed the study described in this paper to transferred isand theplevelaf duestaio. wIchit so allow us to evaluate the content produced by MAGIC, be provided iS an important question. In this pa- to determine both whether information selected for the per, we address this problem in the context of cardiac briefing should be included and whether any informa- surgery. We present a study of information transferred tion was missed. among caregivers from the point at which a patient's Givn the di n s surgery has been completed in the OR (Operating Gientin dgfficultdes rs f subect cartoicpation and Room) through the patient's admission to the CTICU the busy CTICU environment, part of our focus is on (Cardiothoracic Intensive Care Unit). We are using this the developmen vaaon fr t can study~~~~~~~~~~~ to ino. futher developetof AI Ml h eeoment of an evaluation firamework that can stumd Atormct furtherdelopmofIntentiof MaGI (Mul provide us with multiple standards for comparison, re- tImeian AbstractmGental'syterhati of I ve Caren-data)op flecting best practice. Previous work on evaluation for [1], an experimental 'system that we have beenduevelop- naua la a e s stm wihnciialevrnet ing to produce briefings automatically of patient status [6]ual guag y after CA BG (coronary artery bypass grAft). [6] also suggests the use of multiple standards, among after CABG (coronary artery bypass gratt). other criteria. Most evaluations to date focus on sys- When a cardiac patient reaches the CTICU, a va- tems that extract information for later medical appli- riety of information about the patient's condition and cations from free text (e.g., [7, 8, 9]). In contrast, in status must be summarized for the CTICU medical MAGIC, our focus is on evaluating the provision of team, including qxisting medications, ventilation pa- essential information to subsequent caregivers. rameters, laboratory results, demographics, and past medical history. This summary is usually given orally METHODS by a physician, the anesthesia resident, to another physician and nurse in the CTICU. Because the par- Experimental Setup ticipating caregivers are extremely busy, the informa- We selected three standards that capture normal prac- tion is provided only once and only after the patient tice at the New York Presbyterian Hospital CTICU 1067-5027/00/$5.00 i) 2000 AMIA, Inc. 570