ORIGINAL ARTICLE
Objective Structured Assessment of Debriefing
Bringing Science to the Art of Debriefing in Surgery
Sonal Arora, PhD,
∗
Maria Ahmed, MPH,
∗
John Paige, MD,† Debra Nestel, PhD,‡ Jane Runnacles, MBBS,
∗
Louise Hull, MSc,
∗
Ara Darzi, MD, FACS,
∗
and Nick Sevdalis, PhD
∗
Objective: To identify the features of effective debriefing and to use this to
develop and validate a tool for assessing such debriefings.
Introduction: Simulation-based training has become an accepted means of
surgical skill acquisition. A key component of this is debriefing—yet there is
a paucity of research to guide best practice.
Methods: Phase 1—Identification of best practice and tool development.
A search of the Medline, Embase, PsycINFO, and ERIC databases identi-
fied current evidence on debriefing. End-user input was obtained through 33
semistructured interviews conducted with surgeons (n = 18) and other oper-
ating room personnel (n = 15) from 3 continents (UK, USA, Australia) using
standardized qualitative methodology. An expert panel (n = 7) combined the
data to create the Objective Structured Assessment of Debriefing (OSAD) tool.
Phase 2—Psychometric testing. OSAD was tested for feasibility, reliability,
and validity by 2 independent assessors who rated 20 debriefings following
high-fidelity simulations.
Results: Phase 1: 28 reports on debriefing were retrieved from the literature.
Key components of an effective debriefing identified from these reports and
the 33 interviews included: approach to debriefing, learning environment,
learner engagement, reaction, reflection, analysis, diagnosis of strengths and
areas for improvement, and application to clinical practice. Phase 2: OSAD
was feasible, reliable [inter-rater ICC (intraclass correlation coefficient) =
0.88, test–retest ICC = 0.90], and face and content valid (content validity
index = 0.94).
Conclusions: OSAD provides an evidence-based, end-user informed ap-
proach to debriefing in surgery. By quantifying the quality of a debriefing,
OSAD has the potential to identify areas for improving practice and to opti-
mize learning during simulation-based training.
Keywords: debriefing, education, feedback, qualitative, simulation, surgery,
training
(Ann Surg 2012;256: 982–988)
S
imulation-based training has become an integral component of
modern surgical curricula.
1
Drivers include the need to master
new technologies and techniques, curtailed opportunities for experi-
ential learning, and rising concerns for patient safety.
2
With mounting
evidence that skills acquired transfer to the operating room,
3,4
sim-
ulation is now a significant feature of both the American College of
Surgeons-Association of Program Directors in Surgery Curriculum in
From the
∗
Department of Surgery and Cancer, Imperial College, London, UK;
†Louisiana State University Health Sciences Center, USA; and ‡Gippsland
Medical School, School of Rural Health, Faculty of Medicine, Monash Uni-
versity, Australia.
Disclosure: This study was funded by the London Deanery STeLI initiative and by
the Association of Surgical Education CESERT grant. S.A., M.A., L.H., and
N.S. are also affiliated with the Imperial Centre for Patient Safety and Service
Quality, which is funded by the National Institute for Health Research (NIHR).
Reprints: Sonal Arora, PhD, Department of Surgery and Cancer, 10th Floor, QEQM
Building, St Marys Hospital, Praed Street, London W2 1NY, UK. E-mail:
Sonal.Arora06@imperial.ac.uk.
Copyright C 2012 by Lippincott Williams & Wilkins
ISSN: 0003-4932/12/25606-0982
DOI: 10.1097/SLA.0b013e3182610c91
the United States and the Intercollegiate Surgical Curriculum Project
in the United Kingdom.
5
Although undoubtedly useful as an adjunct to clinical practice,
the true potential of simulation can only be realized if it is closely
aligned with the clinical environment.
6
In an attempt to achieve this
harmony, an evolution of simulators from bench-top models to simu-
lated operating rooms and virtual worlds has occurred.
7,8
This focus
on the fidelity of simulators, however, has led to an unanticipated
neglect of the value of simulation and the entire learning experience.
9
A broader perspective is required to maximize the training value of
simulation and to justify the significant faculty and financial input.
10
Practices which permit optimal learning for residents from every
simulated encounter must be actively sought to render such training
cost-effective.
The provision of high-quality debriefing affords such an op-
portunity. Debriefing refers to the process whereby participants reex-
amine the encounter so as to develop the appropriate competencies
required for clinical practice.
11
Viewed through the lens of educa-
tional theory, debriefing is an integral component of Kolb’s experien-
tial learning cycle
12
and Schon’s reflective practice.
13
From a clinical
perspective, debriefing allows trainees to discuss their performance
with a trainer so as to identify gaps and strategies for improvement.
11
In doing so, it reduces adverse events and aids transfer of simulation-
acquired skill to real operating room practice.
14,15
Although extensively employed by other safety-critical
industries,
16
empirical evidence on debriefing in surgery is surpris-
ingly sparse. Only 2 such studies exist—both of which are descrip-
tive accounts of individual debriefing practices.
11,17
Furthermore, de-
spite being widely acknowledged as the most essential component of
simulation-based training,
15
a dearth of truly evidence-based guide-
lines exists as to what constitutes effective debriefing. Consequently,
an ad-hoc approach has evolved potentially missing opportunities to
consistently embed deep learning. An inability to measure debriefing
quality has also hindered the provision of formative feedback to train-
ers (ie, the debriefers) on how to improve their debriefing practices,
despite being something that they would very much value.
17
An understanding of the constituents of an educationally effec-
tive debrief coupled with robust tools for its assessment would allow
the entire surgical community to identify, aspire to, measure, and share
best practices. Enhanced learning through empirically improved de-
briefing may also be the most cost-effective mechanism required for
unlocking the effectiveness of simulation, thereby realizing its full
potential as a training paradigm.
The aims of this study were to firstly identify the components of
an educationally effective debriefing in surgery using best-evidence
and end-user opinion. Following that, the aim was to develop and
validate a psychometrically robust measurement tool to assess the
quality of debriefings, termed the Objective Structured Assessment of
Debriefing (OSAD).
METHODS
This study sought to establish the key features of OSAD de-
lineated in Figure 1. To achieve this goal, the study progressed in
Copyright © 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
982 | www.annalsofsurgery.com Annals of Surgery
Volume 256, Number 6, December 2012