ORIGINAL REPORT Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data Jennifer C. Nelson 1,2 *, Andrea J. Cook 1,2 , Onchee Yu 1 , Clara Dominguez 1,2 , Shanshan Zhao 2 , Sharon K. Greene 3 , Bruce H. Fireman 4 , Steven J. Jacobsen 5 , Eric S. Weintraub 6 and Lisa A. Jackson 7 1 Biostatistics Unit, Group Health Research Institute, Seattle, WA, USA 2 Department of Biostatistics, University of Washington, Seattle, WA, USA 3 Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA 4 Kaiser Permanente Division of Research, Oakland, CA, USA 5 Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA, USA 6 Immunization Safety Office, Centers for Disease Control and Prevention, Atlanta, GA, United States 7 Group Health Research Institute, Seattle, WA, USA ABSTRACT Purpose Many challenges arise when conducting a sequentially monitored medical product safety surveillance evaluation using observa- tional electronic data captured during routine care. We review existing sequential approaches for potential use in this setting, including a continuous sequential testing method that has been utilized within the Vaccine Safety Datalink (VSD) and group sequential methods, which are used widely in randomized clinical trials. Methods Using both simulated data and preliminary data from an ongoing VSD evaluation, we discuss key sequential design considera- tions, including sample size and duration of surveillance, shape of the signaling threshold, and frequency of interim testing. Results and Conclusions All designs control the overall Type 1 error rate across all tests performed, but each yields different tradeoffs between the probability and timing of true and false positive signals. Designs tailored to monitor efficacy outcomes in clinical trials have been well studied, but less consideration has been given to optimizing design choices for observational safety settings, where the hypotheses, population, prevalence and severity of the outcomes, implications of signaling, and costs of false positive and negative findings are very differ- ent. Analytic challenges include confounding, missing and partially accrued data, high misclassification rates for outcomes presumptively de- fined using diagnostic codes, and unpredictable changes in dynamically accessed data over time (e.g., differential product uptake). Many of these factors influence the variability of the adverse events under evaluation and, in turn, the probability of committing a Type 1 error. Thus, to ensure proper Type 1 error control, planned sequential thresholds should be adjusted over time to account for these issues. Copyright © 2012 John Wiley & Sons, Ltd. key words—observational study; pharmacovigilance; postmarketing; sequential testing; study design; vaccine and drug safety INTRODUCTION To bolster postmarket safety surveillance for regulated medical products, innovative systems are being devel- oped to monitor electronic health care records that are routinely collected by insurance plans. Such systems, which include the Vaccine Safety Datalink (VSD) project of the Centers for Disease Control and Preven- tion (CDC) 1–3 and the Sentinel System of the US Food and Drug Administration (FDA), 4 involve capturing and prospectively analyzing data as they accrue across multiple health plan populations. Thus, they offer promise to provide an active safety monitoring frame- work that is rapid, statistically powerful, and cost- effective. To date, the primary goal of systems such as the VSD and the Sentinel System has been to eval- uate previously suggested safety signals (e.g., those identified during premarket trials or from spontaneous *Correspondence to: J. C. Nelson, Biostatistics Unit, Group Health Research Institute, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101, USA. E-mail: nelson.jl@ghc.org Copyright © 2012 John Wiley & Sons, Ltd. pharmacoepidemiology and drug safety 2012; 21(S1): 62–71 Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.2324