2168-2194 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JBHI.2019.2948734, IEEE Journal of Biomedical and Health Informatics A J Batten et al.: A Resampling Based Grid Search Method for with Application to Mixture-Item Response Theory Models of Multimorbid High-Risk Patients Abstract— There are many statistics available to the applied statistician for assessing model fit and even more methods for assessing internal and external validity. We detail a useful approach using a grid search technique that balances the internal model consistency with generalizability and can be used with models that naturally lend themselves to multiple assessment techniques. Our method relies on resampling and a simple grid search method over 3 commonly used statistics that are simple to calculate. We apply this method in a latent traits framework using a mixture Item Response Theory (MIXIRT) model of common chronic health conditions. Model fit is assessed using Akaike’s Information Criteria (AIC), latent class similarity is measured with the Variance of Information (VI), and the consistency of condition complexity and prevalence across latent classes is compared using Kendall’s τ rank order statistic. From two patient cohorts at high risk for hospitalization in 2014 and 2018, we generated 19 MIXIRT models (allowing 2-20 latent classes) on 21 common comorbid conditions identified via healthcare encounter diagnosis codes. We ran these models on 100 bootstrap samples of size 10% for each cohort. Among the resulting models, combined AIC and VI statistics identified 5-7 latent classes, but the rank order correlation of condition complexity revealed that only the 5 class solutions had consistent condition complexity. The 5 class solutions were combined to produce a single parsimonious MIXIRT solution that balanced clinical significance with model fit, cluster similarity, and consistency of condition complexity. Index Terms—chronic conditions, high-risk patients, item response theory, latent class analysis, multimorbidity INTRODUCTION Globally, the number of people with multiple medical conditions, known as multimorbidity, is rapidly increasing. Managing the complex health needs and high costs of medical This work was submitted December 2018 and was undertaken as part of the Veterans Administration’s PACT Demonstration Laboratory initiative, supporting and evaluating VA’s transition to a patient-centered medical home. Data for this report were developed by the national evaluation team at the PACT Demonstration Lab Coordinating Center. Funding for the PACT Demonstration Laboratory Initiative is provided by the VA Office of Primary Care. The views expressed are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs. A. J. Batten is with the Primary Care Analytics Team, Veterans Health Administration, Seattle, WA 98108 USA (e-mail: adam.batten@va.gov) care associated with multimorbidity is a great challenge for patients, providers, and healthcare systems [1, 2]. Although the concept of multimorbidity seems straightforward, there is little consensus on how best to define and measure multimorbidity in patients [3]. Prior work to empirically define types of multimorbidity has generally taken one of two approaches: (1) a dimensional, scoring approach (e.g. latent trait models and factor analysis) where multimorbidity in patients is scored on a hypothesized scale based on the degree or complexity of multimorbidity [4, 5], or (2) a classification approach (e.g. latent class, finite mixture models) where patients with similar combinations of specific chronic illnesses are classified into multimorbidity clusters [6, 7, 8, 9]. Each approach has different strengths and different limitations for targeting clinical interventions. The advantage of the dimensional approach is that patients can be assigned a quantitative score based on the complexity of their comorbidity, which could be used to inform medical professionals on the intensity or urgency of medical attention. The advantage of classification approaches is the ability to separate patients into clusters with similar multimorbidity patterns. This offers the potential for “bundling” healthcare interventions, simultaneously or sequentially, in a way that can more effectively target groups of patients struggling with similar co-occurring illnesses. One solution that derives benefit from the unique strengths of both dimensional approaches (item response theory [IRT] models) and classification approaches (latent class analysis [LCA]) is a class of models known as factor mixture models [7]; and specifically, mixture item response theory (MIXIRT) models [10, 11]. These methods are based on the Rasch model which was originally devised for psychometric tests for estimating both the complexity of test items and test-taker ability [12]. Over the years, Rasch models have been extended to include latent class and mixed methods, and applied to increasingly complex problems in both the social and health sciences where these methods are more commonly referred to J. Thorpe is with the VA Center for Health Equity and Promotion, Pittsburgh, PA 15213 USA (e-mail: Joshua.Thorpe@va.gov). R. Piegari is with Clinical Systems Development & Evaluation, Veterans Health Administration, Canandaigua, NY 14424 USA (e-mail: Rebecca.Piegari@va.gov). A. M. Rosland is with the VA Center for Health Equity and Promotion, Pittsburgh, PA 15213 USA and Department of Internal Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA 15260 USA (e-mail: Ann-Marie.Rosland@va.gov) A Resampling Based Grid Search Method to Improve Reliability and Robustness of Mixture-Item Response Theory Models of Multimorbid High-Risk Patients Adam J. Batten, Joshua Thorpe, Rebecca I. Piegari, Ann-Marie Rosland