Textual Evidence for the Perfunctoriness of Independent Medical Reviews Adrian Brasoveanu abrsvn@ucsc.edu University of California Santa Cruz Santa Cruz, CA Megan Moodie mmoodie@ucsc.edu University of California Santa Cruz Santa Cruz, CA Rakshit Agrawal ragrawal@camio.com Camio Inc. San Mateo, CA ABSTRACT We examine a database of 26,361 Independent Medical Reviews (IMRs) for privately insured patients, handled by the California Department of Managed Health Care (DMHC) through a private contractor. IMR processes are meant to provide protection for pa- tients whose doctors prescribe treatments that are denied by their health insurance (either private insurance or the insurance that is part of their worker comp; we focus on private insurance here). Laws requiring IMR were established in California and other states because patients and their doctors were concerned that health in- surance plans deny coverage for medically necessary services. We analyze the text of the reviews and compare them closely with a sample of 50000 Yelp reviews [19] and the corpus of 50000 IMDB movie reviews [10]. Despite the fact that the IMDB corpus is twice as large as the IMR corpus, and the Yelp sample contains almost twice as many reviews, we can construct a very good language model for the IMR corpus using inductive sequential transfer learn- ing, specifcally ULMFiT [8], as measured by the quality of text generation, as well as low perplexity (11.86) and high categorical accuracy (0.53) on unseen test data, compared to the larger Yelp and IMDB corpora (perplexity: 40.3 and 37, respectively; accuracy: 0.29 and 0.39). We see similar trends in topic models [17] and clas- sifcation models predicting binary IMR outcomes and binarized sentiment for Yelp and IMDB reviews. We also examine four other corpora (drug reviews [6], data science job postings [9], legal case summaries [5] and cooking recipes [11]) to show that the IMR re- sults are not typical for specialized-register corpora. These results indicate that movie and restaurant reviews exhibit a much larger variety, more contentful discussion, and greater attention to detail compared to IMR reviews, which points to the possibility that a crucial consumer protection mandated by law fails a sizeable class of highly vulnerable patients. CCS CONCEPTS · Computing methodologies → Latent Dirichlet allocation; Neural networks. KEYWORDS AI for social good, state-managed medical review processes, language models, topic models, sentiment classifcation In M. Gaur, A. Jaimes, F. Ozcan, S. Shah, A. Sheth, B. Srivastava, Proceedings of the Workshop on Knowledge-infused Mining and Learning (KDD-KiML 2020). San Diego, California, USA, August 24, 2020. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). KiML’20, August 24, 2020, San Diego, California, USA, © 2020 Copyright held by the author(s). https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Reference Format: Adrian Brasoveanu, Megan Moodie, and Rakshit Agrawal. 2020. Textual Ev- idence for the Perfunctoriness of Independent Medical Reviews. In Proceed- ings of KDD Workshop on Knowledge-infused Mining and Learning (KiML’20). , 9 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn 1 INTRODUCTION 1.1 Origin and structure of IMRs Independent Medical Review (IMR) processes are meant to provide protection for patients whose doctors prescribe treatments that are denied by their health insurance ś either private insurance or the insurance that is part of their workers’ compensation. In this paper, we focus exclusively on privately insured patients. Laws requiring IMR processes were established in California and other states in the late 1990s because patients and their doctors were concerned that health insurance plans deny coverage for medically necessary services to maximize proft. 1 As aptly summarized in [1], IMR is regularly used to settle dis- putes between patients and their health insurers over what is medi- cally necessary or experimental/investigational care. Medical ne- cessity disputes occur between health plans and patients because the health plan disagrees with the patient’s doctor about the ap- propriate standard of care or course of treatment for a specifc condition. Under the current system of managed care in the U.S., services rendered by a health care provider are reviewed to de- termine whether the services are medically necessary, a process referred to as utilization review (UR). UR is the oversight mech- anism through which private insurers control costs by ensuring that only medically necessary care, covered under the contractual terms of a patient’s insurance plan, is provided. Services that are not deemed medically necessary or fall outside a particular plan are not covered. Procedures or treatment protocols are deemed experimental or investigational because the health plan ś but not necessarily the patient’s doctor, who in many cases has enough clinical confdence in a treatment to order it ś considers them non-routine medical care, or takes them to be scientifcally unproven to treat the specifc condition, illness, or diagnosis for which their use is proposed. It is important to realize that the IMR process is usually the third and fnal stage in the medical review process. The typical progression is as follows. After in-person and possibly repeated examination of the patient, the doctor recommends a treatment, 1 For California, see the Friedman-Knowles Act of 1996, requiring California health plans to provide external independent medical review (IMR) for coverage denials. As of late 2002, 41 states and the District of Columbia had passed legislation creating an IMR process. In 34 of these states, including California, the decision resulting from the IMR is binding to the health plan. See [1, 15] for summaries of the political and legal history of the IMR system, and [2] for an early partial survey of the DMHC IMR data.