A Comparison of Supervised Machine Learning Algorithms and Feature Vectors for MS Lesion Segmentation Using Multimodal Structural MRI Elizabeth M. Sweeney 1,2 *, Joshua T. Vogelstein 3,4 , Jennifer L. Cuzzocreo 5 , Peter A. Calabresi 6 , Daniel S. Reich 1,2,5,6 , Ciprian M. Crainiceanu 1 , Russell T. Shinohara 7 1 Department of Biostatistics, The Johns Hopkins University, Baltimore, Maryland, United States of America, 2 Translational Neuroradiology Unit, Neuroimmunology Branch, National Institute of Neurological Disease and Stroke, National Institute of Health, Bethesda, Maryland, United States of America, 3 Department of Statistical Science, Duke University, Durham, North Carolina, United States of America, 4 Center for the Developing Brain, Child Mind Institute, New York, New York, United States of America, 5 Department of Radiology, The Johns Hopkins University School of Medicine, Baltimore, Maryland, United States of America, 6 Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, Maryland, United States of America, 7 Department of Biostatistics and Epidemiology, Center for Clinical Epidemiology and Biostatistics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States of America Abstract Machine learning is a popular method for mining and analyzing large collections of medical data. We focus on a particular problem from medical research, supervised multiple sclerosis (MS) lesion segmentation in structural magnetic resonance imaging (MRI). We examine the extent to which the choice of machine learning or classification algorithm and feature extraction function impacts the performance of lesion segmentation methods. As quantitative measures derived from structural MRI are important clinical tools for research into the pathophysiology and natural history of MS, the development of automated lesion segmentation methods is an active research field. Yet, little is known about what drives performance of these methods. We evaluate the performance of automated MS lesion segmentation methods, which consist of a supervised classification algorithm composed with a feature extraction function. These feature extraction functions act on the observed T1-weighted (T1-w), T2-weighted (T2-w) and fluid-attenuated inversion recovery (FLAIR) MRI voxel intensities. Each MRI study has a manual lesion segmentation that we use to train and validate the supervised classification algorithms. Our main finding is that the differences in predictive performance are due more to differences in the feature vectors, rather than the machine learning or classification algorithms. Features that incorporate information from neighboring voxels in the brain were found to increase performance substantially. For lesion segmentation, we conclude that it is better to use simple, interpretable, and fast algorithms, such as logistic regression, linear discriminant analysis, and quadratic discriminant analysis, and to develop the features to improve performance. Citation: Sweeney EM, Vogelstein JT, Cuzzocreo JL, Calabresi PA, Reich DS, et al. (2014) A Comparison of Supervised Machine Learning Algorithms and Feature Vectors for MS Lesion Segmentation Using Multimodal Structural MRI. PLoS ONE 9(4): e95753. doi:10.1371/journal.pone.0095753 Editor: Bogdan Draganski, Centre Hospitalier Universitaire Vaudois Lausanne - CHUV, UNIL, Switzerland Received January 3, 2014; Accepted March 28, 2014; Published April 29, 2014 This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication. Funding: This research was partially supported by the Intramural Research Program of NINDS and NINDS R01 NS070906 and NINDS RO1 NS08521. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: emsweene@jhsph.edu Introduction Machine learning is a popular perspective for mining and analyzing large collections of medical data [1–3]. We focus on the extent to which the choice of machine learning or classification algorithm and the feature extraction function impact performance in one problem from medical research – supervised multiple sclerosis (MS) lesion segmentation in structural magnetic reso- nance imaging (MRI). The evaluation of the classification algorithms employed in supervised lesion segmentation methods is not only a function of classification accuracy. Depending on the application, computational efficiency and interpretability may be valued at the cost of classification accuracy. Therefore, our evaluation also includes the computational time and resources required by each algorithm and the interpretability of the results produced by the algorithm. Comparison of machine learning techniques has been performed in other applications[4–6], but not to our knowledge in multiple sclerosis lesion segmentation. Also many of the currently available comparisons do not consider computational time. MS is a life-long chronic disease of the central nervous system that is diagnosed primarily in young adults who will have a near normal life expectancy. Because of this, the burden of the disease is great, with large economic, social and medical costs. Between 250,000 and 400,000 people in the United States have been diagnosed with MS, and the estimated annual cost of the disease is over six billion dollars. There is currently no cure for MS, but many therapies exist for treating symptoms and delaying accumulation of permanent disability (http://www.ninds.nih. gov/disorders/multiple_ sclerosis/detail_multiple_sclerosis.htm). MS is characterized by demylinating lesions that are predomi- nately located in the white matter of the brain, and MRI of the brain is sensitive to these lesions [7]. Quantitative MRI metrics, such as the number and volume of lesions, are important clinical tools for research into the pathophysiology and natural history of PLOS ONE | www.plosone.org 1 April 2014 | Volume 9 | Issue 4 | e95753