Process-monitoring-for-quality—A robust model selection criterion for the logistic regression algorithm Carlos A. Escobar a,b,⇑ , Ruben Morales-Menendez b a Global Research & Development, General Motors, Warren, MI, USA b Tecnológico de Monterrey, Monterrey, NL, Mexico article info Article history: Received 1 November 2018 Received in revised form 20 June 2019 Accepted 5 September 2019 Available online 6 September 2019 Keywords: Logistic regression Model selection criterion Binary classification Highly unbalanced data structures Big data driven manufacturing Pattern recognition abstract Process Monitoring for Quality is a big data-driven quality philosophy aimed at defect detection through binary classification. The l 1 -regularized Logistic Regression learning algorithm has been successfully applied in manufacturing systems for rare quality event detection. Since the optimal value of the regu- larization parameter is not known in advance, many models should be created and tested to find the final model to be deployed at the plant. In this context, model selection becomes a critical step in the process of developing a manufacturing functional model. Since most mature organizations generate only a few Defects Per Million of Opportunities, a three-dimensional model selection criterion (3D  LR) was initially introduced aimed at analyzing highly/ultra unbalanced binary data structures. The 3D  LR criterion combines three of the most important attributes – prediction, separability, complexity – of each candidate model and map them into a three dimensional space to select the best one. In this letter, the 3D  LR is improved; the fit attribute is replaced by a novel separability index that takes into consideration the clas- sification threshold to reward for robustness of predictions. Updated criterion, 3D  LRI, is an improved version of the initial concept. Ó 2019 Society of Manufacturing Engineers (SME). Published by Elsevier Ltd. All rights reserved. 1. Introduction Process Monitoring for Quality (PMQ) is a big data-driven quality philosophy aimed at defect detection through binary classification [1,2]. It is founded on Big Models, a predictive modeling paradigm based on machine learning, statistics and optimization aimed at developing a manufacturing functional model [3]. In today’s manufacturing world, most mature organizations generate only a few Defects Per Million of Opportunities (DPMO); therefore, manufacturing-derived data sets for binary classification of quality tend to be highly/ultra (minority class count < 1%) unbalanced. Detecting these defects is one of the main intellectual challenges addressed by PMQ. The l 1 -regularized Logistic Regression (LR) Machine Learning Algo- rithm (MLA) has shown high ability detecting rare quality events when the Classification Threshold (CT) is properly set [4,5]. From a hyper-dimensional feature-space, this MLA helps to identify the most relevant features of the manufacturing system. This information is used to plan/design randomize experiments to find optimal operating conditions. The l1-i.e.,regularization term [6], k, is a hyper-parameter [7] used to penalize for model complexity, i.e., number of features. Since its optimal value is not known in advance, many Candidate Models (CM) should be created and tested to find the final model, i.e., classifier to be deployed into production. From generalization (prediction on unseen data) perspective, one of the main objectives of testing many models is to prevent overfitting/underfitting problems [8,9] by finding the right amount of regularization, i.e., hyper-parameter tuning. This task can be broken down into: (1) model creation and (2) Model Selection (MS) problems, with the latter being the only focus of this paper. Model selection criteria are founded on the principle of parsimony [10]. In manufacturing, parsimonious modeling eases information extraction [3], induces model trust [11], and explain- ability [12]; desired characteristics for the successful deployment of a MLA-based model into production. The three-dimensional (3D) MS criterion (3D  LR) was initially introduced in [13]. Proposed criterion combines prediction, fit, and complexity attributes of each CM to project them into a 3D space to select the final model that solves the posed tradeoff between these three competing-attributes the best. Criterion rewards CM with https://doi.org/10.1016/j.mfglet.2019.09.001 2213-8463/Ó 2019 Society of Manufacturing Engineers (SME). Published by Elsevier Ltd. All rights reserved. ⇑ Corresponding author at: Global Research & Development, General Motors, Warren, MI, USA. E-mail address: carlos.1.escobar@gm.com (C.A. Escobar). Manufacturing Letters 22 (2019) 6–10 Contents lists available at ScienceDirect Manufacturing Letters journal homepage: www.elsevier.com/locate/mfglet