Evaluation of Prediction Models for Marketing Campaigns Saharon Rosset Amdocs Ltd. and Stanford University saharonr@amdocs.com Einat Neumann Amdocs (Israel) Ltd. and Tel-Aviv University einatn@amdocs.com Uri Eick Amdocs (Israel) Ltd. urieick@amdocs.com Nurit Vatnik Amdocs (Israel) Ltd. nuritv@amdocs.com Izhak Idan Amdocs (Israel) Ltd. izhaki@amdocs.com ABSTRACT We consider prediction-model evaluation in the context of marketing-campaign planning. In order to evaluate and compare models with specific campaign objectives in mind, we need to concentrate our attention on the appropriate evaluation-criteria. These should portray the model's ability to score accurately and to identify the relevant target population. In this paper we discuss some applicable model-evaluation and selection criteria, their relevance for campaign planning, their robustness under changing population distributions, and their employment when constructing confidence intervals. We illustrate our results with a case study based on our experience from several projects. Keywords Model Evaluation, Marketing Campaigns, Performance Measures, Confidence Intervals. 1. INTRODUCTION When dealing with marketing applications, such as campaign management, the issue of evaluating prediction models is twofold. First, the evaluation has to be statistically sound, allowing us to compare models, choose among them and estimate their expected future performance. Second, and perhaps more important, we need to evaluate models with regard to the way they will be utilized from a business perspective. For example, suppose we are building a scoring model to predict voluntary churn (customer’s propensity for disconnecting services) in order to identify the target population for a retention campaign. If in the campaign we intend to contact only the 2% of our customers who are at highest churn risk, it seems unreasonable to evaluate a suggested model using accuracy over a full test data set. The model’s performance on 98% of the population is irrelevant to the campaign goal. [6] and [8], among others, present flexible and efficient techniques for evaluating models with regard to a wide variety of goal functions. However, we have found the statistical analysis of the most relevant scores for planning campaigns to be lacking, and have compiled an array of tools and techniques to fill the gaps. In this paper we discusses some of the approaches we take when evaluating model-performance in the context of campaign planning and executing. We also present statistical issues that arise when attempting to combine relevance and rigor in the evaluation process. The main results we present are: • Description of the requirements from appropriate evaluation techniques for campaign planning and comparison of various relevant evaluation measures (Section 2). • Methodology for applying some of the evaluation measures (Section 3). This includes issues such as, score adjustment, distribution of scores and methods for constructing confidence intervals. • A case study (Section 4), illustrating the importance and usefulness of combining contextual and statistical considerations in model-evaluation. 2. MODEL EVALUATION We begin our discussion at the point where a scoring model has been constructed. We disregard the method or algorithm that were used to create the model, and concentrate on the means for evaluating it, given the campaign objectives. A different approach would be to consider the objectives while constructing the model ([1] and [4]). Once we have a candidate model, we want to estimate its expected performance on unlabeled data. Our standard model evaluation methodology is: 1. Evaluate the models’ performance on an independent test set (labeled data that has been set aside beforehand and not used in training the model). 2. Adjust the models’ score to fit the full population distribution, in case it is expected to be different from the sample distribution used for training and test (Section 3.1). We focus our discussion on the performance measures, which are of interest for campaign planning and analysis, and their statistical properties. 2.1 Planning Campaigns When planning a campaign, one seeks to identify individuals most likely to respond to the campaign. Due to budget restrictions the number of individuals to be approached in the campaign is limited. Thus there is a need for a good model for selecting the target segment and its performance on the rest of the population is of little or no consequence. The success of such a model is usually measured by the amount of responders captured within the targeted population. This amount can be measured in two different ways: