Lessons Learned from Data Mining Challenges PETR BERKA University of Finance and Administration, Prague, Czech Republic Abstract. It becomes a good habit to organize a data mining cup, a competition or a challenge at machine learning or data mining conferences. Such events can be used for comparison of various approaches and algorithms, they give the participants a possibility to access and analyze real-world data, and they can result in knowledge interesting for the domain experts who provided the data. The paper describes our experience gained when organizing and evaluating the data mining challenges during European Conferences on Data Mining and Machine Learning. It shows the challenge settings, describes the used data and the solved tasks and summarizes the lessons learned. 1 Introduction It becomes a good habit to organize a data mining cup, a competition or a challenge at machine learning or data mining conferences. Such events serve several purposes: they can be used for comparison of various approaches and algorithms, they give the participants a possibility to access and analyze real-world data, and they can result in a knowledge interesting for the domain experts who provided the data. Cups and competitions are usually organized around a clearly specified classification problem. The participants are provided with pre-classified training data and a set of examples to be classified. The goal is to build a model that will perform well on the evaluation data. The models are then ranked according to their performance and the winners (sometimes also the losers) are announced. Thus the first purpose is stressed. Let us mention here e.g. the COIL2000 competition [6], the EUNITE2001 competition and, of course, the KDD cups held since 1997 [1]. Challenges have a less competitive nature. The aim here is to prepare conditions of a real/realistic data mining problem (classification or description) and to find a solution. The results are then discussed with the domain experts. This kind of events is organized e.g. at the European [4] or Pacific-Asian (e.g. [7]) KDD conferences. 2 ECML/PKDD Discovery Challenges The main idea of the Discovery Challenge organized at the European Conferences on Principles and Practice of Knowledge Discovery in Databases since 1999 [2] was to encourage a collaborative research effort, a broad and unified view of knowledge and methods of discovery, and emphasis on business problems and solutions to those problems.