IIF/SAS Grant 2005/6 Research Report – April 2008 Automatic Modelling and Forecasting with Artificial Neural Networks– A forecasting competition evaluation Final Report for the IIF/SAS Grant 2005/6 Sven F. Crone Department of Management Science, Lancaster University Management School, Lancaster, U.K. Konstantinos Nikolopoulos + Division of Business Systems, Manchester Business School, Manchester, UK. Michele Hibon * Department of Decision Sciences, Europe Campus, Fontainebleau, France. Abstract During 2007 we conducted an empirical evaluation of the accuracy of artificial Neural Networks (NN) and other methods of Computational Intelligence (CI) in time series prediction through a dedicated forecasting competition: the NN3 (www.neural-forecasting-competition.com). The competition aimed to resolve two research questions: (a) what is the current performance of CI methods in comparison to established statistical forecasting methods, and (b) what are the current “best practices” regarding the methodologies to model CI such as NN for time series forecasting. The NN3 competition evaluated the ex ante accuracy of multiple step ahead predictions across multiple error metrics. The data sample contained two homogeneous sets of 111 or 11 time series of varying length (short and long) and different time series patterns (seasonal and non-seasonal) taken from the original M3-competition, in order to analyse the conditions under which a particular method would perform well and to compare the accuracy to the contenders in the earlier competition. The NN3 competition attracted 60 submissions of CI methods as well as novel statistical contenders, one of the largest forecasting competitions conducted to date. The final results suggest that for monthly time series of different length and seasonality a variety of different CI methods are capable of forecasting automatically using a consistent methodology and show a robust and comparative performance, but that statistical methods still outperform the majority of CI- methods. + Corresponding Report Author: Email kostas.nikolopoulos@mbs.ac.uk Tel +44 07981 332913 * Dr. M Hibon was not funded by SAS/IIF for her participation in this project. She kindly agreed to analyse the forecasting performance of the competing methods and models using the same metrics and standards that she used to evaluate the original of the the M3 competition. The submission of this report as a full research paper to the International Journal of Forecasting will be co-authored by all three authors