International Journal of Forecasting 27 (2011) 635–660 www.elsevier.com/locate/ijforecast Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction Sven F. Crone a,∗ , Mich` ele Hibon b , Konstantinos Nikolopoulos c a Lancaster Centre for Forecasting, Lancaster University Management School, Lancaster, UK b Decision Sciences, INSEAD, Fontainebleau, France c Decision Sciences Research Centre, Manchester Business School, Manchester, UK Available online 12 May 2011 Abstract This paper reports the results of the NN3 competition, which is a replication of the M3 competition with an extension of the competition towards neural network (NN) and computational intelligence (CI) methods, in order to assess what progress has been made in the 10 years since the M3 competition. Two masked subsets of the M3 monthly industry data, containing 111 and 11 empirical time series respectively, were chosen, controlling for multiple data conditions of time series length (short/long), data patterns (seasonal/non-seasonal) and forecasting horizons (short/medium/long). The relative forecasting accuracy was assessed using the metrics from the M3, together with later extensions of scaled measures, and non-parametric statistical tests. The NN3 competition attracted 59 submissions from NN, CI and statistics, making it the largest CI competition on time series data. Its main findings include: (a) only one NN outperformed the damped trend using the sMAPE, but more contenders outperformed the AutomatANN of the M3; (b) ensembles of CI approaches performed very well, better than combinations of statistical methods; (c) a novel, complex statistical method outperformed all statistical and CI benchmarks; and (d) for the most difficult subset of short and seasonal series, a methodology employing echo state neural networks outperformed all others. The NN3 results highlight the ability of NN to handle complex data, including short and seasonal time series, beyond prior expectations, and thus identify multiple avenues for future research. c ⃝ 2011 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved. Keywords: Time series forecasting; Empirical evaluation; NN3 competition; Artificial neural networks; Computational intelligence 1. Introduction Back in 1993, Chatfield wondered, “Neural net- works: forecasting breakthrough or passing fad?”; and ∗ Corresponding author. Tel.: +44 1524 5 92991. E-mail address: s.crone@lancaster.ac.uk (S.F. Crone). the question still remains largely unanswered today. On the one hand, if we consider only the number of publications relating to artificial neural networks (NN), the answer would seem to indicate that they were a breakthrough: motivated by their theoretical properties of non-parametric, data driven universal ap- proximation of any linear or nonlinear function, the 0169-2070/$ - see front matter c ⃝ 2011 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved. doi:10.1016/j.ijforecast.2011.04.001