1 Testing neural networks using the complete round robin method Boris Kovalerchuk, Clayton Todd, Dan Henderson Dept. of Computer Science, Central Washington University, Ellensburg, WA, 98926-7520 borisk@tahoma.cwu.edu toddc.@cwu.edu hendersond@cwu.edu Abstract The reliability of learning methods depends on testing discovered patterns in the data before they are used for their intended purpose. A novel extension of the round robin testing method is presented in this paper. The novelty includes the mathematical mechanism used based on the theory of monotone Boolean functions and multithreaded parallel processing. This mechanism speeds up required computations. The method has been implemented for backpropagation neural networks and successfully tested for 1024 neural networks by using SP500 data. 1. Approach and Method One common approach used to test learned regularities is to divide the data set D into two parts and use one part for training and another part for validating the discovered patterns. This process is repeated several times and if results are similar to each other than a discovered regularity can be called reliable for data D. Three major methods for selecting subsets of training data are known as: 1. Random selection of subsets, 2. Selection of disjoint subsets, 3. Selection of subsets according the probability distribution. These methods are sometimes called, respectively, bootstrap aggregation (bagging), cross-validated committees, and boosting [Dietterich, 1997]. Problems of sub-sampling . Different sub- samples of a sample, Tr, can be governed by different regularities. Rejecting and accepting regularities heavily depends on a specific splitting of Tr. This is common for non- stationary financial time series, e.g., bear and bull market trends for different time intervals. Assume that 70% of Tr is governed by regularity #2 and only 30% by regularity #1. If accidentally, test set A consists of all cases governed by regularity #1, then regularity #2 found on A’ will be rejected although it is true for 70% of the sample Tr. Therefore, such tests can reflect rather an arbitrary splitting instead of the real strength of regularities on data. A more comprehensive approach is the round robin method . It is designed to eliminate arbitrary splitting by examining several groups of subsets of Tr. If these groups do not cover all possible subsets then round robin approach faces the problem of selecting independent subsets and determining their sizes. The complete round robin method examines all groups of subsets of Tr. The obvious drawback with the complete round robin is that there are 2 n possible subsets, where n is the number of groups of objects in the data set. Learning 2 n neural networks is a computational challenge. Below we present an original implementation of the complete round robin method and techniques to speed up required computations along with experimental testing of 1024 neural networks constructed using SP500 data. This method uses the concepts of monotonicity and multithreaded parallel processing for Windows NT. It is applicable to both attribute-based and relational data mining methods . However in this paper, the method is illustrated only for neural networks. Let M be a learning method and D be a set of N objects, represented by m attributes. D={d i },i=1,…,N, d i =(d i1 , d i2 ,…, d im ). Method M is applied to data D for knowledge discovery. We assume that data are grouped, for example in a stock time series, the first 250 data objects (days) belong to 1980, the next 250 objects (days) belong to 1981, and so on. To simplify the example, assume we have ten groups (years), n=10. Similarly, half years, quarters and other time intervals can be used. There are 210 possible subsets of the data groups. Any of these subsets can be used as training data. If the data do not represent a time series, then all their complements without constraint can be