IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 337 Scientiﬁc Foundations to the Multilevel Method B. Neelakantan and S. V. Raghavan, Senior Member, IEEE Abstract— Conformance testing of protocols is the process of checking whether an implementation under test conforms to the standards. In this paper we prove that the multilevel method (M method) [which splits the speciﬁcation graph into a basic subgraph and one or more higher level subgraph(s)] is capable of diagnosing any number of faults in the higher level subgraph(s), if the basic subgraph is error-free. Heuristics for obtaining the basic subgraph and higher level subgraphs from the speciﬁcation graph are also given. The advantage of the M method is that it has error recovery and, in addition, it does not assume the presence of reliable reset in the implementation under test. We additionally propose an incremental test sequence generation method, which produces a shorter test sequence and whose fault coverage is same as that of the M method. Finally, we compare the M method with other fault detection and fault diagnostic methods. Index Terms—Black box testing, ﬁnite state machines, protocol conformance testing, test methods, test sequence. I. INTRODUCTION P ROTOCOL conformance testing has been an active area of research in the ﬁeld of computer networks and dis- tributed computing. From the time that the International Stan- dards Organization (ISO) started developing standards for conformance testing of network protocols, a lot of research work has been undertaken in this ﬁeld [10]. Conformance test- ing involves testing both the capabilities and the behavior of an implementation and checking what is observed against the conformance requirements in the relevant recommendations and against what the implementor states as the implementa- tion capabilities [10], [20]. This is done by applying a test sequence to the implementation under test (IUT) and giving a verdict by comparing the observed output with the expected output. Conformance testing does not assess the performance, robustness, or reliability of an implementation [10], [20]. Network protocols can be speciﬁed as deterministic ﬁnite state machines (DFSM). The IUT is assumed to be a black box modeled as a DFSM. The internal states of the IUT are not visible and can be identiﬁed only by analyzing its input/output characteristics. A sequence of inputs, called the test sequence, when applied to the IUT, should be able to verify if all of the states that are present in the speciﬁcation are also present in the IUT. In addition to that, it should also verify if all of the speciﬁed transitions have same head and tail states with the same input and output characteristics. As the complexity of the protocols increases, the lengths of the test sequences also increase. This necessitates the devel- Manuscript received September 29, 1994; revised October 11, 1995; ap- proved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor K. Sabnani. The authors are with the Network Systems Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology, Madras 600 036 India (e-mail: svr@shiva.iitm.ernet.in). Publisher Item Identiﬁer S 1063-6692(98)04077-1. opment of systematic methods for test sequence generation. Various test sequence generation methods like transition tour (T), distinguishing sequence (D), characterizing sequence (W), unique input–output (U), extended transition tour (E), regular checking (R), fault resolution (FR), diagnostic approach (DA), and multilevel (M) methods are proposed in the literature [3], [7], [9], [13], [15], [16], [18], [19], [21]. The test sequence generation methods (T, D, W, U, E, R, FR, and DA methods) take the entire speciﬁcation as a single entity for the test sequence generation. Since testing the entire protocol as a single entity is complex in nature, the method of splitting the given graph (i.e., the given DFSM) into smaller subgraphs and applying test sequences on them was advocated by [14] and [15]. This approach, called the M method, can detect and locate any number of faults in the IUT, if the basic subgraph is error-free. The key steps in the M method are: • identiﬁcation of the basic subgraph and higher level subgraphs; • testing the basic subgraph by a method that is capable of detecting faulty IUT even in the presence of multiple faults; • testing the higher level subgraphs by a method which should have fault diagnosis capability. In [14] and [15] the identiﬁcation of the subgraphs at various levels was more of an art than a science. In this paper we provide a formal approach to the M method. In addition to that, we propose an incremental test sequence generation method, which is capable of diagnosing any number of faults in the higher level subgraph if the basic subgraph is error-free. The added advantage of this method is that the length of the test sequence is usually less than that of the M method. If the basic subgraph is the same or close to the speciﬁcation graph, then the advantage of this method will be somewhat lost. The rest of the paper is organized as follows. Section II introduces the terminologies that are used in the rest of the paper. Section III deals with the state-of-the-art in conformance testing of protocols. Section IV brieﬂy outlines the R method [16]. Section V discusses the M method in detail. Section VI compares the M method with other testing methods and brings out the advantages and shortcomings of the M method. Section VII discusses the incremental test sequence generation method to protocol testing. Section VIII summarizes the contributions of this paper. II. PRELIMINARIES A DFSM is a quintuple , where , , and are ﬁnite nonempty sets of inputs, outputs, and states, respectively, is the state transition 1063–6692/98$10.00  1998 IEEE