- - 1 Cost Evaluation of Coverage Directed Test Generation for the IBM Mainframe Gilly Nativ, Steven Mittermaier, Shmuel Ur, Avi Ziv IBM Corporation Abstract Test generation and simulation tools have input stimuli that can direct them to cover specific events. However, the cost of completely covering a verification plan is still very high. While coverage analysis tools can find events that have not been covered, they do not provide an automated covering method. This paper presents the first implementation of a generation framework that uses feedback from coverage analysis to direct microarchitecture simulation. This framework uses a coverage analysis tool to find events that have not been simulated and then utilizes information about the design to determine which directives should be given to the simulation environment. This paper describes, in detail, the system and its operation process, an experiment that uses the system, and the results of the experiment. This system was shown to reduce the machine time and person time required to cover the test plan. Implications of this work suggest the types of verification plans appropriate for the utilization of this system and the further experiments and developments required. 1. Introduction Nowadays, microarchitecture simulation is the major technique used for processor verification. Microarchitecture simulation requires a lot of expert time and computer resources. Formal Verification techniques [1][2] cannot handle the size of modern processors and can only be used for specific portions of the design. In current industrial practice, test generation and simulation tools perform large portions of the verification plan. In this technique, huge amounts of tests are generated and executed over a simulation model and the results are checked to match the expected results. The test space is enormous and even a large number of tests can cover only a small portion of the test space. In practice, actual coverage of the global test space is unknown and the verification process does not provide feedback on the quality of the tests that are simulated. The main technique used to measure the thoroughness of the verification is called coverage analysis [3][4]. The idea of coverage analysis is to compile, in some systematic fashion, a large and comprehensive list of tasks called a coverage model. Each item in this list is called a coverage task and represents some event that should be covered in verification. Coverage analysis tools can provide data about the verification status and progress over the subject coverage model. The coverage analysis tools can find the coverage tasks that have not been covered, since the tasks require changes and biases to be performed in the verification environment. Coverage analysis provides a measurement of the quality of verification 1 and helps find tasks that have not been covered. However, it does not provide a method to cover these tasks or achieve better coverage in the verification process. Covering each task by manually redirecting the verification process would take a large amount of expert time. A verification methodology called Coverage Directed Generation (CDG) aims to solve this problem. There are two major techniques for CDG: by Construction, and by Feedback. CDG by Construction [5][6][7][8] is based on a provided translation engine that can translate a verification task into a simulation test. In general, it involves describing the coverage tasks in some fashion. It then uses the translation engine to translate each one of these tasks into a simulation test. This method's underlying assumption is that the translation is exact, and therefore, the derived test covers the verification tasks with complete certainty. Task coverage is assured as early as the time of construction of the test. CDG by Feedback also requires a translation engine, but it does not assume that it is faultless. This technique uses coverage analysis to obtain feedback on the success of the translation and reiterates the translation process for the tasks that have not been covered. Here, the underlying assumption is that if the translation is inaccurate one or more times, it can still succeed on an additional attempt to produce a covering test. Given the huge size and complexity of the simulation model, building a faultless translation engine would require an enormous investment of expert time, and would only be suitable for a specific coverage model. Feedback CDG, which demands less from 1 In the environment used for this work, test generation is done cycle-by-cycle, alongside simulation. For each cycle, there is a generation phase, immediately followed by a simulation phase. Therefore, the term simulation, in this paper, also includes generation.