November, 1994 Page 1 MEASURING THE VALUE OF KNOWLEDGE 1 Yoram Reich Department of Solid Mechanics, Materials and Structures Faculty of Engineering Tel Aviv University Ramat Aviv 69978 Israel Key words (beside the title): evaluation metrics, measurement theory, verification and validation of expert systems, software engineering, design, machine learning, knowledge acquisition, research methodology ABSTRACT The quality of knowledge a system has substantiallyinfluences its performance. Often, the terms knowledge, its quality, and how it is measured or valuated, are left vague enough to accommodate several ad hoc interpretations. This paper articulates two definitions of knowledge and their associated value measures. The paper focuses on the theory underlying measurements and its application to knowledge valuation; it stresses the issue of constructing meaningful measures rather than discussingsome of the desirable properties of measures (e.g., reliability or validity). A detailed example of knowledge valuation using the measures is described. The example demonstrates the importance for system understanding and the difficulty of valuating knowledge. It shows the importance of employing several different measures simultaneously for a single valuation. The paper concludes by discussing the scope of and relationships between the measures. 1 INTRODUCTION In a world with information highways, many kinds of data, information, or knowledge become commodities whose trade will be based on, or requires methods of, valuation (Mowshowitz, 1994). The study of knowledge valuation methods is also motivated by more immediate reasons. The first and most general motivation is related to education and knowledge acquisition: Knowledge valuation methods can help identify good knowledge to be used by people or for inclusion in computer systems. The second motivation is methodological: Knowledge valuation methods can support the evaluation of systems developed in research or practice and the determination of their relative merit. This evaluation is essential for providing feedback on research progress and for supporting the refinement of ideas. In some situations, such as when developing systems by prototyping or developing learning systems, knowledge valuation methods must be used within projects and not only for comparing between them. A third motivation is related to building integrated systems: When a complex computer system has several competing modules for solving each of its task, knowledge valuation can identify which module to invoke for solving the task. There may be other motivations to study knowledge valuation but in this study, we limit the discussion to the second function: The valuation of knowledge embedded in, or to be used by, computer support systems. This motivation is related to the general need to evaluate intelligent systems and to the active field of verification 1 An different version appeared in the Proceedings of the Seventh Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Canada, 1992 Reich Internation Journal of Human-Computer Studies (was Knowledge Acquisition) (1995, in press)