497 MEDINFO 2004 M. Fieschi et al. (Eds) Amsterdam: IOS Press © 2004 IMIA. All rights reserved Overcoming Barriers to Evaluation of Terminological Systems Ronald Cornet a , Nicolette F. de Keizer a , Daniëlle G.T. Arts a,b a dept. of Medical Informatics, Academic Medical Center, Amsterdam, The Netherlands b dept. of Intensive Care, Academic Medical Center, Amsterdam, The Netherlands Ronald Cornet, Nicolette F. de Keizer, Daniëlle G.T. Arts Abstract Evaluation of terminological systems has been demonstrated to be a complicated task. This is due to the broad range of termino- logical systems, their application, and the clinical contexts in which they can be applied. We propose an evaluation framework that explicitly distinguishes an application-independent descrip- tion of terminological systems, methods to determine applica- tion-dependent requirements, and methods to assess the applicability. In order to support a systematic application-inde- pendent description of terminological systems, we present a cat- egorization of characteristics, including explicit questions. The answers to these questions can be mapped to the requirements for a certain application of a terminological system. This frame- work aims at reducing the efforts for determining which termi- nological system is applicable for a certain clinical setting. Keywords: Terminology, Evaluation. Introduction After decades in which the number and complexity of termino- logical systems have grown, the topics of standardization and understanding of these systems are getting increasing attention. This has resulted in various publications that address description and evaluation of terminological systems. Among these publica- tions is [1], in which seven barriers to evaluation have been de- termined: 1. Application dependence: quality of terminological sys- tems is defined relative to its intended use; 2. Empirical vs. independent assessment: subjective exami- nation to measure characteristics of terminological sys- tems instead of objective measures 3. Dichotomous vs. continuous measures of characteristics: presence or absence of features versus the extent of use of features; 4. Poor definition of characteristics: different authors use the same name for different characteristics; 5. Number of characteristics: evaluation is hardened by the large number of characteristics; 6. Multiple levels of significance of characteristics: rele- vance of characteristics varies and is user-dependent; 7. Interdependence of characteristics: characteristics may influence each other. Among the publications that aim to overcome one or more of these barriers is [2], specifying 12 desiderata that were distilled (mainly) from literature from the nineties of the previous centu- ry. The "Standards Specification for Quality Indicators for Con- trolled Health Vocabularies" [3] is a further step towards structured specification of terminological systems, which distin- guishes between: general information, structure of the terminol- ogy model, maintenance, and (methods for) evaluation. The Object Management Group (OMG) has used a functionality-ori- ented approach in the Lexicon Query Service Specification (LQS) [4], which defines "methods for accessing the content of medical terminology systems". Earlier work at our department resulted in a “framework for understanding terminological sys- tems”, which addresses the characteristics that distinguish vari- ous types of terminological systems [5-7]. In 2003, the National Committee on Vital and Health Statistics Subcommittee on Standards and Security has made an inventory of about 40 termi- nological systems to come to national terminology standards for Patient Medical Record Information [8]. This inventory was based on a questionnaire that contained between 40 and 100 (de- pending on the level of detail considered) questions regarding various aspects of terminological systems (and their developers). This overview shows that various efforts have been made to- wards evaluation of terminological systems, overcoming at least some of the barriers. In this paper we propose a general framework with methods to evaluate terminological systems that pay attention to all seven barriers. We will first focus on the first three barriers; application dependence, assessment and measures of characterization, which all involve the evaluation methods. Thereafter, we will elaborate on a categorization of characteristics, overcoming the fourth to seventh barrier, as these are all related to the character- istics of terminological systems which have to be taken into ac- count during an evaluation. Next, the categorization of characteristics is used to present an example of the application- independent description of terminological systems. We demon- strate how the literature discussed earlier relates to the various parts of the proposed framework, and we will address issues that require further research. Evaluation Methods Evaluation of the applicability of terminological systems for a specific clinical setting is a complex endeavor. This complexity originates from the lack of agreed upon yardsticks to represent