Offline Evaluation of Term Utility Functions Alexander K. Seewald, Christian Holzbaur and Gerhard Widmer Austrian Research Institute for Artificial Intelligence Schottengasse 3, A-1010 Wien, Austria [alexsee,christian,gerhard]@ai.univie.ac.at Abstract In this paper we investigate characterizing ontology nodes corresponding to human- comprehensible concepts from the tool Melvil by a set of terms. We choose a variety of term utility functions, commonly use in text mining, to determine relative importance of terms for the task of deciding if a given document is part of a certain concept or not. We evaluated each utility function both quantitatively by considering precision and recall of the top ten terms returned and qualitatively by analyzing which of the original patterns and obviously related terms were recovered. This approach could be used to suggest promising terms to a human ontology editor during creation of a new node. Our results look somewhat promising but still needful of improvement – so we also report on probable causes of unsatisfactory results. 1 Introduction The tool melvil allows to create ontology nodes, each one based on a human-comprehensible concept, and organize them in an arbitrary hierarchy. Each ontology node has an associated regular expression pattern which is used to retrieve all relevant documents of the correspond- ing concept. A concept such as Internet may have a pattern such as binternet 1 b bweb b bwww b, which consists of several subpatterns separated by . Not all of these patterns must be single words as in our example – multiword patterns also appear frequently, at least in the research ontology which was provided by uma. Notice also that b is used to denote word boundaries – in our example, the subpatterns match whole words only. In this paper, we investigate characterizing concepts by a short list of appropriate terms, i.e. single words. We consider a variety of term utility functions, each of which map every tuple (term, concept) to a numeric value which signifies the usefulness of the respective term to decide if documents are part of the respective concept or not. Such a characterization by terms could also suggest new single word patterns to the user during creation of a new ontology node, if real-time performance is achievable. We will first give an overview about the research ontology, March15, which was provided by uma, focussing on common data statistics. Afterwards, we will introduce the term utility functions used throughout our experiments. These are more concisely referred to as measures. Then we will shortly describe the experimental setup, discuss major experimental results in the Results section and discuss minor experimental results and other issues in the Discussion section. At last, we will conclude this paper. 1 i.e. the term internet. 1