A Benchmark Evaluation of the French MeSH Indexing Systems Aurélie Névéol 1,2 , Vincent Mary 3 , Arnaud Gaudinat 4 , Alexandrina Rogozan 1 , and Stefan J. Darmoni 1,2 1 PSI Laboratory - FRE 2645 CNRS - INSA de Rouen, France {aneveol ,arogozan}@insa-rouen.fr} 2 CISMeF & L@stics - Rouen University Hospital and Rouen Medical School, France stefan.darmoni@chu-rouen.fr http://www.cismef.org 3 Rennes Medical School, France vincent.mary@univ-rennes1.fr 4 HON Foundation, Geneva, Switzerland Arnaud.Gaudinat@healthonthenet.org Abstract. The increasing number of health documents available in electronic form, and the demand on both practitioners and librarians to encode these documents with controlled vocabularies calls for automatic tools and methods to help them perform this task efficiently. This paper presents the Benchmark evaluation of the French MeSH indexing systems carried out under the umbrella of the VUMeF consortium. The CISMeF, NOMINDEX and HONMeSHMapper systems are introduced, and evaluated on a set of 82 resources randomly taken from the CISMeF catalogue. The automatic MeSH indexing produced by each system was compared to the manual gold standard provided by the CISMeF medical librarian team. The automatic systems achieve at best a precision close to 50% at rank 1 (HONMeSHMapper, CISMeF) and HONMeSHMapper achieves the best overall F-measure. A qualitative evaluation of the indexing provided for a sample resource indicates that all systems tend to misevaluate the specificity of the terms to retrieve. 1 Introduction Internet has become a very prosperous source of information in numerous fields, including health and molecular biology. Several projects have been initiated in order to meet the users' need to find precisely what they are looking for among the numerous documents related to these fields available online. Among them, the Health On the Net foundation (HON 1 ) aims at guiding both lay and specialist audiences to trustworthy medical information in various European languages. HON has developed automatic search engines to crawl and index the web, and an accreditation system based on their HONcode principles. Some 4,600 websites are currently accredited and annually reviewed. All the pages belonging to these sites are indexed. Specialised search engines developed for the medical field can now supply trustworthiness 1 http://www.hon.ch/ (accessed on February 1 st , 2005)