The CULTURA Evaluation Model: An Approach Responding to the Evaluation Needs of an Innovative Research Environment Christina M. Steiner, Eva-C. Hillemann, Alexander Nussbaumer, Dietrich Albert Knowledge Technologies Institute, Graz University of Technology Inffeldgasse 13/V, 8010 Graz, Austria {christina.steiner, eva.hillemann, alexander.nussbaumer, dietrich.albert}@tugraz.at Mark S. Sweetnam Department of History, Trinity College Dublin Arts Building Dublin 2, Ireland sweetnam@tcd.ie Cormac Hampson, Owen Conlan KDEG, School of Computer Science and Statistics, Trinity College Dublin O'Reilly Institute Dublin 2, Ireland {cormac.hampson, owen.conlan}@cs.tcd.ie ABSTRACT This paper presents the evaluation approach taken for an innovative research environment for digital cultural heritage collections in the CULTURA project. The integration of novel services of information retrieval to support exploration and (re)search of digital artefacts in this research environment, as well as the intended corpus agnosticism and diversity of target users posed additional challenges to evaluation. Starting from a methodology for evaluating digital libraries an evaluation model was established that captures the qualities specific to the objectives of the CULTURA environment, and that builds a common ground for empirical evaluations. A case study illustrates how the model was translated into a concrete evaluation procedure. The obtained outcomes indicate a positive user perception of the CULTURA environment and provide valuable information for further development. Keywords Cultural heritage, research environment, evaluation model, evaluation qualities, empirical study. 1. INTRODUCTION Information retrieval technologies open up a range of possibilities for providing support in exploring, searching, and researching cultural heritage artefacts. Examples are automatic indexing and searching methods [9], normalisation of spelling variations in historical documents [10], extraction of entities representing persons, locations, events etc. from documents [3], the processing and mapping of dates and time intervals [8], or the application of social and influencer network analysis to digital collections. These new technologies and their integration and application in cultural heritage research environments and electronic information services require appropriate evaluation methodologies and thus pose specific requirements and challenges to evaluators. The present paper describes a comprehensive evaluation model accounting for that, which has been developed in the context of the CULTURA project 1 . The project aims at delivering a corpus agnostic research environment integrating a range of innovative 1 http://www.cultura-strep.eu/ services that guide, assist, and empower users’ interaction with cultural heritage artefacts and take into account the diverse needs of different user groups. The novel methodological and technical approaches integrated, as well as the intended reusability of the technology with different collections and the diversity of users addressed as target audience challenges evaluation with respect to the use of sound and suitable methods across contrasting digital collections and diverse communities and users. Through the development of an evaluation model serving as a common ground for different evaluation studies, an appropriate level of comparability and generalizability of evaluation results can be maintained. The evaluation model is presented and a case study conducted with historians is outlined to illustrate how the investigation of the different evaluation axes and qualities of the model provide service-specific insights on the quality of the CULTURA environment and meaningful information for further development. 2. THE CULTURA PROJECT & SYSTEM The interdisciplinary field of digital humanities is concerned with the intersection of computer science, knowledge management and a wide range of humanities disciplines. Recent large-scale digitisation initiatives have made many important cultural heritage collections available online. This makes them accessible to the global research community and interested public for the first time. However, simple "one size fits all" web access is, in many cases, not appropriate in the digital humanities, due to the size and complexity of the artefacts. Furthermore, different types of users need varying levels of support, and every individual user has their own particular interests and priorities. Personalised and adaptive systems are thus important in helping users gain optimum engagement with these new digital humanities assets. Improved quality of access to cultural collections is a key objective of the CULTURA project [7]. Moreover, CULTURA supports a wide spectrum of users, ranging from members of the general public with specific interests, to users who may have a deep engagement with the cultural artefacts, such as professional and trainee researchers. To this end, CULTURA is delivering a corpus agnostic environment, with a suite of services to provide the supports and features required for such a diverse range of users. Proceedings of ENRICH 2013 - SIGIR 2013 Workshop August 1, 2013, Dublin, Ireland 43