Adaptive Evaluation of Portal Quality: An eGovernment Case Babis Magoutas, Gregoris Mentzas Information Management Unit National Technical University of Athens (NTUA) Athens, Greece elbabmag@mail.ntua.gr, gmentzas@mail.ntua.gr Abstract— In this paper we describe a knowledge-based approach that enables the subjective evaluation of portal and e-service quality by users in an adaptive manner. The model for adaptive quality measurement (MAQM) comprises different ontologies including concepts regarding quality aspects, questions and questionnaires, portal characteristics and problems encountered by users while using the portal. A prototype adaptive questionnaire system (SALT) which uses the MAQM model is also described. The system applies three axes of adaptation: based on real-time feedback from users through questionnaires, based on problems encountered by the user and based on metadata of the pages visited by the user. The results of evaluating SALT, compared to a static questionnaire, in an Austrian e-Government portal, are indicative of the value added of our approach. Adaptive systems; e-Government; ontology; quality measurement I. INTRODUCTION For every portal, regardless of its mission or objectives, the competitor is only a link away and attracting and keeping users becomes paramount [1]. The quality of portals and their e-services has become an important way for owners to satisfy their users [2], and an important differentiator factor between portals [3]. For example in the e-Government domain, which is the domain we used for applying our approach, the need for quality assessment of e-Government portals and their services is becoming more and more acute, because of the increasing number of public e-services [4] and the existence of manifold problems related to their quality [5]. The measurement of a portal’s and e-services’ quality, forms the basis of an improvement process, since something that cannot be measured cannot be managed and improved [6]. The most usually applied instruments for obtaining user feedback on the overall quality of portals and e-services are web surveys [7]. Although several existing generic questionnaires can be used for that purpose (e.g. QUIS [8] and WAMMI [9]), they suffer from four major limitations. First, they focus on usability issues and thus may not cover all the desirable aspects that must be evaluated (e.g. security and reliability issues). Second, they are considered as ‘quick and dirty’ scales with no demonstrated reliability and validity [10]. This means that they may not give the same results when filled out by likeminded people in similar circumstances and even worse that they may not actually measure or collect data about what they are intended to do. Third and most important, they adopt a “one size fits all” approach for quality evaluation, as they treat all users in a similar manner. This means that all users are asked the same questions, without taking into account their characteristics and the pages they visited. Fourth, they achieve low response rates, especially in case the number of questions asked or the time needed for answering them increase [11]. This limitation is closely related to the third one, as low response rates are partially attributed to the “one size fits all” approach of quality evaluation. In this paper we propose an adaptive process for evaluation of portal quality by users, using an ontological knowledge representation, in an attempt to overcome the aforementioned limitations. Our approach is based on previous work on quality of portals and e-services, which resulted in a comprehensive, validated and reliable quality model [12], [13]. The quality model, which defines in a hierarchical manner the characteristics of portals and e- services that are important for users and drive their satisfaction, addresses the first and second limitations discussed above. The focus of this paper is on the work done in order to overcome the third and fourth limitations. More specifically we use ontologies to represent both the concepts and the concrete information surrounding the adaptive evaluation of portal quality. The ontologies include concepts regarding quality aspects, questions and questionnaires, portal characteristics and problems encountered by users while using the portal. The integration and use of ontologies enables the sharing of conceptual structures and constitutes a sound basis for adapting the questionnaire based on user characteristics, on real-time user feedback trough the questionnaire and on types of visited pages. Adaptive questionnaires have already been applied mainly in the educational domain, as in [14] where adaptive questionnaires/tests for teaching are employed in order to enable the self-training of students in various topics, drawing their attention to the topics they need to study more, or in