What Makes a Good Pervasive Middleware? Fahim Kawsar, Kaori Fujinami, Tatsuo Nakajima Department of Computer Science, Waseda University, Japan {fahim,fujinami,tatsuo}@dcl.info.waseda.ac.jp ABSTRACT We present 3D Evaluation Model for pervasive middlewares with dimensions of Pervasive Environment specific, Quantitative and Qualitative metrics. We expect our approach is viable to evaluate pervasive middleware since it attempts to cover diverse aspects of ubiquitous computing. In this short paper, we have briefly introduced our 3D Evaluation Model and put forth this issue for open discussion in the poster session. Keywords Pervasive, Middleware, Evaluation, Context Awareness. 1. INTRODUCTION Perhaps one of the difficult tasks of the pervasive middleware developers is to evaluate the middleware. This is because of two reasons: First, the standard computing system evaluation methodology is not applicable to pervasive middlewares considering the diverse feature attributes of ubicomp inherited from different domains with different evaluation schemes. Second, there is no standard guideline for evaluation. Several attempts have been made so far by the research community to guide the evaluation phase of ubicomp systems. But unfortunately most of those guidelines are targeted to the ubiquitous applications. So, it is not clear what approaches are important to evaluate the system that provides the base support for the development of these applications. In this paper, we have targeted this particular aspect and present a 3D Evaluation Model for pervasive middleware with dimensions of Pervasive Environment specific, Quantitative and Qualitative metrics. Each of these metrics attempts to evaluate specific aspects of ubiquitous middleware. In the next section, a snapshot of the efforts done on the evaluation schemes for proactive applications is presented. Then we argue why these approaches fail to guide the middleware developers. Finally we present our 3D Evaluation Model with a peek into its immediate implication. 2. EVALUATION AND UBICOMP One and a half decades ago Mark Weiser emphasized that real prototyping is one of the better alternatives for the evaluation of the ubiquitous systems [9]. Later Abowd and Mynatt [1] rephrased that real prototyping, as “Living Laboratory” is the appropriate evaluation approach for ubiquitous systems. Bellotti and her colleagues suggested five interaction challenges that the ubiquitous system designer should follow [2]. These are: Address, Attention, Action, Alignment and Accident. Mainly she suggested these points to assist interaction designers to develop and evaluate systems that are not GUI based. Jameson proposed five usability challenges for adaptive systems: predictability and transparency, controllability, unobtrusiveness, privacy and breadth of experience [4]. From social aspect, Friedman et. al. suggested 12 key human values with ethical importance [3]: human welfare, ownership and property, universal usability, trust, autonomy, informed consent, accountability, identity, calmness and environmental sustainability. These key points basically cover all social aspects of human computer interaction and can be equally applied to evaluate ubiquitous systems from social aspect point of view. The outcome of the workshop “Evaluation Methodologies for Ubiquitous Computing” at ubicomp 2001 conference formulated a 4 axes framework: Universality, Utility, Usability and Ubiquity [8]. Another interesting proposition is Ubiquitous Evaluation Areas (UEA) proposed by Jean Scholtz et. al. [7]. Under UEA umbrella, they include the following areas for ubiquitous system evaluation: Attention, Adoption, Trust, Conceptual Model, Interaction, Invisibility, Impact and Side Effects, Appeal and Application Robustness. Considering all these propositions, it is understandable that key focus of ubiquitous system evaluation is on the user end rather than on the systems. This is in contrast to other computer system research where we explicitly focus on system performance by some benchmark tools. However, middleware is not the application instead it is a kind of tool that assists the application development. So setting up a stage story for real life prototyping is not strictly applicable to evaluate the performance of the middleware. It is true that all these key points in UEA and others propositions are recurring in nature in applications, but these are mostly end application designers responsibility where they have to consider the scenario in hand and how these requirements can be satisfied while providing the actual service. Strictly some of these requirements are not middleware specific, like: universality, ubiquity, attention or usability. So, middlewares’ responsibility could be assisting application developers to provide all system related support so that developers can focus on the application level challenges considering user level metrics for pervasive applications. 3. 3D EVALUATION MODEL In typical system middlewares, there are specific measures for evaluation; commonly we term that “Quality of Service (QoS)”. Usually several quantitative and qualitative metrics are considered to evaluate the performance of the system. In ubiquitous systems, in addition to these measures we have to satisfy some human factor issues that are equally important as quantitative and qualitative metrics. Theses issues are recurring in nature, so support provision of these facilities is a requirement of ubiquitous middleware. Taking into account these factors, we propose a 3D Evaluation Model for evaluating ubiquitous middleware as shown in figure 1. Figure 1: 3D Evaluation Model for Pervasive Middleware 3.1 Pervasive Environment Evaluation This dimension is important because the goal of pervasive middleware is to support proactive application development. In proactive applications, many features are related to environment and interaction and are not common to typical computing system. The performance of these features contributes to the overall acceptability of the applications. So, supporting these recurring features is an inherent requirement for the quality of pervasive middlewares. Considering this, we have included following metrics in this evaluation dimension: 1. Interaction: The middleware should have support for recurring interaction mechanism support in a way not perceivable by the end users. For example, sensors are used extensively in pervasive applications, so middleware should provide a common base for supporting sensor based user interaction while minimizing application developers’ tasks. This interaction