Performance Evaluation of a Context Provisioning Middleware Eike Steffen Reetz *† , Michael Knappmeyer *† , Saad Liaquat Kiani * , Nigel Baker * and Ralf T¨ onjes † * Faculty of Engineering and Technology, University of the West of England, Bristol, UK † Faculty of Engineering and Computer Science, University of Applied Sciences Osnabr¨ uck, Osnabr¨ uck, Germany Abstract—The evaluation of context middleware systems is a challenging endeavour. Testbed investigations suffer from an unrealistic environment in terms of number of users, high implementation effort for changes and questionable portability of results. Simulation of middleware systems is complex due to the high abstraction of implementation. This paper contributes towards the understanding of a broker based context provisioning system based on black-box measurements of a testbed which are further utilised to increase the accuracy of a simulation model. Both simulations and measurements help in understanding the complex behaviour of a context provisioning middleware and en- able the evaluation of complex distributed systems. The presented investigations identify significant parameters and corresponding models for the response delay of the key components of a context provisioning middleware and discuss their integration into an overall simulation model. Index Terms—context, middleware, evaluation, simulation I. I NTRODUCTION Many facets of context provisioning have been investigated during the past decade. Different approaches in terms of communication paradigms, context modelling and represen- tation, extendibility, entity diversity as well as processing and management scalability have been well addressed from the functional and qualitative point of view. Nevertheless, most of the proposed solutions have not proven their large-scale capabilities either by simulations or prototype deployment. Previous results in [1] have indicated that simulations can aid in proving specific aspects of a context provisioning system but is inadequate just at the functional level. In addition it is required to derive an appropriate model from testbed implementations and assessments. This model includes a functional model of the investigated context provisioning middleware and the related context as well as a performance model of the underlying application server. Many simulation approaches suffer from partial investigation limited to high level abstraction and their inadequate understanding of the whole system. Moreover, prototypes provide only very limited evidence with regard to scalability since creating a realistic environment is almost impossible in terms of heterogeneity, numbers of context sources and sinks, network traffic, etc. If assessed appropriately testbeds can reveal significant system- level behaviour as measurement results include hardware and software constraints outside the logical context provisioning framework as well. Our approach is to overcome the current incomplete understanding of functionally described and eval- uated context provisioning systems by improved simulation models based on black-box tests in a testbed. The measure- ments allow for identifying significant parameters and are utilised in order to build a realistic simulation model. The investigated model will guide us to a deeper understanding of the functional and physical understanding of our proposed broker based context provision system and will also assist in identifying and avoiding performance bottlenecks. Our context provisioning middleware has been implemented using Java Enterprise Edition (JavaEE). Therefore the performance is also under investigation. The rest of the paper is structured as follows. Section II discusses related work in the field of the evaluation of context provisioning systems and also relates to the general evaluation of JavaEE applications as well. Afterwards Section III outlines our proposed broker based context provisioning framework (entitled C-ProMiSE). The evaluation approaches are derived and discussed in Section IV. The simulation models and results are discussed afterwards and Section VII finally concludes the paper. II. RELATED WORK Developments in Java and its enterprise application en- vironment offer many advantages over other programming languages like C/C++ including garbage collection, resource management, principle of portability, programming safety and multi-threading. But portability can reduce performance due to Java Virtual Machine overheads. The management of resources is difficult to estimate and predict since autonomous and adaptive background processes like the garbage collector in- fluence the overall performance significantly. These technolo- gies are applied in development of component-based, multi- layer enterprise applications, which benefit from modular and reusable components in order to handle several application steps automatically. Nevertheless, this results in a complex system which has to be evaluated and understood at different levels: (1) Application Level, i.e. Application design and Application Server; (2) System Level, i.e. Operating System, network/system topology; (3) Hardware Level, i.e. processor, cache, multi-processor efficiency. Multiplicity of different in- fluencing factors make it very complex to estimate the overall performance correctly. The performance of JavaEE based Ap- plication Servers have been studied in several different ways (cp. [2]). The majority of related work focusses on a specific Application Server while other researchers compare different Application Servers with each other [3]. Further efforts have