Quality Assessment of Multiobjective Optimisation Algorithms in Component Deployment Aldeida Aleti Faculty of ICT, Swinburne University of Technology Hawthorn, VIC 3122, Australia aaleti@swin.edu.au ABSTRACT Measuring the quality of the approximate sets in a quantita- tive way is important to asses the performance of multiob- jective optimisation algorithms and decide which algorithm performs best in a problem domain. In the case of com- ponent deployment optimisation of automotive systems, de- spite the wide range of optimisation methods already pub- lished, it is still unknown which algorithm is the optimal choice. Several studies can be found in the literature that address the problem of comparing approximate sets in a quantitative manner, reflecting a specific feature of the opti- misation method, i.e. either convergence or diversity. How- ever, both convergence and diversity are important quality aspects and both should be considered to define dominance relations. The aim of this study is a new quality assessment method for approximate sets, which will indicate dominance relations based on both convergence and diversity. Categories and Subject Descriptors D.2 [Software]: Software Engineering General Terms Performance 1. INTRODUCTION The quality of embedded systems in the automotive in- dustry is highly dependent on the design decisions that map software components to hardware hosts [4]. Since this prob- lem is known to be NP hard, approximate methods should be employed. Moreover, the component deployment prob- lem often involves simultaneous optimisation of several com- peting quality objectives and constraints. The solution to such problems is usually a set of design alternatives, which assures a tradeoff between the conflicting qualities, referred to as Pareto front. The employment of approximate meth- ods for multiobjective optimisation yields approximations of the Pareto front, known as approximate sets. Many approximate optimisation methods exist in the lit- erature for the component deployment optimisation prob- lem [4, 5]. While researchers generally agree on using ap- proximate methods for component deployment optimisation, the performance of these methods is not considered. As Copyright is held by the author/owner(s). ESEC-FSE Doctoral Symposium’09, Aug. 25, 2009, Amsterdam, The Netherlands. ACM 978-1-60558-731-8/09/08. Blum [2] states, the performance of an approximate opti- misation method is highly related to the context in which it is being used. Accordingly, knowledge about the perfor- mance would not only enable monitoring and improving the optimisation method but also it would provide means for comparing and contrasting the performance of different op- timization methods in a specific problem domain. Unfortunately, measuring the performance of multiobjec- tive optimisation methods is a non-trivial task. Zitzler et al. [10] suggest three aspects to define appropriate quality- measures for approximate sets: (i) The distance of the re- sulting approximate set to the Pareto front should be mini- mized, (ii) A good, i.e uniform, distribution of the solutions should be found, and (iii) The extent of the obtained ap- proximate set should be maximized. The first one deals with the quality of the solutions, known as convergence property. The latter two deal with the distribution of solutions, which is known as diversity property. To measure the quality of approximate sets, various quality metrics exist in the litera- ture [12]. However, it is not very clear what the advantages and disadvantages of these quality measures are, and in what way they relate to each other [12]. Moreover, despite of the variety of the quality metrics, none of them indicates how good an approximate set is with respect to both the desired features, i.e. diversity and convergence properties. As Laumanns et al. [7] points out, the diversity of so- lutions is as important as the convergence to the Pareto front. Accordingly, the main goal of component deployment optimisation problem is to find an approximate set that is as close as possible to the Pareto front and covers a wide range of diverse solutions. This study aims to develop a new method to measure the quality of approximate sets which will indicate both the convergence of the algorithm and the preservation of diversity of the solutions. 2. RELATED WORK Optimisation Methods: Considerable research has been done to help the designers find an optimal deployment of software components on the hardware platform [3]. Typi- cal representatives of optimisation algorithms in the compo- nent deployment optimisation problem are Evolutionary Al- gorithms [4], Simulated Annealing [5], and Tabu Search [6]. The main problems with these algorithms are the time com- plexity for the convergence to the Pareto front and their proneness to confinement in local optima. Because of their probabilistic nature, approximate opti- misation algorithms behave differently. Some of these algo- rithms can only generate a part of the solutions in the real