Process-based evaluation and comparison of OTS software alternatives Maria Jesus Faundes, Hernan Astudillo and Bernhard Hitpass Universidad T´ ecnica Federico Santa Mar´ ıa, Departamento de Inform´ atica, Avenida Vicu˜ na Mackena 3939, San Joaqu´ ın, Santiago, Chile Email: mariajesus.faundes@usm.cl, hernan@inf.utfsm.cl, bernhard.hitpass@usm.cl Abstract—Many Off-The-Shelf Software (OTSS) assessment techniques have been proposed, most of them using criteria related to standard quality models. However, these techniques are not as useful to evaluate and compare alternative OTSS as solutions to specific process-driven organizational changes. This article proposes PBEC-OTSS (Process-Based Evaluation and Comparison of OTSS), a technique for evaluating and comparing OTSS regarding impact in the organization, based on process models, and using fuzzy decision making systems. The technique was compared with an Ad-Hoc approach (systemized from the literature) in an experimental study with IT professionals, some new to BPM and some experts; the experts obtained similarly good results with either approach, but the novice professionals obtained better results with PBEC-OTSS than with Ad-Hoc. These results suggest that organizations can improve their Business/IT alignment with this technique even if no process experts are available. I. I NTRODUCTION T HE software selection process is a subjective and un- certain decision process [23], where meeting the specific needs of the organization is complex and requires time [8], considering that a company may lose its competitive advantage by investing in wrong alternative technologies or by investing too much time in selecting the right one [18]. In practice, there are numerous organizations that lack a rigorous selection process [11], which is often made under pressure by evaluators who may not have time or the expertise to plan it [8], or that select according to their experience or intuition [20], [21]. A systematic and repeatable selection methodology for Off- The-Shelf Software (OTSS), is a crucial need to minimize uncertainty and risk in companies [19]; so, choosing a suitable OTSS, is a non-trivial task for organizations and requires a thorough assessment process. Therefore, the key question seems to remain: how to identify the most appropriate OTSS to meet organizational needs? The recommendation to managers is to choose an appropriate IT infrastructure to facilitate the alignment between strategy and organizational structure, achieving higher performance levels [4]. However, the benefits of an organization’s systems are generally known only after some period of use [2]. In the literature, there are studies that emphasize the impor- tant relationship between business processes and IT, stating: IT will be used if, and only if, the functions available for the user, support, or fit to their activities [5]. The selection and implementation of proper IT applications, is an important pre- condition for the efficient execution of business processes [17]. IT will only have a positive effect in the organizational performance, if this fits with the business processes [9]. Therefore, based on the close relationship between business processes and IT, is that this study proposes a new approach to evaluate and compare OTSS, based on processes models and fuzzy decision making systems, which allows: • Generate alternative reconfigurations of processes mod- els, which serve as input data for processes improvement and processes standardization. • Identify and measure the potential contribution of the OTSS to the organization, through impact indicators (coverage, automation, and implementation). • Generate OTSS traceability regarding the processes and activities of the organization. • Identify collaboration between OTSS, and implemented systems in the organization. All of which improves decision making, and promotes the Business/IT alignment. The remainder of this paper is structured as follows: section II presents the Related Work; in Section III the proposed approach, along with an illustrative example; in section IV the Case Study, and finally Section V summarizes and concludes. II. RELATED WORK In literature, the software assessment has been a subject of interest of suppliers, and topic of study for academics and professionals, developing and proposing: • Preliminary proposals, such as: general recommendations for evaluation of commercial software [3], generation of Domain-specific quality model to assess software [7], and approach for determining the software selection strat- egy [25]. • Proposals for the evaluation of specific types of software, such as: ERP [6], data warehouse system [13], and workflow type software [14], among others. • Frameworks, methods, and tools of assessment and/or selection of OTSS (see Table I). However, in most of these proposals, criteria are repeated or are similar, and related to standard quality models. Unfortunately, criteria associated to software quality evaluate the product and its interaction with the user, but ignore the importance Preprints of the Federated Conference on Computer Science and Information Systems pp. 1115–1122 c 2013 1115