Test Oracles Based on Metamorphic Relations for Image Processing Applications Tahir Jameel, Mengxiang Lin State key Lab of Software Development Environment Beihang University Beijing, China tahir@nlsde.buaa.edu.cn, linmx@buaa.edu.cn Liu Chao Institute of Software Engineering Beihang University Beijing, China liuchao@buaa.edu.cn Abstract— Testing of image processing applications is a challenging job especially, when evaluating the correctness of output image. Generally, output images are evaluated manually by visual inspection carried out by an expert tester, which is the main hindrance in automation of testing process. Recently, statistical and metamorphic testing approaches are presented to automate output evaluation of image processing applications. The statistical method is dependent on availability of statistical distribution of output images, whereas metamorphic testing require more research efforts to make it widely used in practice. Metamorphic testing is a well-known technique to alleviate the test oracle problem and eliminates the required manual efforts by using relations of input and output images. Follow-up test cases are generated based on these relations and their expected output is evaluated. This paper addresses test oracle problem for image processing applications and demonstrates how properties of implementation under test can be adopted as metamorphic relations. We have studied general and specific metamorphic relations of morphological image operations such as dilation and erosion. Selection of metamorphic relations and their effectiveness by mutation analysis is demonstrated. The results show that metamorphic testing is useful for evaluation of output images in the absence of a perfect test oracle. Keywords— Testing Imaging Applications, Metamorphic Testing, Test Oracle I. INTRODUCTION Image processing applications continue to provide more features, handle larger data and generate complex outputs. Advancement in signal processing and digital imaging devices has increased their extensive use. These applications play a vital role in our daily life such as medical imaging, surveillance, biometrics etc., and human dependence require well tested reliable systems. Testing is a practical approach to evaluate the quality of these systems which includes input generation, test execution and evaluation of output. Generation of test images and evaluation of output images are considered a challenging task due to complex nature of images. Currently, these applications are tested by giving manually crafted test images or commonly used standard test images. The outputs of these image are scrutinized by visual inspection carried out by an expert human tester. This procedure is tedious and sometimes ineffective in a way that a human inspector can only analyze a limited amount of images required for testing. Automatic output image content analysis is required to make the testing procedure large scaled and cost effective. Let function ܨሺݔሻ be an implementation of algorithm ܣሺݔሻ and  be a subset of input space  given by ൌ ሼ ݐ ଵ ǡ ݐ ଶ ǡǥǡ ݐ ௡ ሽ ؿ. Function ܨሺݔሻ is executed to produce output ൌሼܨሺ ݐ ଵ ሻǡ ܨሺ ݐ ଶ ሻǡ ǥ ǡ ܨሺ ݐ ௡ ሻሽ, whereas set of expected output of function ܨሺݔሻ is given by ܧൌ ሼܧሺ ݐ ଵ ሻǡ ܧሺ ݐ ଶ ሻǡ ǥ ǡ ܧሺ ݐ ௡ ሻሽ. Test oracle is a mechanism to determine whether the test has passed or failed. In testing process, the actual output is compared with the expected output to analyze the correctness i.e. ܨሺ ݐ ௜ ሻൌ ܧሺ ݐ ௜ ሻ [9]. In practice, the test oracle is either not available or very expensive to apply, which is known as oracle problem [1, 10]. In case of image processing operations, sometimes test oracle cannot be clearly defined, e.g. comparing two images pixel by pixel may have little difference but visually they are similar. Recently, test oracles such as statistical, metamorphic, pseudo-oracles etc. are proposed for image processing applications where an oracle is not available. Meyer [2] proposed a statistical method to automatically analyze the correctness of output images and used Boolean model to generate tests images. In this technique, a hypothesis is framed, and program under test is executed to generate a number of outputs. The statistical evaluation of output is carried out to test the hypothesis. Based on the output statistics, the hypothesis is accepted or rejected whereas previous error probabilities can also be taken in to account. This approach have some limitations, as it can only be applied if the statistical distribution of the output images is already available. Furthermore, this approach checks for some characteristics of the output images only. In his further work Meyer [3, 4] used metamorphic relations to analyze the correctness of output and to generate additional test cases. Random and Boolean models are used to generate test images and metamorphic relations are used to analyze the correctness of output. The strategy used to evaluate output is independent of input generation. Meyer proposed some general as well as algorithm specific metamorphic relations for image processing applications, and also demonstrated their effectiveness using mutation analysis. Statistical metamorphic testing [5] was proposed to augment the limitation of statistical hypothesis tests. The input relation 978-1-4799-8676-7/15/$31.00 copyright 2015 IEEE SNPD 2015, June 1-3 2015, Takamatsu, Japan