Towards Realism in Facial Image Transformation: Results of a Wavelet MRF Method B.P. Tiddeman † , M.R. Stirrat ‡ and D.I. Perrett ‡ Schools of Computer Science † and Psychology ‡ , University of St Andrews, Fife, Scotland, UK. Abstract The ability to transform facial images between groups (e.g. from young to old, or from male to female) has applications in psychological research, police investigations, medicine and entertainment. Current techniques suffer either from a lack of realism due to unrealistic or inappropriate textures in the output images, or a lack of statistical validity, e.g. by using only a single example image for training. This paper describes a new method for improving the realism and effectiveness of facial transformations (e.g. ageing, feminising etc.) of individuals. The method aims to transform low resolution image data using the mean differences between the two groups, but converges on more specific texture features at the finer resolutions. We separate high and low resolution information by transforming the image into a wavelet domain. At each point we calculate a mapping from the original set to the target set based on the probability distributions of the input and output wavelet values. These distributions are estimated from the example images, using the assumption that the distribution depends on the values in a local neighbourhood of the point (the Markov Random Field (MRF) assumption). We use a causal neighbourhood that spans multiple coarser scales of the wavelet pyramid. The distributions are estimated by smoothing the histogram of example values. By increasing the smoothing of the histograms at coarser resolutions we are able to maintain perceived identity across the transforms while producing realistic fine-scale textures. We use perceptual testing to validate the new method, and the results show that it can produce more accurate shifts in perceived age and an increase in realism. Categories and Subject Descriptors (according to ACM CCS): I.4.7 [Image Processing and Computer Vision] Feature Measurement (Texture), I.3.8 [Computer Graphics] Applications. 1. Introduction The ability to alter perceived attributes of a facial image, such as age, race or sex has found application in psychological research for producing controlled experimental stimuli. Other application areas include the ageing of photographs of wanted or missing persons, predicting the outcome of medical intervention (e.g. for skin conditions) and modifying the appearance of actors in the film industry. Previous methods based on group differences have suffered because of a lack of texture in facial blends used to define the transform. Methods to compensate for this, by modifying the amplitude of multi-scale edges (via wavelets), have produced some improvements, but the resulting images still lack realism. They also fail to sufficiently alter the perceived age when rejuvenating and the perceived sex when transforming from male to female. More recent methods based on single images have produced some more realistic results, but do not necessarily define the typical or most likely alterations to a given face, i.e. they could be biased by the choice of example. In this paper we propose a new method based on locally estimated probability distributions. These distributions are conditional on the points in a neighbourhood of the point being synthesised. The estimated conditional distributions for the original and output pixel defines a mapping from original to the new value. Performing the estimation in the wavelet domain offers several advantages over spatial domain processing. It is inherently multi-scale, improving the reconstruction quality for the same size neighbourhood. The low and high resolution information is cleanly separated, so that processing at high resolutions does not overwrite lower resolution information. For this application it also has the advantage that we can alter the transformation parameters with scale, so that we retain the low resolution information (which codes mainly for face shape) but can select more specific textures (e.g. wrinkles and stubble) at higher resolutions. In the remainder of this paper we first review the relevant literature in texture synthesis, image-fusion, facial image prototyping and facial image transformation. We then describe the new method in detail, followed by visual transformation examples for facial ageing and gender change. Finally, we present the results of a perceptual study that demonstrates that the new technique can accurately age and rejuvenate faces to the age of the target group, and offer a significant increase in perceived realism. 2. Literature Review 2.1 Texture synthesis by analysis The main problem in previous facial transformation methods has been identified as the lack of appropriate transformation of the facial textures. Synthesising patches of texture from examples has been the focus of a considerable number of research papers. The statistical nature of textures inspired methods that used global