Colour-difference based demosaicked image postprocessing R. Lukac, K. Martin and K.N. Plataniotis An efficient postprocessing method capable of reducing visual arti- facts introduced during the colour filter array interpolation process is presented. The method produces excellent results in terms of both objective and subjective image quality measures. Introduction: Colour filter array (CFA) interpolation or demosaicking is an essential step in single-sensor imaging solutions. The Bayer pattern [1], the most popular CFA solution, is commonly used in image-enabled wireless phones, pocket devices and visual sensors for surveillance and automotive applications. Since only a single colour component is available at each spatial position of the CFA input, the restored colour image output is obtained by interpolating the missing colour components from the spatially adjacent CFA data. Most of the available demosaicking designs introduce visual artifacts in the form of blurred edges and false colours [2–4]. Therefore, demosaicked colour image postprocessing (DCIP) techniques, imple- mented either directly at the hardware level or as additive software module should be used to reduce visual impairments introduced during CFA interpolation. Proposed postprocessing: Since there is no method to objectively determine which colour component is inaccurate, the introduced DCIP framework utilises the differences between the interpolated colour components and the original Bayer (CFA) data included in the restored colour output to complete the task. Let x(i): Z 2 ! Z 3 be a K 1  K 2 red–green–blue (RGB) colour image representing a two-dimensional matrix of the restored, three-component colour samples x( p, q) ¼ (x ( p,q)1 , x ( p,q)2 , x ( p,q)3 ) 2 Z 2 . Assuming that p ¼ 1, 2, ... , K 1 and q ¼ 1, 2, ... , K 2 denote the spatial position of a pixel in vertical (image rows) and horizontal (image columns) direc- tions, the original Bayer data included the red (R) components x ( p,q)1 located at (odd p, even q), the blue (B) components x ( p,q)3 located at (even p, odd q) and the green (G) components x ( p,q)2 located at (odd p, odd q) and (even p, even q). Employing the spectral correlation between G and R (or B) compo- nents of the restored, full colour image x(i) into the DCIP process further improves the colour appearance in the rest of x(i). Based on the colour correlation model of [5], the proposed DCIP scheme re-evaluates the interpolated G component x (p,q)2 as follows: x ð p;qÞ2 ¼ x ð p;qÞk þ P 4 i¼1 w 0 i ðx ðiÞ2  x ðiÞk Þ ð1Þ where x (1)k , x (2)k , x (3)k , x (4)k correspond to x ( p1,q)k , x ( p,q1)k , x ( p,qþ1)k , x ( pþ1,q)k . These samples denote the interpolated (inaccurate) R compo- nents corresponding to (odd p, even q) and k ¼ 1, or B components corresponding to (even p, odd q) and k ¼ 3, respectively. The notation x (1)2 , x (2)2 , x (3)2 , x (4)2 is used here to indicate the original, Bayer G components x ( p1,q)2 , x ( p,q1)2 , x ( p,qþ1)2 , x ( pþ1,q)2 . The original R (or B) component x ( p,q)k is used to normalise the output value to the desired intensity range. The contribution of each input x (i)2 , for i ¼ 1, 2, ... , 4 in (1), is regulated by the normalised weight w i 0 ¼ w i = P j¼1 4 w j , where the edge- sensing coefficient w i ¼ 1=(1 þ d i ) is defined via d i ¼ P 4 j¼1 jx ðiÞ2  x ð jÞ2 j ð2Þ When no edge is positioned across the directions considered by d i , w j ! 1 and x (i)2 contributes significantly to x (p,q)2 . If x (i)2 deviates from its neighbours, d i decreases w i (i.e. w i ! 0 for d i !1), appropriately penalising x (i)2 . For the R (or B) components located at the ( p, q) co-ordinates of the restored, full colour image which correspond to the original Bayer B (or R) components, the re-evaluation is performed as follows: x ð p;qÞk ¼ x ð p;qÞ2 þ P 4 i¼1 w 0 i ðx ðiÞk  x ðiÞ2 Þ ð3Þ where x (1)2 , x (2)2 , x (3)2 , x (4)2 denote previously re-evaluated (improved) G components x ( p1,q1)2 , x ( p1,qþ1)2 , x ( pþ1,q1)2 , x ( pþ1,qþ1)2 . The re-evaluated G component x ( p,q)2 is used to normalise the DCIP scheme output. The samples x (1)k , x (2)k , x (3)k , x (4)k correspond to x ( p1,q1)k , x ( p1,qþ1)k , x ( pþ1,q1)k , x ( pþ1,qþ1)k . For (odd p, even q) and k ¼ 3, or (even p, odd q) and k ¼ 1; the inputs x ( p1,q1)k , x ( p1,qþ1)k , x ( pþ1,q1)k , x ( pþ1,qþ1)k denote the original B or R components, respec- tively. Each x (i)k , for i ¼ 1, 2, ... , 4, is associated with w i 0 defined via d i ¼ P 4 j¼1 jx ðiÞk  x ð jÞk j ð4Þ For the R and B components located at spatial position ( p, q) in the restored, full colour output corresponding to an original G component in the Bayer pattern, the re-evaluation is performed by repeating the DCIP process of (3). In this case x (1)k , x (2)k , x (3)k , x (4)k denote the mixed (original and re-evaluated) R (for k ¼ 1) or B (for k ¼ 3) components x ( p1,q)k , x ( p,q1)k , x ( p,qþ1)k , x ( pþ1,q)k . These components are surrounding the normalising original G component x ( p,q)2 located at spatial positions (odd p, odd q) and (even p, even q). x (1)2 , x (2)2 , x (3)2 , x (4)2 denote the re-evaluated G components x ( p1,q)2 , x ( p,q1)2 , x ( p,qþ1)2 , x ( pþ1,q)2 . The weight coefficients of (3) are defined using (4) with the inputs x (1)k , x (2)k , x (3)k , x (4)k equivalent to x ( p1,q)k , x ( p,q1)k , x ( p,qþ1)k , x ( pþ1,q)k . Results: A number of colour images have been used to evaluate the proposed DCIP framework. Examples are shown in Fig. 1. All images have been normalised to the standard 512  512, 8-bit per channel RGB representation. The efficiency of the DCIP method is measured, objectively, via the mean absolute error (MAE), the mean square error (MSE) and the normalised colour difference (NCD) measure. Follow- ing standard evaluation practices, the original colour images are transformed into Bayer CFA images. Since in this Letter the emphasis is on postprocessing, the interpolated colour output are obtained using popular methods, such as the nearest-neighbour interpolation (NNI) [2], the bilinear CFA interpolation (BI) [3] and the high-definition colour (HDC) CFA interpolation scheme [4]. Objective evaluation results of the image quality obtained with and without the DCIP module are summarised in Table 1. As can be seen, the proposed DCIP scheme significantly improves the performance of the referred demosaicking approaches and provides the best results in terms of MAE, MSE and NCD objective criteria. a b Fig. 1 Test images a Face b Butterfly Table 1: Objective comparison of performance Image Face Butterfly Method MAE MSE NCD MAE MSE NCD NNI 4.342 96.6 0.1029 4.452 172.2 0.0727 BI 3.066 42.6 0.0697 2.797 59.4 0.0435 HDC 2.543 34.5 0.0612 1.875 19.8 0.0313 NNI þ DCIP 1.986 18.0 0.0510 2.063 20.8 0.0363 BI þ DCIP 1.626 11.6 0.0418 1.483 9.4 0.0262 HDC þ DCIP 1.746 14.3 0.0470 1.565 10.4 0.0272 Corresponding zoomed parts of the obtained results are shown in Figs. 2 and 3 and they can be used for visual (subjective) evaluation. The best visual quality is obtained by applying the DCIP method to the demosaicked colour images. The DCIP outputs correspond to naturally ELECTRONICS LETTERS 11th December 2003 Vol. 39 No. 25