Context-aware joint dictionary learning for color image demosaicking q,qq Kai-Lung Hua a , Shintami Chusnul Hidayati a , Fang-Lin He a , Chia-Po Wei b , Yu-Chiang Frank Wang b,⇑ a Dept. of CSIE, National Taiwan University of Science and Technology, Taipei, Taiwan b Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan article info Article history: Received 17 October 2014 Revised 23 October 2015 Accepted 1 March 2016 Available online 9 March 2016 Keywords: Color demosaicking Dictionary learning Self-learning Sparse representation abstract Most digital cameras are overlaid with color filter arrays (CFA) on their electronic sensors, and thus only one particular color value would be captured at every pixel location. When producing the output image, one needs to recover the full color image from such incomplete color samples, and this process is known as demosaicking. In this paper, we propose a novel context-constrained demosaicking algorithm via sparse-representation based joint dictionary learning. Given a single mosaicked image with incomplete color samples, we perform color and texture constrained image segmentation and learn a dictionary with different context categories. A joint sparse representation is employed on different image components for predicting the missing color information in the resulting high-resolution image. During the dictionary learning and sparse coding processes, we advocate a locality constraint in our algorithm, which allows us to locate most relevant image data and thus achieve improved demosaicking performance. Experimental results show that the proposed method outperforms several existing or state-of-the-art techniques in terms of both subjective and objective evaluations. Ó 2016 Elsevier Inc. All rights reserved. 1. Introduction A digital color image is made up of pixels, which consist of three independent primary color components: red, green, and blue. In order to reduce costs like hardware size, cost, or power consump- tion, most digital cameras capture one color component at each pixel location by a single monochromatic image sensor (CCD or CMOS), which is overlaid with a color filter array (CFA). As shown in Fig. 1, the most commonly used CFA is the Bayer pattern [1], which places green color pixels in a quincunx lattice and red/blue color pixels in rectangular ones. To obtain the full resolution color image from such CFA samples, two unknown color components need to be estimated from neighboring pixels. Since the CFA sam- ples carry mosaic patterns for different colors, this color restora- tion process has been widely known as demosaicking. To address the problem of demosaicking problems and conse- quently many approaches have been proposed, interpolation- based methods such as bilinear, bicubic, and spline interpolation were initially proposed. However, this type of approaches yield severe artifacts like false color information or zippering, especially along the edges or highly textural regions due to the smooth tran- sition. To reduce such artifacts, recent works suggest edge-directed interpolation and/or the joint exploitation of both intra and inter-channel dependencies during the interpolation process for demosaicking [2–9]. Nevertheless, due to the abundance and diversity of natural images, it is still very challenging to solve the image demosaicking problem through direct analytical analysis or construction of image processing models. Lately, researchers further advance machine learning techniques for tackling this problem. For exam- ple, Zhang et al. [10] considered the calculation of intra and inter-band interpolation, and applied minimum mean-square error (LMMSE) and support vector regression (SVR) to determine the weighting factors for the above interpolation for producing the final demosaicked output. Mairal et al. [11,12] proposed non- local sparse models for image restoration. Their proposed method is based on the idea of jointly decomposing groups of similar sig- nals on subsets of the learned dictionary. Wu et al. [13] proposed a sparsity-based method by introducing the sparse representations for both intra and inter-channels. They derived such representa- tions via an ‘ 1 minimization formulation, and thus the observed model can be applied for demosaicking. http://dx.doi.org/10.1016/j.jvcir.2016.03.004 1047-3203/Ó 2016 Elsevier Inc. All rights reserved. q This paper has been recommended for acceptance by O. Au. qq This work was supported in part by Ministry of Science and Technology of Taiwan via MOST104-2221-E-011-091-MY2, MOST103-2221-E-011-105, and MOST103-2221-E-001-021-MY2. ⇑ Corresponding author. E-mail address: ycwang@citi.sinica.edu.tw (Y.-C.F Wang). J. Vis. Commun. Image R. 38 (2016) 230–245 Contents lists available at ScienceDirect J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci