eISP: a Programmable Processing Architecture for Smart Phone Image Enhancement Mathieu Thevenin, Laurent Letellier and Renaud Schmit CEA, LIST Embedded Computing Lab Bd 528 MB 94 F-91191 Gif sur Yvette – France Email: FirstName.LastName@cea.fr Barthelemy Heyrman and Michel Paindavoine UMR CNRS 5158 Rue Alain Savary F-21000 Dijon – France Email: FirstName.LastName@u-bourgogne.fr Abstract—Today’s smart phones, with their embedded high- resolution video sensors, require computing capacities that are too high to easily meet stringent silicon area and power consump- tion requirements (some one and a half square millimeters and half a watt) especially when programmable components are used. To develop such capacities, integrators still rely on dedicated low resolution video processing components, whose drawback is low flexibility. With this in mind, our paper presents eISP – a new, fully programmable Embedded Image Signal Processor architecture, now validated in TSMC 65nm technology to achieve a capacity of 16.8 GOPs at 233 MHz, for 1.5 mm 2 of silicon area and a power consumption of 250 mW. Its resulting efficiency (67 MOPs/mW), has made eISP the leading programmable architecture for signal processing, especially for HD 1080p video processing on embedded devices such as smart phone. I. I NTRODUCTION Video sensors, particularly as used in smart phones, have become a familiar part of everyday life. This market is placing drastic constraints on the power consumption and silicon areas of video image processors. Sensors are thus necessarily associ- ated with signal processors, not only for color reconstruction purposes, but also for intrinsic image enhancement. The al- lowable power consumption of embedded image processors is currently a few hundred milliwatts, while smart phone computing capacities exceed many billions of operations per second (GOPs). To afford such capacities, today’s phone integrators rely on dedicated video processing components that offer limited flexibility. With the programmable computing architectures now on the market, High Definition (HD) video processing, which requires several tens of GOPs, cannot be built into mobile devices. Because integrators prefer to embed their own image enhancement functions, it is crucial to make the computing resources behind the currently available sensors as programmable and as flexible as possible. Section two of this paper presents some common color reconstruction and image enhancement methods, followed, in section three, by an estimate of the computing resources necessary to implement them. Section four then describes the proposed Embedded Image Signal Processor (eISP) architecture - a fully programmable architecture designed to handle HD 1080p video streams (i.e. 1920×1080 resolution at 25 frames per second), thereby anticipating the capacities needed for next- generation smart phone video sensors. This discussion of eISP architecture, which is synthesized and routed for TSMC 65 nm technology, likewise details the silicon area, power consumption and computational characteristics of our new concept. II. VIDEO PIPELINE A set of embeddable processes is necessary for the capture and enhancement of video images and photographs produced by Complementary Metal Oxide Semiconductor (CMOS) sensors. Various processing steps and algorithms can be incorporated into the image reconstruction chain, also known as the ”video pipeline”. Because evaluation of sensor exposure parameters is difficult, captured images do not usually cover the complete sensor dynamic range. Histogram normalization [1] or local adaptive methods [2] can, however, be applied to make optimum use of the available dynamic range for pixel value coding. B G V G R R B G G G R R B G B G V G R R B G G G R R B G B G V G R R B G G G R R B G Fig. 1. Standard Bayer color array filter. Noise reduction is an essential processing step. Image signal- to-noise ratio diminishes with a reduction in pixel size, which is also the trend observed in mobile consumer products. Fixed-pattern noise (FPN) is essentially linked to disparities in substrate properties and can be successfully characterized and removed. Electronic noise caused by thermal excitation is an additive white Gaussian noise. Its effect can be limited by applying a Gaussian blur or local adaptive filter. Finally, so-called ”salt and pepper” noise occurs when pixel values are corrupted with respect to their neighborhoods. This type of noise is particularly visible to the human eye. Median filters or techniques enabling removal of extreme pixel values from a given group are best suited to this type of denoising [3].