Real-time Video Quality Analysis on Mobile Devices Hannes Fassold, Stefanie Wechtitsch, Marcus Thaler, Krzysztof Kozlowski, Werner Bailer JOANNEUM RESEARCH, DIGITAL – Institute for Information and Communication Technologies Steyrergasse 17, 8010 Graz, Austria {firstname.lastname}@joanneum.at ABSTRACT Media capture of live events such as concerts can be im- proved by including user generated content. In this paper we propose methods for visual quality analysis on mobile devices, in order to provide direct feedback to the contribut- ing user about the quality of the captured content and avoid wasting bandwidth and battery for uploading/streaming low- quality content. We focus on quality analysis that comple- ments information we can obtain from other sensors. The proposed methods include real-time algorithms for sharp- ness, noise and over-/underexposure. Categories and Subject Descriptors H.5.1 [Information Interfaces and Presentation]: Mul- timedia Information Systems; I.4.1 [Image Processing and Computer Vision]: Digitization and Image Capture 1. INTRODUCTION The video capabilities of mobile devices have improved tremendously in recent years, reaching a level that only used to be found in mid-range consumer cameras. While the ca- pabilities of sensors as well as the quality of processing and encoding have improved, many user generated videos suffer from quality issues. Some of these problems may be indi- cated by other sensors of the mobile device (e.g., accelerom- eter readings may indicate shakiness), while others can only be identified by inspecting the image content. In this paper we focus on visual quality analysis on mo- bile devices, in order to provide direct feedback to the user about quality problems and avoid wasting bandwidth and battery for uploading/streaming low-quality content. We focus on quality analysis that complements information we can obtain from other sensors, and we do not consider qual- ity problems that may occur due to encoding and transmis- sion later in the processing chain. The proposed methods include real-time algorithms for sharpness, noise and over- /underexposure. The rest of this paper is organized as fol- lows. The remainder of this section discusses the application Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. scenario. Section 2 presents the three quality analysis algo- rithms in detail. In Section 3 we discuss implementation details and present the Android app that has been imple- mented. Section 4 contains evaluation results for the algo- rithms and Section 5 concludes the paper. 2. QUALITY ANALYSIS ALGORITHMS 2.1 Sharpness For the sharpness estimation we apply a Laplace opera- tor which is performed as the sum of the second derivatives of the input image. The Laplacian is computed by filter- ing the image with the following a 3 × 3 kernel. Thus, the algorithm is using the Laplace operator for edge detection. The response values at edges indicate the slope of the corre- sponding edge. The processed image is divided into equally sized blocks, e.g., of size 32 × 32 pixels, and a representative sharpness value is computed for each block by storing the maximum value of the filtered image. According to the HVS, which tends to make its judgment based on the sharpest re- gion in the image [1], we finally determine a sharpness score from a subset of blocks yielding the highest values (typically using 5-25% of the blocks). 2.2 Noise The workflow of the proposed lightweight noise level es- timation algorithm is described in this section. First, the luminance component of the input image is calculated. All further processing is done using only the luminance compo- nent. A Laplacian operator is applied and the filtered image is then divided into equally sized blocks. In the next steps, blocks which are not homogeneous or which correspond to nearly black image areas are discarded. A block is discarded as being not homogeneous if the maximum response within the block is higher than 30% of the maximum response in the whole Laplacian image. For all remaining blocks, a noise level is calculated separately for each block. The per-block noise level value is calculated by applying a median filter of size 3 × 3 to the luminance component, subtracting the median-filtered block from the original block and calculat- ing the average (over the block) of the absolute values of the differences. Finally, an overall noise level value is obtained as the median of all per-block noise level values. 2.3 Over-/Underexposure For the detection of over- and underexposure, we use only the luminance (Y) channel of an image and additionally we scale down the image to its half or quarter size in order to