HandBench: A Benchmarking Suite for Processors Embedded in Handheld Devices Pepijn de Langen Ben Juurlink Stamatis Vassiliadis Computer Engineering Laboratory Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology Mekelweg 4 2628 CD Delft The Netherlands phone: +31-15-278-3644 email:{pepijn|benj|stamatis}@ce.et.tudelft.nl Abstract — Most microprocessors are employed in em- bedded systems. Since many of these systems are powered by batteries, energy consumption has become an important design aspect. In order to quantify the performance and energy consumption of different architectures, a collection of benchmarks is required that closely represents the appli- cations found on these devices. In this paper, we present a benchmarking suite called HandBench that consists of benchmarks which can be considered typical workloads of contemporary and emerging handheld devices. Existing benchmark suites such as MediaBench [4] and MiBench [3] are inappropriate for our goal for the following reasons. First, these suites contain a number of applications that are, in general, not found on battery-powered devices (e.g., applications from the automotive domain). Also, we in- cluded workloads that where not yet present in either of these collections. Furthermore, for a number of applica- tions that are contained in these suites, we have found that the input datasets are not representative of those used in real systems (e.g., using an incorrect sampling-rate). keywords: benchmarking, handheld, embedded processors I. Introduction The majority of manufactured processors are used in embedded systems. As technology increases, these processors are becoming increasingly important. The number of cellular phones, digital camera’s, and sim- ilar equipment sold each year is still rising, as is the demand that is put on these devices. Besides the need for computational power, there is another important design aspect for these devices. Being powered by a battery, the amount of available energy is limited. This battery lifetime is also very important to users, since the device becomes useless if the battery runs out. The choice between energy and performance is often not an easy one. In some cases, however, it is clear that increased performance is only useful if it can be achieved with an equal amount of energy usage or less. An example of this is that any device being able to decode a DVD, should also be able to perform this for at least the duration of a normal movie. In order to measure the performance and energy consumption of different embedded processors, a num- ber of representative applications is required, often called benchmarks or workloads. II. Related Work Currently, there exists a number of different bench- marking suites, all targeted at a certain domain. The benchmarking suites by SPEC [1] (Standard Per- formance and Evaluation Corporation) are probably most well-known. Several suites have been published by SPEC, like SPECint and SPECfp, and more re- cently also more specific ones, for example SPEC- mail2001 for benchmarking mail servers. None of these, however, was useful for research in handheld embedded systems. Lee et.al. introduced the Media- Bench [4] suite, usefull for systems that run applica- tions from the multimedia domain. Many of the hand- held devices we consider run applications from this do- main. However, besides the lack of typical handheld applications from other domains, also the formats and sizes of the inputs and outputs of the workloads were not always representative. In 1997, the Embedded Microprocessor Benchmark Consortium [6] (EEMBC) was formed to develop benchmarks for the hardware and software used in embedded systems. The goal of the EEMBC, however, is to deliver benchmarks and certified scores to processor manufacturers. In order to get access to the benchmarks, a high fee must be payed to join the consortium. Guthaus et.al. pro- posed MiBench [3], a free benchmarking suite for em- bedded systems. This suite makes an effort to provide 57