Lorcan Coyle · Juan Ye · Emerson Loureiro · Stephen Knox · Simon Dobson · Paddy Nixon A Proposed Approach to Evaluate the Accuracy of Tag-based Location Systems Received: June 11 2007 / Accepted: July 3 2007 Abstract Location detection systems that use tags are a popular means of determining a user’s location. These systems are characterised as requiring the user to carry an identity tag that is detected by sensors, which typi- cally use some form of triangulation to determine loca- tion. Although estimates for precision for these systems are published by the respective manufacturers the cus- tomer experience can vary widely. This paper proposes an evaluation framework which will allow different sys- tems to be compared more directly. This framework is specifically targeted at evaluating the experiences of tag- ging humans, which can cause particular difficulties due to the fact that many tag-based systems use communi- cation frequencies that cannot pass easily through the human body. Keywords Evaluation Frameworks · Location-based Systems 1 Introduction Location based systems are becoming more commonly applied to a number of problems in ubiquitous comput- ing and beyond [11,5]. When dealing with these systems it is important to have a gauge of their accuracy. High- tower and Borriello performed an evaluation of location systems and defined the quality of location systems by looking at the percentage of readings that fall within a certain distance of the true value [6]. The distances denote the accuracy, or grain size, of the position in- formation GPS can provide and the percentages denote precision, or how often we can expect to get that ac- curacy. Dobson et al. derive three factors for accuracy This work is partially supported by Science Foundation Ire- land under grant numbers 05/RFP/CMS0062 “Towards a se- mantics of pervasive computing” and 04/RPI/1544 “Secure and predictable pervasive computing” L. Coyle Systems Research Group, School of Computer Science and In- formatics, UCD Dublin, Ireland;E-mail: lorcan.coyle@ucd.ie of location information: precision, which captures inac- curacy due to the resolution of sensor readings; decay, which captures inaccuracy due to the staleness of sensor readings; and confidence, which measures the degree of belief on a sensor reading [2]. We use a combination of these definitions: using Hightower and Borriello’s defi- nition of accuracy and precision to calculate values for Dobson et al’s precision component of location accuracy. When we use the term precision in this paper, we are using Dobson et al’s meaning of the word. This paper describes an evaluation framework for cal- culating the precision of a tag-based location system. Other papers have evaluated location based technology individually, but these evaluations are usually performed by the stakeholders, and the evaluation criteria are not always transparent. Additionally, it is often unclear whe- ther the published precisions are determined from eval- uations using tags alone or from tagged human subjects. Our work focuses on human subjects due to the inherent interference caused by the human body on the communi- cations mechanisms used by the tested location systems. While the examples described in this paper focus on hu- man tagging, the framework itself is agnostic to what is being tagged. The important point to take is that eval- uations using this framework should take the intrinsic physical properties of what is being tagged into account when determining system precision. Section 2 briefly surveys a number of tag-based loca- tion systems that could be evaluated using this frame- work. Section 3 describes the evaluation framework and Section 4 describes an exemplar evaluation that is under- way to test the precision of an installation of a tag-based Location based system. Finally, Section 6 concludes the paper, lays out the hypotheses underpinning our evalu- ation, predicts some results and outlines some ongoing research related to this paper.