Automatic fine-grained area detection for thin client systems Bert Vankeirsbilck a,n , Dieter Verslype a , Nicolas Staelens a , Pieter Simoens a,b , Chris Develder a , Bart Dhoedt a , Filip De Turck a , Piet Demeester a a Ghent University, Department of Information Technology (INTEC), IBBT, Gaston Crommenlaan 8, bus 201, 9050 Gent, Belgium b Ghent University College, Department INWE, Valentyn Vaerwyckweg 1, 9000 Gent, Belgium article info Article history: Received 1 September 2011 Received in revised form 11 February 2012 Accepted 14 March 2012 Available online 23 March 2012 Keywords: Thin client Area detection Encoding Differentiation Content classification abstract The widespread availability of cloud infrastructures is fueling new interest in the thin client computing paradigm. However, current thin client protocols are not designed to handle new content types as often encountered in state-of-the-art applications (e.g. multimedia editing, gaming, multimedia playback). Conveying this content using traditional thin client protocols typically results in a combination of excessive resource usage and low visual quality. In this paper, we propose an approach where the content type can vary for different portions of the screen (e.g. combination of static text and video). Once the different content types have been detected, each of them can be encoded using the most appropriate algorithm. We present two algorithms for this runtime detection. The first algorithm is operating at the pixel level, thereby being independent of the actual thin client protocol used. The second algorithm assumes the presence of a rectangle-based thin client protocol (such as the popular VNC protocol), trading independence for improved performance. The appropriate parameter settings for these algorithms are experimentally determined. Furthermore, their influence is studied in detail in terms of detection accuracy, and the time to perform the algorithms is analysed. Accurate hints are derived within less than 10 ms, indicating the high potential of this approach for use in next generation thin client systems. & 2012 Elsevier Ltd. All rights reserved. 1. Introduction The thin client computing paradigm consists of off-loading as many resource intensive operations as possible to a remote server and limits the client functionality to forwarding user events to the server and displaying the returned graphical content. The server handles complex computations and remote storage. The benefits from this approach are numerous and well-known (Petrovic and Fertalj, 2009), including decreased exploitation cost, prevention of data loss, shared hardware resources, easier data exchange, universal access of applications, easier software update manage- ment and maintenance, etc. Clearly, the currently witnessed increase in cloud deployment is a strong enabler to leverage the functionalities and possibilities of the constrained devices such as tablets and smartphones, used to access these cloud infrastruc- tures. Hence, a cloud architecture based on thin client computing can provide access to data and applications from any place, at any time with any device and most importantly with unhampered user experience. Given the current network connectivity and thin client proto- cols to convey information between client and server, the delivery of dynamic visual content such as games and video is often an issue. Thin client protocols are typically optimized for delivering static content, and adopting the same approach for delivering dynamic content results in excessive bandwidth usage and mediocre visual experience. An approach frequently adopted in commercial thin client protocols consists of redirecting multi- media content directly to the client device for local decoding. The concept is used in Microsoft Remote Desktop Protocol (RDP) version 7 (Microsoft Corporation) and the High Definition User Experience (HDX) MediaStream feature in Citrix’ products (Citrix Systems, Inc.). Although this approach works well for supported video formats, it is not applicable to applications that do not use a video playback codec for their visualization (such as games and visualization software). To alleviate these problems, we have previously proposed a hybrid approach (Simoens et al., 2008) that switches between two modes of operation: the classic thin client mode (suitable for static content) and a media streaming mode (preferable for dynamic content). However, a single encoding is used for the complete screen, leaving substantial room for further improve- ment by applying different encoding methods for different areas of the screen. Contents lists available at SciVerse ScienceDirect journal homepage: www.elsevier.com/locate/jnca Journal of Network and Computer Applications 1084-8045/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.jnca.2012.03.007 n Corresponding author. Tel.: þ32 93314938; fax: þ32 93314899. E-mail address: Bert.Vankeirsbilck@intec.ugent.be (B. Vankeirsbilck). Journal of Network and Computer Applications 35 (2012) 1620–1632