Cloud Based Real-Time Collaborative Filtering for Item-Item Recommendations Rafael Pereira, Hélio Lopes, Karin Breitman Departamento de Informática PUC-Rio Rio de Janeiro, Brazil {rpereira, lopes, karin}@inf.puc-rio.br Vicente Mundim, Wandenberg Peixoto WebMedia Globo.com Rio de Janeiro, Brazil {vicente.mundim, wandenberg}@corp.globo.com Abstract—In this paper we argue that the combination of collaborative filtering techniques, particularly for item-item recommendations, with emergent cloud computing technology can drastically improve algorithm efficiency, particularly in situations where the number of items and users scales up to several million objects. We introduce a real-time item-item recommendation architecture, which rationalizes the use of resources by exploring on-demand computing. The proposed architecture provides a real-time solution for computing online item similarity, without having to resort to either model simplification or the use of input data sampling. We present results from a real life case study to show that it is possible to greatly reduce recommendation times (and overall financial costs) by using dynamic resource provisioning in the Cloud. Finally, we also discuss potential research opportunities that arise from this paradigm shift. Keywords - Distributed Architecture, Cloud Computing, System Architecture, Service Orientation, Collaborative Filtering I. INTRODUCTION The ability to offer relevant suggestions to users is becoming extremely relevant for web applications, particularly those whose business models are dependent on audience ratings, e.g., content commercialization and product sales [1]. Moreover, because effective recommendations play such an important role in user experience, users frequently resort to recommendations as a means for content discovery. This is the case with Amazon [2, 3] and Netflix [4, 5, 6]. Recommendation systems [7, 8] usually combine user profiles and item and product information to generate recommendations. In other words, a recommendation system is composed of contextual data, which is the information that the system has before it starts the recommendation process, input data, which is the information received for the recommendation process, and an algorithm that uses the context and input data to model recommendations. Collaborative filtering [9, 10, 11, 12] is a recommendation technique that resorts to the user-item interaction history to find relationships between them. One example of such a relationship is computing the similarity between two items, such as videos [13], both viewed by the same group of users. In this case, no contextual information about the items is considered, which means that the recommendation algorithm does not have any information on “which are the items” and “what are their types or characteristics”. There are several challenges associated with collaborative filtering engines. These algorithms must have sufficient performance to manipulate large sparse datasets, scale as the numbers of users and items grow, and retrieve relevant recommendations within a reasonable timeframe. Thus, increasing the numbers of users and items poses great challenges for this type of system. One of these challenges is to improve the quality of recommendations made to users, since good recommendations increase user fidelity. A further challenge is the appropriate scaling of the system to retain its effectiveness. These two challenges tend to be conflicting; to obtain fast recommendations the model should be simpler, but this reduces the overall quality of the recommendations. Nevertheless, an efficient recommendation engine must take both aspects into consideration. Additionally, there are specific scenarios that require the real-time processing of input data to produce relevant recommendations. This is the case in breaking news, where the content of interest is very volatile, and may become obsolete in a manner of minutes after being posted [14, 15]. In cases such as this, in additional to scalability, there is the challenge of producing relevant recommendations in real- time. This scenario adds an element of complexity to the recommendation models used by Amazon and Netflix, for example, which deal with less volatile items. An ideal recommendation engine is capable of both scaling up, as the number of items and/or users grows, and producing a relevant recommendation in as small a time as possible (almost real-time). One approach for scaling up the recommendation engine is through input data sampling. With this alternative it is possible to produce recommendations within a reasonable amount of time, using available, but often limited, computational resources. This process, of course, tends to reduce the accuracy of the recommendation, which may not be relevant to the user in question. Therefore, it is highly desirable that all available data is used to produce the best recommendation possible. The ability to contract computing resources on demand, from a cloud computing vendor, lifts the resource availability limitation, and thus enables the