SOCIAL SIGNAL PROCESSING FOR PAIN MONITORING USING A HIDDEN CONDITIONAL RANDOM FIELD Afsane Ghasemi, Xinyu Wei, Patrick Lucey, Sridha Sridharan, Clinton Fookes Image and Video Laboratory, Queensland University of Technology, Australia ABSTRACT Automatic pain monitoring has the potential to greatly im- prove patient diagnosis and outcomes by providing a contin- uous objective measure. One of the most promising methods is to do this via automatically detecting facial expressions. However, current approaches have failed due to their inability to: 1) integrate the rigid and non-rigid head motion into a sin- gle feature representation, and 2) incorporate the salient tem- poral patterns into the classification stage. In this paper, we tackle the first problem by developing a “histogram of facial action units” representation using Active Appearance Model (AAM) face features, and then utilize a Hidden Conditional Random Field (HCRF) to overcome the second issue. We show that both of these methods improve the performance on the task of pain detection in sequence level compared to cur- rent state-of-the-art-methods on the UNBC-McMaster Shoul- der Pain Archive. Index Terms— Biomedical Monitoring, Social Signal Processing, Action Units, Pain, Hidden Conditional Random Field 1. INTRODUCTION Most of the work in biomedical monitoring uses physiologi- cal sensors which are required to be placed on the patient. An alternative to wearable sensors is to utilize Social Signal Pro- cessing which aims at providing computers with the ability to sense and understand human social signals unobtrusively through measuring human behavior via cameras and micro- phones [1]. A key example of this approach is in pain moni- toring. Recent improvements in patient outcomes in intensive care units (ICU) of hospitals have been achieved through ad- hering to standardized hygiene and monitoring checklists [2]. One of the items on the checklist is pain monitoring, where continuous monitoring and assessment of a patient’s pain level can lead to better and more effective treatments and diagnosis. However, this is problematic as pain is normally measured by self-report, where a human care-giver checks on the status of the patient at least every four hours. In sit- uations where a person can not communicate verbally (e.g., unconscious patient or a child), pain is normally estimated   2  1  1 0 1 2 3 4 5 k 1 2 k     0 1 2 3 4 5 Fig. 1. We automatically detect the pain rating from a video by first detecting the various facial action units (AUs), then we select the most salient segments of the video sequence based on the histogram of AUs which we incorporate into our hidden Conditional Random Field (HCRF) model to classify the pain rating. via associating the face of the patient by a care-giver to one of the faces on a chart (see bottom of Figure 1). Not only is this subjective, it is a heavy burden on staff as it requires continual monitoring and documentation of patients. As facial expres- sions can detect pain more reliably than any current wearable sensor [3, 4], automating pain detection through social signal processing can alleviate this burden while improving patient outcomes. The goal of this paper is to emulate an expert human ob- server by automatically choosing the most similar pain rating on the “faces-of-pain” scale based solely on the input video. A diagram of our approach is shown in Figure 1. Firstly, we take a video as our input. We then detect and track the face and facial features via an active appearance model (AAM) [5]. From these features, we then detect the individual facial ac- tion units (AUs). As pain is a high-level emotion incorpo- rating various facial expressions and motions, we developed a high-level representation by using a “histogram of AUs” representation which incorporates the various modes of pain. Pain is also a temporally varying signal, sometimes consist- ing of many modes and can be either short or long in duration (e.g., Figure 2). As these segments can be quite random, the entropy based approach applied to select salient short-term 2014 IEEE Workshop on Statistical Signal Processing (SSP) 978-1-4799-4975-5/14/$31.00 ©2014 IEEE 61