Generation for Things Unknown: Accounting for First-Time Users and Hidden Scenes Verena Rieser Interaction Lab Heriot-Watt University Edinburgh EH14 4AS , UK v.t.rieser@hw.ac.uk Dimitra Gkatzia Interaction Lab Heriot-Watt University Edinburgh EH14 4AS , UK D.Gkatzia@hw.ac.uk Abstract This work presents a new approach for modelling “unknown unknowns” in Natu- ral Language Generation (NLG) systems. We first address the question of how to adapt to unknown users, using a combi- nation of cluster-based user modelling and Pareto optimal Multi-Objective Optimisa- tion. Next, we present a corpus study on generating referring expressions (RE) for instruction giving in the real-world, where the visual scene complexity is (currently) unknown. We find significant differences between REs generated in the virtual and the real world and draw general conclu- sions for NLG systems. 1 Introduction This line of research investigates Natural Lan- guage Generation (NLG) under uncertainty. There are two general types of uncertainty: Aleatoric and epistemic uncertainty. The former addresses variation observed in the real world, and is cur- rently accounted for by, e.g. modelling probabil- ity distributions over likely user responses (Rieser et al., 2014). The latter are “things we cannot know”, i.e. true unknowns which cannot be es- timated from data. These type of uncertainties are currently not taken into account when gener- ating system outputs. In this paper, we summarise our work on modelling “unknown unknowns” for NLG systems. In particular, we address two prob- lems: 1. How to adapt output to unknown user types? 2. How to generate referring expressions for ob- jects in unknown visual scenes? 2 Talking to Unknown Users Handling first time users is a common problem for NLG and interactive systems in general: the sys- tem cannot adapt to user preferences without prior knowledge. Previous work has shown that it is im- portant for NLG systems to adapt their output to specific users or user groups, such as nurses and patients (Gatt et al., 2009), or lecturers and stu- dents (Gkatzia et al., 2014a). However, adapta- tion becomes impossible when no prior informa- tion about this user exists, as is often the case for first time users. Furthermore, user preferences of- ten vary significantly (p< 0.001) for the same ut- terance (Walker et al., 2007), which will naturally affect the performance of models based on popu- lation average. We propose a novel model for first time users, which is based on clusters of potential user types. We apply this framework to medical first aid decision support systems, where we automatically generate short reports of sensor data data, includ- ing Breathing Rate (BR), the Blood Oxygen Satu- ration (SpO 2 ) and the Heart Rate (HR). In a med- ical emergency a patient’s survival often depends upon the prompt response and appropriate first aid given by the first person on scene, also known as “bystander”, who typically is a first-time user. NLG user modelling approaches assume that (1) the system is only used by a single user, and (2) the type of user is known in advance. In order to remedy the first assumption, Gkatzia et al. (2014a) suggest a multi-objective approach to NLG. In this framework, the preferences of lecturers and students are modelled as objective functions that need to be optimized simultaneously. A Rein- forcement Learning agent is then trained by us- ing the weighted sum of the modelled preferences as a reward function. However, their experiments show that this approach is unsuccessful – possibly because their linearised reward function cancels out the preferences of each user group. We pro- pose a different MOO approach and we actually demonstrate a Pareto optimal approach to Multi- Objective Optimisation (MOO) for NLG, which