Artificial Intelligence systems as a solution to subjective video sensing in Contemporary Performing Arts. Dr Garth Paine Senior Lecturer, Music Technology, School of Contemporary Arts Research Associate, MARCS Auditory Laboratories University of Western Sydney Email: ga.paine@uws.edu.au Abstract This paper discusses approaches to realtime motion tracking in contemporary dance. It outlines some problems with current techniques and proposes, through previous research, some alternative approaches that could provide much richer data sets for realtime sonification and visualization of choreographic patterns. 1. Background. While undertaking my PhD (Paine, 2002b), I developed a number of approaches to the mapping of gesture (tracked using video based systems, VNS1 and Cyclops), which identify issues involved in the sonification of gesture (Mulder et al., 1997) in a manner that creates a visceral engagement with the quality of outcome. The following are excerpts from that research as a way of contextualizing the need for the AI research discussed below. During 2000, while I was the Australia Council for the Arts, New Media Arts fellow at RMIT University, I began some research that attempted to expand the interactive environment research (Moser & MacLeod, 1996) I had been doing over the years prior to include dynamic levels of intelligence in video tracking of movement and behaviour patterns. Through my previous research involving interactive immersive sound installations and composing using interactive music systems for the Australian Dance company, Company in Space, I formed the opinion that for an interactive system to be substantially more complex and sophisticated than the current first or second order responses, a level of artificial intelligence had to be introduced between the sensing stage and the mapping of the sensed data to synthesis parameters (Bongers., 2000; Paine, 2002a). A 1 See http://homepage.mac.com/davidrokeby/vns.html (08/08/05) relatively linear mapping of input data to a limited and fixed number of synthesis parameters does not support the evolution of system response over time. In order to pursue a model of interaction that goes beyond ‘response’ to a dynamic and intelligent relationship between the interactive agent(s), (human(s), engaged with the system/environment/installation, the system must be conditioned by it’s accumulated experience (histogram), being able to evolve responses accordingly (Ascott, 1997). Such a system would require a level of cognition in the form of a software infrastructure that could establish the patterns of interaction based on historical knowledge, and act accordingly. Neural networks are one possible approach, as are Hidden Markov models, both developed for pattern recognition tasks, and capable of being trained with an initial set of sensitivities, and able to evolve those sensitivities in response to varied input over time. Such systems built into interactive environments or interactive dance works may allow the system to be trained to: • Recognize individuals from their gesture patterns and movement characteristics (a useful feature for training an interactive environment to respond independently to different people, and also useful for interactive dance performances where it could be compositionally valuable to attribute different response patterns to different dancers). • Make subjective, qualitative judgments about the observed movement or gesture patterns, so that the system could determine the intent of the movement or gesture. Qualitative data of this kind would greatly extend the scope of current systems that respond to changes in light intensity per frame only, providing data that allow the calculation of speed of movement, position of movement, acceleration, relationship between two bodies etc. The accumulation of subjective, qualitative data would make additional layers of intention based aesthetic responses