Subhead Body Text Adaptive maternal synchrony: Multimodal practices are tailored to infantsʼ attention Iris Nomikou 1,2 , Katrin Solveig Lohan 1,3 & Katharina J. Rohlfing 1,2,3 1 Bielefeld University, 2 CITEC, Emergentist Semantics Group, 3 CoR-Lab, Applied Informatics Group BCCCD 2012, Budapest contact: inomikou@techfak.uni-bielefeld.de Background Research in social cognition suggests that infantsʼ attention is educated in interactions [1]. By synchronizing speech and gesture, caregivers recruit infantsʼ attention and assist them in binding information from different modalities [2]. Recent studies have looked micro-analytically into maternal behavior in early natural mother-infant interactions and have revealed that mothers vocalize in a tight relationship with action “packaging” their actions acoustically and making language perceivable and tangible [3]. This shows that the input provided to the infants is highly organized and suggests that a ʻmultimodal bootstrappingʼ mechanism may exist. A further body of research (ex. [4]) has focussed on the ostensive nature of infant directed behavior, which allows infants to recognize that they are the addressees of communication without necessarily knowing what exactly the content of this communication is. However, it is currently still under investigation what signals constitute ostension and how infantsʼ sensitivity towards ostensive cues develops. Method Participants ‣ 15 German speaking mothers and their infants. (7 male, 8 female) ‣ Age M = 14 weeks ; 6 days Data Coding Data were coded continuously using INTERACT ® 9. Following coding categories were used for the analysis: ‣ modalities of synchrony: For maternal behavior we used the Body Movement Classes defined in [3]. We coded following modalities of vocal-motor synchrony: head, hands, face, torso ‣ gaze: We coded infant gaze behavior, i.e coded the intervals during which the infant was gazing at the motherʼs face or was gazing away Analysis ‣ We calculated the overlap of each class of body movement (face, head, hands, torso) with each class of infant gaze (gaze at, gaze away) ‣ We compared the duration of infant gaze-maternal body movement overlap with the overall duration of the classes of body movement ‣ We compared the average duration of overlaps with the average duration of events in the classes of body movement Conclusions To capture the ways in which the multimodal input provided to young infants during natural interactions is coupled to their own behavior we calculated the temporal overlap between classes of maternal vocal-motor synchrony and the infantsʼ gaze behavior. We found that mothers synchronized their vocal behavior with body movement significantly more when the infants were looking at them than when their infants were looking away. We then compared the duration of infant gaze events and the duration of maternal synchrony events and found a strong correlation between them for the HEAD and FACE modalities. This suggests that mothers not only provide more synchrony when the infants are looking at them but they also sustain the synchrony for the duration of the infantʼs gaze. We propose that by actively adapting their multimodal practices to the infantʼs attention, mothers are reinforcing the input, making it more ostensive, more unambiguously addressed to the infant. This may be an early mechanism for the education of infantsʼ engagement and attention, leading to sensitivity for ostension and to joint attention in the months to follow. References [1] Zukow-Goldring, P. (1997). A social ecological realist approach to the emergence of the lexicon: Educating attention to amodal invariants in gesture and speech. In: C. Dent-Read, & P. Zukow-Goldring (Eds.), Evolving explanations of development: Ecological approaches to organism-environment systems: 199-250. Washington, DC.: American Psychological Association. [2] Gogate, L. J., Bahrick, L. E & Watson, J. D. (2000): A study of multimodal motherese: The role of temporal synchrony between verbal labels and gestures. In: Child Development 71: 878–894. [3] Nomikou, I. & Rohlfing, K. J. (2011). Language does something: Body action and language in maternal input to three-month-olds. In: IEEE Transactions on Autonomous Mental Development, Vol. 3 (2): pp. 113-128. [4] Csibra, G. (2010). Recognizing Communicative Intentions in Infancy. In: Mind And Language, 25(2): pp. 141–168. We would like to thank all the mothers and children participating in this study as well as Elena Justus and Sabrina Böckmann for their support in data coding. The present research was supported by a Dilthey Fellowship of the Volkswagen Foundation. Katrin Lohan and Katharina Rohlfing gratefully acknowledge the financial support from the FP7 European Project ITALK (ICT-214668) Analysis speech-torso “(pause) [a:] [a:] [a:m] (pause) [start position] [bends forward] [reaches end position] speech-face “Hello (flick sound) (pause)” [start position] [releases] [raises eyebrows] speech-hands speech-head “(pause) Was denn?” “(pause) What is it?” [start position] [moves head up] [returns to start] Results We found that all synchrony modalities occurred significantly more during infant gaze at the mother. (fig. 4) For: FACE t(13) = 11.47 ; p < 0.001, HEAD t(14) = 10.97 ; p < .001, TORSO t(14) = 4.88 ; p < .001 and HANDS t(14) = 2.78 ; p = .015 We analyzed the ways in which mothersʼ vocal behavior was temporally coordinated with their body movement. Movement in the visible parts of the body when co-occurring with vocal behavior were coded as Classes of Body Movement (see fig. 2). Such a Class was annotated when: (a) vocal behavior co-occurred in perfect synchrony with body movement; and (b) both verbal and motor behavior were continuously active. Acknowledgments We pursue the idea that infants are educated towards ostensive cues by: 1. the multimodal input of the mother being highly coupled (i.e. responsive) to their behavior 2. the mothersʼ monitoring the gaze behavior of the infant when designing their multimodal practices, thus reinforcing the ostensive character of their behavior Aim We visited the mothers and their infants at their homes and filmed them during a natural everyday activity: changing a diaper. Mothers were given no instructions. They were just asked to change their infantʼs diaper “as they normally do”. Fig. 1: The setup Vocal-motor Synchrony Maternal synchrony-infant gaze overlap We calculated the duration of overlap between gaze and synchrony and compared this overlap with the overall duration of occurrence of events in the synchrony class. Then we calculated the average duration of overlaps and compared it with the average duration of events in the synchrony class (fig. 3). We found a strong correlation between the infantʼs gaze duration at the mother and the duration of maternal synchrony for the FACE (r = .93 ; p < .001) (fig. 5) and HEAD (r = .98 ; p < .001) (fig. 6) modality. This correlation was further supported by a weak correlation of maternal synchrony and infantʼs gaze away: FACE (r = .37 ; p = .198), HEAD (r = .23 ; p = .403) “h u i” [lifts shirt] [passes over head] [pushes behind head] Fig. 2: Examples for all Classes of Body Movement Fig. 3: Maternal Synchrony-Infant Gaze overlap Fig. 4 Fig. 5 Fig. 6 Tuesday, January 10, 2012