Cognitive Psychology Emotion-modulated Recall: Congruency Effects of Nonverbal Facial and Vocal Cues on Semantic Recall Arianne Herrera-Bennett 1 a , Shermain Puah 2 , Lisa Hasenbein 3 , Dirk Wildgruber 4 1 Psychology, University of California, Davis, USA; Psychology, Ludwig Maximilian University of Munich, Germany, 2 Psychology, Ludwig Maximilian University of Munich, Germany, 3 Psychology, Ludwig Maximilian University of Munich, Germany; Education Sciences and Psychology, University of Tübingen, Germany, 4 Psychiatry and Psychotherapy, University of Tübingen, Germany Keywords: bimodal integration, semantic recall, emotional prosody, facial expression, congruency efects https://doi.org/10.1525/collabra.31601 Collabra: Psychology Vol. 8, Issue 1, 2022 The current study had two main goals: First, to replicate the ‘bimodal integration’ effect (i.e. the automatic integration of crossmodal stimuli, namely facial emotions and emotional prosody); and second, to investigate whether this phenomenon facilitates or impairs the intake and retention of unattended verbal content. The study borrowed from previous bimodal integration designs and included a two-alternative forced-choice (2AFC) task, where subjects were instructed to identify the emotion of a face (as either ‘angry’ or ‘happy’) while ignoring a concurrently presented sentence (spoken in an angry, happy, or neutral prosody), after which a surprise recall was administered to investigate effects on semantic content retention. While bimodal integration effects were replicated (i.e. faster and more accurate emotion identifcation under congruent conditions), congruency effects were not found for semantic recall. Overall, semantic recall was better for trials with emotional (vs. neutral) faces, and worse in trials with happy (vs. angry or neutral) prosody. Taken together, our fndings suggest that when individuals focus their attention on evaluation of facial expressions, they implicitly integrate nonverbal emotional vocal cues (i.e. hedonic valence or emotional tone of accompanying sentences), and devote less attention to their semantic content. While the impairing effect of happy prosody on recall may indicate an emotional interference effect, more research is required to uncover potential prosody-specifc effects. All supplemental online materials can be found on OSF ( https://osf.io/am9p2/). Introduction To date, the bimodal integration literature has systemati- cally demonstrated, across a range of discrete emotions (i.e. happy, fear, angry, sad), enhanced perception of emotional information when cues are conveyed simultaneously via dif- ferent channels (e.g., face and voice). Specifcally, emotions of faces were identifed more quickly and accurately when paired with words or sentences spoken in the congruent emotional prosody (de Gelder & Vroomen, 2000; Dolan et al., 2001; Massaro & Egan, 1996). Notably, because such effects were found even when subjects were tasked to ig- nore the accompanying voice, fndings have led authors to conclude that individuals naturally and automatically in- tegrate unattended emotional information, such as emo- tional prosody (Ethofer et al., 2006). Interestingly, while a body of evidence exists on the facilitating effects of cross- modal congruency on perception of emotional stimuli (e.g., Paulmann & Pell, 2011; for review, see Klasen et al., 2012), research has seemingly neglected to investigate effects on subsequent recall. Congruency effects, specifcally in the context of multi- sensory integration, have been strongly conceptualized in connection with semantic meaning, whereby here the term ‘semantic’ is used to convey a “correspondence of incoming signals” (for review, see Doehrmann & Naumer, 2008, p. 137). For instance, matching emotional connotations con- veyed at the level of verbal content and affective prosody (e.g., happy words spoken in a happy prosody) have been found to lead to faster lexical processing, even when sub- jects were not explicitly instructed to attend to the emo- tional content or tone, or when prosody and content varied from trial to trial (Nygaard & Queen, 2008). Authors pro- posed that emotional prosody may serve as a top-down con- straint on linguistic processing, whereby “the integration of emotional tone of voice with linguistic content occurs rela- tively early in the processing of spoken language” (Nygaard & Queen, 2008, p. 1025). Such fndings emphasize the nat- ural interaction of verbal and nonverbal aspects of speech during access and recognition of spoken language (Nygaard & Queen, 2008). The facilitatory effects of multisensory presentations aherrerabennett@ucdavis.edu a Herrera-Bennett, A., Puah, S., Hasenbein, L., & Wildgruber, D. (2022). Emotion- modulated Recall: Congruency Efects of Nonverbal Facial and Vocal Cues on Semantic Recall. Collabra: Psychology, 8(1). https://doi.org/10.1525/collabra.31601 Downloaded from http://online.ucpress.edu/collabra/article-pdf/8/1/31601/491430/collabra_2022_8_1_31601.pdf by guest on 26 January 2022