Speech to sign language translation system for Spanish R. San-Segundo a, * , R. Barra a , R. Co ´ rdoba a , L.F. D’Haro a , F. Ferna ´ndez a , J. Ferreiros a , J.M. Lucas a , J. Macı ´as-Guarasa b , J.M. Montero a , J.M. Pardo a a Grupo de Tecnologı ´a del Habla, Departamento de Ingenierı ´a Electro ´ nica, ETSI Telecomunicacio ´ n, Universidad Polite ´cnica de Madrid, Ciudad Universitaria s/n, 28040 Madrid, Spain b Department of Electronics, University of Alcala ´ , Spain Received 6 June 2007; received in revised form 3 December 2007; accepted 3 February 2008 Abstract This paper describes the development of and the first experiments in a Spanish to sign language translation system in a real domain. The developed system focuses on the sentences spoken by an official when assisting people applying for, or renewing their Identity Card. The system translates official explanations into Spanish Sign Language (LSE: Lengua de Signos Espan ˜ ola) for Deaf people. The trans- lation system is made up of a speech recognizer (for decoding the spoken utterance into a word sequence), a natural language translator (for converting a word sequence into a sequence of signs belonging to the sign language), and a 3D avatar animation module (for playing back the hand movements). Two proposals for natural language translation have been evaluated: a rule-based translation module (that computes sign confidence measures from the word confidence measures obtained in the speech recognition module) and a statistical translation module (in this case, parallel corpora were used for training the statistical model). The best configuration reported 31.6% SER (Sign Error Rate) and 0.5780 BLEU (BiLingual Evaluation Understudy). The paper also describes the eSIGN 3D avatar animation module (considering the sign confidence), and the limitations found when implementing a strategy for reducing the delay between the spoken utterance and the sign sequence animation. Ó 2008 Elsevier B.V. All rights reserved. Keywords: Spanish Sign Language (LSE); Spoken language translation; Sign animation 1. Introduction During the last two decades, there have been important advances in the three technological areas that support the implementation of an automatic speech to sign language translation system: sign language studies, spoken language translation and 3D avatar animation. Sign language presents a great variability depending on the country, even between different areas in the same coun- try. Because of this, from 1960 sign language studies have appeared not only in USA (Stokoe, 1960; Christopoulos and Bonvillian, 1985; Pyers, in press), but also in Europe (Engberg-Pedersen, 2003; Atherton, 1999; Meurant, 2004), Africa (Nyst, 2004) and Asia (Abdel-Fattah, 2005; Masa- taka et al., 2006). In Spain, there have been several propos- als for normalizing Spanish Sign Language (LSE: Lengua de Signos Espan ˜ola), but none of them has been accepted by the Deaf community. From their point of view, these pro- posals tend to constrain the sign language, limiting its flexi- bility. In 1991, (Rodrı ´guez, 1991) carried out a detailed analysis of Spanish Sign Language showing its main charac- teristics. She showed the differences between the sign lan- guage used by Deaf people and the standard proposals. This work has been expanded with new studies (Gallardo and Montserrat, 2002; Herrero-Blanco and Salazar-Garcia, 2005; Reyes, 2005). Spoken language translation has been and is being stud- ied in a number of joint projects such as C-Star, ATR, Ver- mobil, Eutrans, LC-Star, PF-Star and TC-Star. Apart from the TC-Star project, these projects addressed translation 0167-6393/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.specom.2008.02.001 * Corresponding author. Tel.: +34 915495700x4228; fax: +34 913367323. E-mail address: lapiz@die.upm.es (R. San-Segundo). www.elsevier.com/locate/specom Available online at www.sciencedirect.com Speech Communication 50 (2008) 1009–1020