LSESpeak: A spoken language generator for Deaf people Verónica López-Ludeña ⇑ , Roberto Barra-Chicote, Syaheerah Lutfi, Juan Manuel Montero, Rubén San-Segundo Departamento de Ingeniería Electrónica, ETSI Telecomunicación, Universidad Politécnica de Madrid, Spain article info Keywords: Spanish sign language LSE: Lengua de Signos Española SMS to Spanish translation Speech generation from LSE Emotional speech synthesis abstract This paper describes the development of LSESpeak, a spoken Spanish generator for Deaf people. This sys- tem integrates two main tools: a sign language into speech translation system and an SMS (Short Mes- sage Service) into speech translation system. The first tool is made up of three modules: an advanced visual interface (where a deaf person can specify a sequence of signs), a language translator (for gener- ating the sequence of words in Spanish), and finally, an emotional text to speech (TTS) converter to gen- erate spoken Spanish. The visual interface allows a sign sequence to be defined using several utilities. The emotional TTS converter is based on Hidden Semi-Markov Models (HSMMs) permitting voice gender, type of emotion, and emotional strength to be controlled. The second tool is made up of an SMS message editor, a language translator and the same emotional text to speech converter. Both translation tools use a phrase-based translation strategy where translation and target language models are trained from parallel corpora. In the experiments carried out to evaluate the translation performance, the sign language- speech translation system reported a 96.45 BLEU and the SMS-speech system a 44.36 BLEU in a specific domain: the renewal of the Identity Document and Driving License. In the evaluation of the emotional TTS, it is important to highlight the improvement in the naturalness thanks to the morpho-syntactic fea- tures, and the high flexibility provided by HSMMs when generating different emotional strengths. Ó 2012 Elsevier Ltd. All rights reserved. 1. Introduction In the world, there are around 70 million people with hearing deficiencies (information from World Federation of the Deaf). Deaf- ness brings about significant communication problems: deaf peo- ple cannot hear and most of them are unable to use written languages, having serious problems when expressing themselves in these languages or understanding written texts. They have prob- lems with verb tenses, concordances of gender and number, etc., and they have difficulties when creating a mental image of abstract concepts. This fact can cause deaf people to have problems when accessing information, education, job, social relationship, culture, etc. Deaf people use a sign language (their mother tongue) for com- municating and there are not enough sign-language interpreters and communication systems. In the USA, there are 650,000 Deaf people (who use a sign language), although there are more people with hearing deficiencies, but only 7000 sign-language interpret- ers, i.e., a ratio of 93 deaf people to 1 interpreter. In Finland we can find the best ratio, 6–1, and in Slovakia the worst with 3000 users to 1 interpreter (Wheatley and Pabsch, 2010). In Spain this ratio is 221–1. This information shows the need to develop auto- matic translation systems with new technologies for helping hear- ing and deaf people to communicate between themselves. It is necessary to make a difference between ‘‘deaf’’ and ‘‘Deaf’’: the first one refers to non-hearing people, and the second one re- fers to people who use a sign language as the first way to commu- nicate being part of the ‘‘Deaf community’’. Each country has a different sign language, but there may even be different sign lan- guages in different regions in the same country. There is also an international sign language, but most of deaf people do not know it. However, national sign languages are fully-fledged languages that have a grammar and lexicon just like any spoken language, contrary to what most people think. Traditionally, deafness has been associated to people with learning problems but this is not true. The use of sign languages defines the Deaf as a linguistic minority, with learning skills, cultural and group rights similar to other minority language communities. According to information from INE (Statistic Spanish Institute), there are 1,064,000 deaf people in Spain and 50% are more than 65 years old. They are a geographically dispersed population, pro- ducing more social isolation. 47% of deaf population do not have basic studies or are illiterate, and only between 1% and 3% have fin- ished their studies (as opposed to 21% of Spanish hearing people). Also, 20% of the deaf population is unemployed (30% for women). 0957-4174/$ - see front matter Ó 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.eswa.2012.08.062 ⇑ Corresponding author. Address: Grupo de Tecnología del Habla, Dpto. Ingeniería Electrónica, ETSI Telecomunicación, Universidad Politécnica de Madrid, Ciudad Universitaria s/n, 28040-Madrid, Spain. E-mail address: veronicalopez@die.upm.es (V. López-Ludeña). Expert Systems with Applications 40 (2013) 1283–1295 Contents lists available at SciVerse ScienceDirect Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa