Vol.:(0123456789) SN Computer Science (2021) 2:220 https://doi.org/10.1007/s42979-021-00627-3 SN Computer Science ORIGINAL RESEARCH Design and Development of a Humanoid Robot for Sign Language Interpretation Ragib Amin Nihal 1  · Nawara Mahmood Broti 1  · Shamim Ahmed Deowan 1  · Sejuti Rahman 1 Received: 14 December 2020 / Accepted: 31 March 2021 / Published online: 20 April 2021 © The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd 2021 Abstract Purpose With an indisputable complexity of communication for hearing and speaking impaired people, most sign language recognition systems utilize virtual reality or onscreen robots. This paper presents the design and development of a special and low-cost humanoid robot that can perform as a sign language interpreter. To the best of our knowledge, this is the frst endeavor to fabricate a humanoid robot for Bangla sign language (BdSL) and Medical signs interpretation. Methods Considering the plethora of design criteria and balancing between rigidity and fexibility 3D models of the robot- ics parts are designed and 3D printed ensuring cost efciency. With the help of modern fabrication technology, the robot is developed and assembled with proper actuators and circuitry. An image dataset is built comprising 950 images for BdSL recognition and made publicly available. We utilized the Recurrent neural network (RNN) and Convolutional neural network (CNN) for deep learning model establishment and feature extraction from video and image data. Results The developed humanoid robot has 43 Degrees of freedom (DoF) which includes two 15 DoF hands. It can imitate 16 BdSL alphabets in sign language, can capture a video or image input in real-time from the user, and recognize 10 medical signs and 38 alphabets of BdSL. The learning model for video-based medical sign recognition achieved 87.5% test accu- racy. Image-based Bangla sign language recognition achieved an overall test accuracy of 98.19% in our dataset and 93.8% in another available dataset. Conclusion Compared to the state-of-the-art robotic systems for sign language interpretation, our approach has achieved higher kinematic characteristics, remarkable results in sign recognition, and impressive competency in sign imitation; all at almost 10 times lower cost than the state-of-the-art systems. The results are evidence that our approach is efcient and suitable in helping hearing and speaking impaired people. Moreover, this work initiates a research scope that can be further extended for creating equal opportunities for the hearing and speaking impaired community. Keywords Sign language · Recognition · Humanoid robot · Recurrent neural network · Convolutional neural network Introduction For years, sign language (SL) has been a bridge that elimi- nates the communication barrier between people with hear- ing and speaking disabilities and the rest of the world. About 5% of the world’s population have hearing and speaking dis- abilities and being aided by SL [1]. Still, there remains the issue that general people have insufcient knowledge about SL. While general people fnd it challenging to understand SL, an automatic sign language recognition (SLR) system can be a very useful tool. The urge to develop SLR systems has already drawn the attention of researchers. Diferent techniques are being applied to recognize SL from diferent countries. * Shamim Ahmed Deowan shamimdeowan.rme@du.ac.bd Ragib Amin Nihal rgbnihal2@gmail.com Nawara Mahmood Broti brotimahmood@gmail.com Sejuti Rahman sejuti.rahman@du.ac.bd 1 Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh