A Comprehensive Deep Learning Based System for Real Time Sign Language Recognition and Translation Using Raspberry Pi |
||
 |
 |
|
| © 2024 by IJCTT Journal | ||
| Volume-72 Issue-12 |
||
| Year of Publication : 2024 | ||
| Authors : Abini M.A, Divya Lakshmi P, Sharan K.S, Sulphiya V. N | ||
| DOI : 10.14445/22312803/IJCTT-V72I12P102 | ||
How to Cite?
Abini M.A, Divya Lakshmi P, Sharan K.S, Sulphiya V. N, "A Comprehensive Deep Learning Based System for Real Time Sign Language Recognition and Translation Using Raspberry Pi," International Journal of Computer Trends and Technology, vol. 72, no. 12, pp. 8-16, 2024. Crossref, https://doi.org/10.14445/22312803/IJCTT-V72I12P102
Abstract
Sign language is an important aspect of human communication for a variety of reasons, particularly when deaf and dumb individuals are communicating. This study describes a novel method for translating sign language into spoken language that employs a Raspberry Pi 3 and the MobileNet-V2 deep learning model. Technology has advanced significantly, and many studies have been conducted to assist the deaf and dumb. Deep learning and computer vision can also be utilized to support the cause and have an impact on it. The system includes a camera that collects images of the signer's hand gestures and processes them for classification using the MobileNet V2 model. The translated text is entered into text-to-speech software. The system was trained on a huge dataset of sign language movements using transfer learning techniques, and it attained an accuracy of 99.52% on the validation set. The Raspberry Pi 3 was chosen as the hardware platform for its low cost, portability, and suitability for various applications and environments.
Keywords
MobileNet-V2, Deep learning, Sign language translator, Raspberry Pi 3.
Reference
[1] Ashish S. Nikam, and Aarti G. Ambekar, “Sign Language Recognition Using Image Based Hand Gesture Recognition Techniques,” 2016 Online International Conference on Green Engineering and Technologies (IC-GET), Coimbatore, India, pp. 1-5, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Hernando Gonzalez, Silvia Hernández, and Oscar Calderón, “Design of a Sign Language-to-Natural Language Translator Using Artificial Intelligence,” International Journal of Online and Biomedical Engineering, vol. 20, no. 3, pp. 89-98, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Muhaimin Bin Munir et al., “A Machine Learning Based Sign Language Interpretation System for Communication with Deaf-Mute People,” 21: Proceedings of the XXI International Conference on Human Computer Interaction, Málaga Spain, pp. 1-9, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Gopireddy Sirisha et al., “An Image Processing Based American Sign Language Fingerspelling Interpreter,” International Virtual Conference on Industry 4.0, pp. 201-211, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Dipali Dhake et al., “Sign Language Communication with Dumb and Deaf People,” International Journal of Engineering Applications and Technology, vol. 5, no. 4, pp. 254-258, 2020.
[Google Scholar] [Publisher Link]
[6] Samar Mouti, and Samer Rihawi, “IoT and Sign Language System (SLS),” International Journal of Engineering Research and Technology, vol. 13, no. 12, pp. 4199-4205, 2020.
[Google Scholar] [Publisher Link]
[7] Mannava Vivek, and Vitapu Gnanasagar, “Portable Sign Language Translator for Emergency Response Teams,” International Journal of Scientific Research & Engineering Trends, vol. 6, no. 3, pp. 1203-1207, 2020.
[Google Scholar]
[8] N.M. Ramalingeswara Rao et al., “Conversion Techniques of Sign and Speech into Text Using Raspberry Pi,” International Journal for Modern Trends in Science and Technology, vol. 8, no. S05, pp. 121-125, 2022.
[Publisher Link]
[9] Jakub Gałka et al., “Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition,” IEEE Sensors Journal, vol. 16, no. 16, pp. 6310-6316, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Mohamed Deriche, Salihu O. Aliyu, and Mohamed Mohandes, “An Intelligent Arabic Sign Language Recognition System Using a Pair of LMCs With GMM Based Classification,” IEEE Sensors Journal, vol. 19, no. 18, pp. 8067-8078, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Salma A. Essam El-Din, and Mohamed A. Abd El-Ghany, “Sign Language Interpreter System: An Alternative System for Machine Learning,” 2020 2nd Novel Intelligent and Leading Emerging Sciences Conference, Giza, Egypt, pp. 332-337, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[12] CH Ramasuri Appalanaidu et al., “Sign Language Recognition and Speech Conversion Using Raspberry Pi,” International Journal of Creative Research Thoughts, vol. 8, no. 5, pp. 2103-2106, 2020.
[Google Scholar] [Publisher Link]
[13] Daniel S. Breland et al., “Deep Learning-Based Sign Language Digits Recognition from Thermal Images with Edge Computing System,” IEEE Sensors Journal, vol. 21, no. 9, pp. 10445-10453, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Yande Li et al., “Hand Gesture Recognition and Real-Time Game Control Based on a Wearable Band with 6-Axis Sensors,” 2018 International Joint Conference on Neural Networks, Rio de Janeiro, Brazil, pp. 1-6, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Vaibhav Mehra, Aakash Choudhury, and Rishu Ranjan Choubey, “Gesture To Speech Conversion Using Flex Sensors, MPU6050 and Python,” International Journal of Engineering and Advanced Technology, vol. 8, no. 6, pp. 4686-4690, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[16] Lean Karlo S. Tolentino et al., “Static Sign Language Recognition Using Deep Learning,” International Journal of Machine Learning and Computing, vol. 9, no. 6, pp. 821-827, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Ulzhalgas Seidaliyeva et al., “Real-Time and Accurate Drone Detection in a Video with a Static Background,” Sensors, vol. 20, no. 14, pp. 1-19, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[18] Miguel Rivera-Acosta et al., “American Sign Language Alphabet Recognition Using a Neuromorphic Sensor and an Artificial Neural Network,” Sensors, vol. 17, no. 10, pp. 1-17, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Gokulnath Anand, and Ashok Kumar Kumawat, “Object Detection and Position Tracking in Real Time Using Raspberry Pi,” Materials Today: Proceedings, vol. 47, no. 11, pp. 3221-3226, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[20] Dushyant Kumar Singh, Anshu Kumar, and Mohd. Aquib Ansari, “Robust Modelling of Static Hand Gestures Using Deep Convolutional Network for Sign Language Translation,” 2021 International Conference on Computing, Communication, and Intelligent Systems, Greater Noida, India, pp. 487-492, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[21] U. Fadlilah et al., “Modelling of Basic Indonesian Sign Language Translator Based on Raspberry Pi Technology,” Scientific and Technical Journal of Information Technologies, Mechanics and Optics, vol. 22, no. 3, pp. 574-584, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[22] V. Subashini et al., “Sign Language Translation Using Image Processing to Audio Conversion,” 2024 Third International Conference on Intelligent Techniques in Control, Optimization and Signal Processing, Krishnankoil, Virudhunagar District, Tamil Nadu, India, pp. 1-6, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[23] Gempur Bayu Aji, Fazmah Arif Yulianto, and Andrian Rakhmatsyah, “Sign Language Translator Based on Raspberry Pi Camera Using the Haar Cascade Classifier Method,” Building of Informatics, Technology and Science, vol. 4, no. 4, pp. 1747-1753, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[24] Saleh Ahmad Khan et al., “An Efficient Sign Language Translator Device Using Convolutional Neural Network and Customized ROI Segmentation,” 2019 2nd International Conference on Communication Engineering and Technology, Nagoya, Japan, pp. 152-156, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[25] Ke Dong et al., “MobileNetV2 Model for Image Classification,” 2020 2nd International Conference on Information Technology and Computer Application, Guangzhou, China, pp. 476-480, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[26] Shubhendu Apoorv, Sudharshan Kumar Bhowmick, and R Sakthi Prabha, “Indian Sign Language Interpreter Using Image Processing and Machine Learning,” IOP Conference Series: Materials Science and Engineering, Second International Conference on Materials Science and Manufacturing Technology, Coimbatore, Tamil Nadu, India, vol. 872, pp. 1-6, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[27] Sruthi Chandrasekaran, “American Sign Language Recognition and Translation Using Deep Learning and Computer Vision,” National College of Ireland, MSc Research Project, pp. 1-18, 2021.
[Google Scholar] [Publisher Link]