Sign-to-Speech Model for Sign Language Understanding: A Case Study of Nigerian Sign Language

Sign-to-Speech Model for Sign Language Understanding: A Case Study of Nigerian Sign Language

Steven Kolawole, Opeyemi Osakuade, Nayan Saxena, Babatunde Kazeem Olorisade

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Demo Track. Pages 5924-5927. https://doi.org/10.24963/ijcai.2022/855

Through this paper, we seek to reduce the communication barrier between the hearing-impaired community and the larger society who are usually not familiar with sign language in the sub-Saharan region of Africa with the largest occurrences of hearing disability cases, while using Nigeria as a case study. The dataset is a pioneer dataset for the Nigerian Sign Language and was created in collaboration with relevant stakeholders. We pre-processed the data in readiness for two different object detection models and a classification model and employed diverse evaluation metrics to gauge model performance on sign-language to text conversion tasks. Finally, we convert the predicted sign texts to speech and deploy the best performing model in a lightweight application that works in real-time and achieves impressive results converting sign words/phrases to text and subsequently, into speech.
Keywords:
AI Ethics, Trust, Fairness: Societal Impact of AI
AI Ethics, Trust, Fairness: Fairness & Diversity
Computer Vision: Biometrics, Face, Gesture and Pose Recognition
Computer Vision: Recognition (object detection, categorization)