Sign Language-to-Text Dictionary with Lightweight Transformer Models

Sign Language-to-Text Dictionary with Lightweight Transformer Models

Jérôme Fink, Pierre Poitier, Maxime André, Loup Meurice, Benoît Frénay, Anthony Cleve, Bruno Dumas, Laurence Meurant

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
AI for Good. Pages 5968-5976. https://doi.org/10.24963/ijcai.2023/662

The recent advances in deep learning have been beneficial to automatic sign language recognition (SLR). However, free-to-access, usable, and accessible tools are still not widely available to the deaf community. The need for a sign language-to-text dictionary was raised by a bilingual deaf school in Belgium and linguist experts in sign languages (SL) in order to improve the autonomy of students. To meet that need, an efficient SLR system was built based on a specific transformer model. The proposed system is able to recognize 700 different signs, with a top-10 accuracy of 83%. Those results are competitive with other systems in the literature while using 10 times less parameters than existing solutions. The integration of this model into a usable and accessible web application for the dictionary is also introduced. A user-centered human-computer interaction (HCI) methodology was followed to design and implement the user interface. To the best of our knowledge, this is the first publicly released sign language-to-text dictionary using video captured by a standard camera.
Keywords:
AI for Good: Multidisciplinary Topics and Applications
AI for Good: Computer Vision
AI for Good: Humans and AI
AI for Good: Natural Language Processing