AI-Driven Sign Language Interpretation for Nigerian Children at Home

1 Introduction

Deaf and Hard of Hearing (DHH) individuals constitute a significant portion of the population in Nigeria, where as many as 22 million Nigerians, in an estimated population of about 155 million people suffer from severe to profound hearing loss and are considered as persons with disabilities [Eleweke, 2002; Treat, 2016]. Studies have also indicated that up to 14% of school age children in Nigeria between the ages of 5 and 14 suffer from such severe hearing loss [McPherson and Swart, 1997]. These age ranges make up some of the largest population groups in Nigeria. In addition, as many as 84% of the deaf population in Nigeria remain under-educated and economically underdeveloped [Eleweke et al., 2015], indicating a close connection between deafness and poverty in the country [Asonye, 2016].

In an extensive study of the Deaf culture in Nigeria [Asonye et al., 2018], the authors collected data from a total of six primary and secondary schools for the Deaf, from the Northcentral and Southeastern regions of Nigeria which are traditionally very different cultural settings. They found the Deaf cultures across the regions to be similar, having comparable socioeconomic and assimilation challenges. For example, the earliest age that DHH children started kindergarten was about 6-7 years, the age where their hearing peers were already in the equivalent of 2nd or 3rd grade in elementary school. Similarly, students could be as old as 15 years, when starting elementary school, significantly older age ranges than their hearing peers.

As reported by the Nigerian Deaf study [Asonye et al., 2018], about 75% of pupils/students observed in their study did not have congenital hearing loss, but rather developed deafness because of various diseases contacted in childhood 1 . Many parents of the deaf students therefore are themselves not deaf and for many cultural reasons, do not embrace the Deaf culture.

Early language acquisition has been shown to involve linguistic and neural processes that have to do with the postnatal brain development interacting with its environment [Grimshaw et al., 1998], In the absence of social environments that can provide the DHH children with opportunities for communicative experiences, early language acquisition is missed and the associated neural, cognitive and linguistic processes are delayed or missed [Hoff, 2006]. In a 2018 study on the quality of life of deaf students in Southwest Nigeria [Jaiyeola and Adeyemo, 2018], it was found that stigmatization of deafness originated in the students' families and propagated to their immediate communities primarily due to problems with communication. The study stated that this communication gap between the Deaf students and their hearing family members continued into the years, sometimes having adverse consequences; the children are often discriminated against and excluded, they experience inequalities in society, and are exposed to vulnerabilities that tend to leave them behind.

1 More details of the various causes of hearing loss they observed can be found in the Nigerian Deaf study [Asonye et al., 2018].

AI research on automating two-way Sign Language (SL) translation can play a critical role in bridging this communication and engagement gap for Deaf students at home. This is a very challenging AI problem but one that can allow signers to provide and receive information in their own natural language, without requiring their interlocutors to learn a new language.

1.1 Problem Statement

We propose an iterative end-to-end approach to deploying AI-driven sign language translation devices based on existing sign language tutoring processes. Such devices can provide 2-way sign language interpretation services to young DHH students and their families in a pre-identified study area in Nigeria, to foster inclusivity at home, among hearing family members and peers.

1.2 Link to Specific SDGs and LNOB

This project goes beyond the traditional research objectives by connecting to specific UN Sustainable Goals - "Quality Education", "Decent Work and Economic Growth", and "Reduced Inequalities". Improving language accessibility and communication tools for DHH students in their home settings ensures inclusive and equitable social interactions and promotes lifelong learning opportunities for them outside of the school environment. Ensuring this creates increased opportunities for full and productive future employment for the DHH children, thus resulting in reduced inequalities.

Similarly, the Leave no one behind (LNOB) principle aims to combat discrimination and rising inequalities within and amongst countries. The UN 2030 agenda [UN Department of Economic and Social Affairs, 2023] cites that a major cause of people being left behind is persistent forms of discrimination,, resulting in individuals, families and whole communities such as the Deaf community in developing countries such as in Nigeria, being marginalized and excluded [Chapple, 2019; Leigh et al., 1998; Best, 2015]. Providing easier methods of communication between DHH students and their hearing family members and peers can potentially reduce discrimination, exclusion and unconscious biases; it can foster acceptance within the community and also diminish the vulnerabilities that leave DHH children behind, undermining their individual potentials.

To this end, this proposed project is in collaboration with Save the Deaf and Endangered Languages Initiative (SDELI) 2 , a community based nongovernmental organization (NGO) committed to the development of Deaf children and adults in Nigeria. S-DELI is comprised of (i) a research component that documents collects data about sign languages and Deaf demographic data across Nigerian Deaf communities; (ii) an advocacy and outreach component that successfully pushed the policy to include the implementation of Newborn Hearing Screening and Early Intervention Program (NBHS/EI) in Nigeria; and (iii) an interpretation service component that trains and provides interpreting services for deaf individuals and groups, families and corporate bodies.

2 https://www.s-deli.org/

The proposed project is also in close collaboration with the Demonstration School for Deaf Children (DSDC), located in Kaduna, in Northern Nigeria. DSDC was founded by Brenda Woosman, a Canadian citizen in 1987, as she visited Northern Nigeria where she learned about the plight of deaf children in the area. Currently operating as non-profit organization, DSDC is partially supported by the Kaduna State Ministry of Human Services and Social Development (KDHSSD), a governmental organization focused on developing the rights and privileges of women, children, the socially disadvantaged and physically challenged in the State. There are currently 115 students from kindergarten 1 through Primary six all the way to Junior Secondary school level 3. The school has 13 teachers of which 7 are proficient in sign language. The Principal of DSDC, Mrs Victoria Adesina is a Project Collaborator.

Lastly, we are also collaborating with the National Technical Institute for the Deaf (NTID) via Ruchi Gamta, a faculty member and Instructor. NTID is the first and largest technological college in the world for DHH students [McCarthy, 2018] and is one of the nine colleges at the Rochester Institute of Technology (RIT), where three of the Investigators on this proposal are either current or past faculty members. More on this will be discussed in Section 4.

1.3 A Brief History of ASL in Nigeria

American Sign Language (ASL) is the primary sign language used in Nigeria, although over the years, various "dialects" of it are used across different regions of the country. ASL was introduced to the Nigerian Deaf Community by Andrew Forster, a Black American missionary and Galladuet University alumnus. Foster established the Ibadan Mission School for the Deaf in Southwestern Nigeria in 1960, and this school is credited to have significantly impacted the development and advancement of formal deaf education across Nigeria.

Till date, ASL along with its regional variants, is the primary sign language used in the schools for the Deaf, special education centers and deaf units established within regular schools across the country. But in addition to ASL being the "sign lingua franca" in Nigeria, there are many different native sign languages that we will be interested in engaging with and documenting over the course of this work.

2 Strategy and Case Study

The project will comprise of four major research thrusts: (i) user-centered interface design; (ii) 2-way neural ASL translation development (detailed description in Section 3; (iii) timely realization of the interpretation device; and (iv) field evaluation study. The project will be implemented across two study areas, in the US and in Nigeria where the device will be deployed to parents and family members, to enhance their communication with the DHH students at home.

User-centered Interface Design:

This component of the research will include iterative codesign sessions engaging various stakeholders such as DHH students, hearing students, Instructors, and Interpreters with the goal of developing a working 2-way sign-language interpreting agent. It is important to note that the agent is not a replacement for human interpreters, rather it is can act as an agent to bridge the hearing-signing communication gap in cases when human interpreters are not readily available. The initial design work will be carried out at RIT. .

Because of the interdependence nature of the project, this thrust will employ a deliverable-oriented approach [Hakiel, 1995] to the user-centered design, so that the outputs of the design can guide the development of other research products in the project pipeline.

Timely Realization of the Interpretation Device

Previous research has established the importance of timing [Bai et al., 2022] and pauses in the use of animations [Alkhazraji et al., 2018] for ASL among the Deaf population. In this research thrust, we seek to answer two primary research questions in realizing ASL signing using robots: RQ1: How can we autonomously detect, track and follow the intended communicator for two-way ASL interpretation? and RQ2: How can we ensure end-to-end timeliness of communication in accordance to accepted ASL signing between humans to best use an avatar to be an ASL interpreter? RQ3: How can we constrain the 2-way translation AI engine sufficiently, to successfully operate on a small mobile device that can be deployed in a low-resource third-world country?

We aim to address these research questions in this thrust.

Field Evaluation Study

The project will be initially iteratively designed, developed and tested at RIT, to meet predefined usability metrics: (i) effectiveness (accuracy and completeness), (ii) efficiency (resource usage) and (iii) satisfaction (comfort and acceptability of use). The next phase of the project will take place in Kaduna, Nigeria at the the DSDC, where the S-DELI NGO will host the project. They will assist with the user training of parents at the school, assisting them in setting up the tool as an interpretation device, monitoring the deployment of the tools to the home and managing the data collection for further refinement.

At the DSDC, we are currently in discussions with the principal to identify 20 initial families of children ages 3-7 year old, in the early stages of language usage, where their vocabulary is still limited. We will enroll their parents and family members in the first round of the study. Currently, DSDC runs a weekend ASL tutorial session for one hour, where parents and other family members especially siblings of the DHH students are invited and taught basic ASL. The goal is to encourage continued communication beyond the time in school, for the DHH student. This project will leverage such existing processes.

DSDC will conduct quarterly sign language tests (sign speed and accuracy, sign recognition rates, etc) for parents using to the tool, compared with a control matched goup not using the tool. Participants in the study will be given prizes intermittently, to encourage compliance and ensure participants continue to use the tool throughout the duration of the project. Surveys, questionnaires and on-site interviews will be given to measure the quantity and quality of communication taking place at home with the DHH child.

3 Technical Proposal : Two-way Neural ASL Translations

We aim to develop robust neural models capable of faithfully translating sign language to gloss 3 as well as in the reverse direction, from gloss to sign, a more challenging AI problem. Sign-to-gloss translation is getting more attention in the AI community , unlike its reverse problem, gloss-to-sign production. To complete the process, we will also train gloss-totext and text-to-gloss translation models. Generated texts can be converted to speech using open-source speech generators and spoken words can be converted to texts using commercial ASRs (automated speech recognizer).

For word representations, we will use the fastText 4 word embedding, a faster and more compact Facebook extension of Word2Vec 5 . Although fastText does not consider word ordering, a trait generally considered negatively, in this work, in gloss recognition and production, not paying special attention to word ordering could prove to be beneficial since the grammatical rules of ASL are different from those of spoken English.

For this research, we will conduct the following activities:

1. Create a new multicultural word-level ASL dataset involving subjects from Nigeria, using the sign language version of the Swadesh list [Emmorey and Lane, 2013]. This new dataset will be combined with the existing word-level, 2000-word ASL dataset [Li et al., 2020], comprising of over 21,000 video samples isolated signed words.
2. Extract non-identifiable 3D features from the signing videos, to be released publicly, under creative commons licenses.
3. Optimize our current gloss-to-sign neural model;
4. Apply the dual machine learning paradigm to sign-togloss and gloss-to-sign tasks, so that the notion of duality can aid in the learning of each task.
5. Train the auxiliary gloss-to-English and English-togloss translation models, to extend the existing signgloss system using the continuous, fully-annotated ASL dataset, How2Sign [Duarte et al., 2021] comprising of 35,000 video samples of continuous signed phrases annotated with glosses.
6. Test the end-to-end process on ASLing [Ananthanarayana et al., 2021b], our own diverse, real-life ASLing dataset (consisting of over 1,200 video samples of signed phrases), collected in uncontrolled, unconstrained everyday situations at RIT. See Figure 1 for sample frames.

The novel contributions of this research thrust include the exploring innovative loss functions to improve the glossto-sign translation, taking advantage of word embedding arithmetic to recognize out-of-vocabulary words and using a dual learning approach to enhance both source-to-target and target-to-source translations.

3 Gloss is the transcribed form of sign language, which includes various notations to account for the facial and body grammar involved in the signs, but unfortunately, not all signs have a direct meaning in the spoken equivalent.

4 https://github.com/facebookresearch/fastText

5 https://code.google.com/p/word2vec/

Past Related Works in Sign Production

Neural machine translation (NMT) [Bahdanau et al., 2014], that addresses the translation of one spoken language to another has been the precursor to computational sign language understanding. Rastgoo et al. [Rastgoo et al., 2021a] provide an extensive survey of current neural-based sign recognition or sign-to-gloss methods.

But given the significant advances reported for the sign-togloss/text recognition task, we focus our research here more on sign production, or gloss-to-sign generation, a more challenging AI problem. One of the earlier sign production models, Text2Sign [Stoll et al., 2020], involved the use of Generative Adversarial Networks (GANs) to generate sign videos. Zelinka et al. [Zelinka and Kanis, 2020] created word-level signs using Openpose [Cao et al., 2021] sequences. Saunders et al. [Saunders et al., 2020] generated 3D continuous signs and also created a multi-channel approach using mixture density networks [Saunders et al., 2021]. A GAN based method was used to learn co-articulations in [Saunders et al., 2022].

From our previous works in sign language modeling [Ananthanarayana et al., 2021c; Ananthanarayana et al., 2021a; Wilkins et al., 2020], we developed various architectures for sign language recognition, which we will use in this research. In our most recent work [Ananthanarayana et al., 2023], we developed a continuous text-to-sign model and more recently, we presented a first approach at dual learning for 2-way sign language translation [Chaudhary et al., 2023]. Rastgoo et al. also provide a survey of current neuralbased sign production or gloss-to-sign methods [Rastgoo et al., 2021b].

Current State of Computational Sign Language Research

To explore the extent of sign language understanding as an area of technical research, we mined text data from IEEE Xplore digital library 6 .

Although other technical document libraries exist, these IEEE Xplore numbers show the trend in the research. Figure 2 shows the progression of the total number of sign language related technical publications through the years, obtained by searching IEEE Xplore for titles containing the phrase sign language. The dark blue bars in the chart indicate the number of publications (within those returned for the sign language) that contain the word production, generation or animation, which have to do with sign generation, a main focus of this work. The highest number of papers published was 201, in 2021, compared with 5,860 - the number of titles with the word speech in the same year; the highest number for sign generation was 5.

6 The IEEE Xplore is a resource for scientific and technical content providing web access to more than five million publications in electrical engineering, computer science, and electronics.

Sign language as a technical AI research area is only just budding. This is likely due to recent advances in deep learning technologies that have resulted in significant improvements in spoken language translation [Sutskever et al., 2014; Vaswani et al., 2017], video captioning [Venugopalan et al., 2015; Aafaq et al., 2019], visual storyboarding [Chen et al., 2019] and gesture recognition [Karpathy et al., 2014; Pigou et al., 2018]. There have been many other works in these areas, and these are a few of the pivotal works cited.

Text-to-sign Generation Model Although the generation of continuous ASL from its spoken counterpart using neural models, continues to make progress, its rate of success is slow compared to how fast individual ASL sign recognition is progressing [Adaloglou et al., 2022]. We therefore propose a new 2-stage approach to address text-to-sign language translation, where the first phase involves a text-to-gloss translation, and the second stage will involve single word to sign translation.

Figure 3 shows the proposed architecture where the two neural translation models are variants of the traditional encoder-decoder transformers [Vaswani et al., 2017]. We will initially train the model on the 2000 words fromWLASL [Li et al., 2020] and other word-level datasets.

So why not just have a lookup table for 2000 words? By training the model to specifically learn the mappings between word embeddings and signs, we can extrapolate signs from unseen words, under the assumption that words that are close in the word embedding space will be close in the sign space.

Motivated by our previous work in continuous sign language generation, we introduce a hybrid loss consisting of the traditional regression loss and a novel triplet loss for recognition.

L2 Regression loss (La) The objective here is to learn the probability p(V |W) of producing a sequence of pose frames V = (s1, . . . , sT ) over T time steps, given a spoken/written language word W.

Given the text word embedding W as input, the decoder will output a sequence of pose frames that can be expressed as ˆ s1:T = ˆ s1, . . . , ˆ sT . The Mean Squared Error (MSE) loss between the predicted sequence, ˆ s1:T , and the ground truth,sT1:T

Sign similarity metric-based loss (Lb) For this loss, we are interested in ensuring that the pose frames predicted by the architecture are as similar to the ground-truth signs as possible, and as distant as possible to other signs in the same training batch. To accomplish this, we have:

where B is a baseline sign, T is a truth sign required to be as similar to B as possible and S is a false sign (not as similar to the baseline); d(.) is the distance function. To avoid a trivial solution where our function f(.) = 0 , or f(B) = f(T), we define a distance function such that

We refer to the loss derived based on this distance as the sign similarity metric-based loss function.

The sign similarity based loss over all M samples can thus

Total loss: The overall architecture is therefore trained using a weighted combination of the regression and metricbased losses: LWord2Pose = λaLa +λbLb. . We will also explore the Connectionist Temporal Classification (CTC) loss, given that the ordering of the predicted and ground-truth signs would be similar.

The novelty here arises from the fact that the proposed model considers the input word embedding during training, therefore, for inference, if it encounters a word that it is not familiar with, as long as that word is close enough in word embedding space to another word that the model has encountered, it will generate a sign to potentially preserve the meaning of the seen word, given the novel one.

Dual learning for 2-way translation We will use a modified version of our previously developed sign-to-text model [Ananthanarayana et al., 2021a] (the primal task), which at the time was SoTA algorithm on the RWTH PHOENIXWeather-2014T benchmark dataset [Necati Cihan Camg¨oz, 2018] and combine this with our new gloss-to-sign model described above (the dual task), in a a dual learning mechanism motivated by [He et al., 2016].

If we define a sign phrase as x and its textual translation as y, then for a bilingual sign-text sentence pair (x, y), ideally p(x, y) = p(x)p(x|y) = p(y)p(y|x).

If the two models are only trained apart, it becomes challenging to satisfy p(x)p(x|y) = p(y)p(y|x); but joint training of the two models can be performed as:

where we ˆ p(x) and ˆ p(y) can be viewed as empirical statistics of the data. We anticipate that this form of joint training will enhance both the sign-to-text and text-to-sign translations.

Experimental Plan and Asessment of Outcomes

We will use the annotated continuous datasets to train a text-to-gloss model, and use the isolated sign datasets to train for gloss sign production. We will test the full text-to-sign model on the non-glossed continuous datasets with captions but no gloss. These will be useful for the overarching text-to-sign translation.

Using back translation 7 , we will run the generated signs thorough SoTA pose-to-text models and evaluate the resulting phrases via BLEU 8 and ROGUE 9 metrics. Empirically, a translation system is understandable and has good translations when its BLEU 1 and ROGUE scores are between 30 and 40; hence, this is our target metric. We will pay particular attention to corresponding BLEU 4 scores, to ensure it falls between 10 and 20 or higher.

4 Project Implementation Plan

In this section, we discuss the various stakeholders of the project, the NGO in Nigeria that will be critical to the launch of the project in Nigeria, the details of how to translate the research to practice, and evaluation of the project implementation.

4.1 Project Team - Roles and Responsibilities

Ifeoma Nwogu works on AI based human behavior modeling and computational sign language understanding [Ananthanarayana et al., 2021c; Ananthanarayana et al., 2021a; Wilkins et al., 2020]. As a faculty member at both UB and RIT, (UB as an Associate Professor in CSE and Adjunct Professor in Computer Information Science PhD School at RIT), she is very familiar with research processes at both institutions. Her prior work in modeling dynamical systems relating to human behavior will be well suited for this research.

7 Back translation is the process of re-translating content from the target language back to its source language in literal terms.

8 BLEU stands for BiLingual Evaluation Understudy, a metric that evaluates how good a translation is by comparing the predicted text to its ground truth equivalent. BLEU 1 - BLEU 4 scores evaluate the performance based on 1-gram (individual words) to 4-gram words (group of consecutive four words).

9 ROGUE stands for Recall-Oriented Understudy for Gisting Evaluation [1]. It is a set of metrics that compare an automatically produced summary or translation with human-produced references.

Nwogu will be responsible for the overall project coordination both in the US and in Nigeria.

Roshan Peiris is a faculty member in the Golisano College of Computing and Information Sciences at RIT. His research focuses on human computer interaction, including designing, developing and deploying technologies for disabled individuals especially hapticsdevices for visually impaired individuals [Teh et al., 2008]. For this project, Peiris will be responsible for the front-end usability design of the interpretation device. He will also lead the initial efforts in requirements gathering.

Karthik Dantu is a faculty member in the CSE at UB, and has worked extensively on perception and coordination in multi-robot systems and mobile systems. This includes multi-modal Visual SLAM [Adhivarahan and Dantu, 2019], permanence reasoning, resource constraints in Visual SLAM and semantic mapping. He was part of the team that helped write the five-year US roadmap in robotics [Christensen et al., 2021] in 2020. For this project, Dantu will work closely with the AI team on the timely realization of the device.

Ruchi Gamta is currently a faculty member at NTID is a member of the Deaf community. She has done research in Special Education, Science Education and Secondary Education. Gamta was previously an Instructor in various science subjects at the Model Secondary School for the Deaf in Washington, D.C. Given her background in this area, her role in this project will be that of a subject matter expert.

Emmanuel Asonye is a Visiting Research Scholar at University of New Mexico. By training, Asonye is a Field Linguist specializing in the studying and documenting endangered sign languages around the world. He has also done extensive research to better understand the Deaf culture in Nigeria [Asonye et al., 2018]. Asonye is the CEO of S-DELI, the advocacy NGO in Nigeria, slated to host the project. For this project, he will assist in the data collection in Nigeria as well provide the platform to execute the project successfully there.

Victoria Adesina is the Principal of the DSDC in Kaduna, Nigeria, the case study for the proposed project. She has been with the school since 1999. Having being trained as an Educator from the University of Port Harcourt Nigeria, and taking a special interest in ASL, Mrs Adesina has worked with DHH children in the elementary and early secondary schools across Nigeria for over 30 years. She is a major stakeholder in this project, after having shared her vision of the DHH students being provided with the opportunity to communicate extensively beyond their time in school during the day.

4.2 S-DELI: Nongovernmental Organization

Save the Deaf and Endangered Languages Initiative (S-DELI) a community based nongovernmental organization (NGO) committed to the development of Deaf children and adults in Nigeria, founded by co-Author Emmanuel Asonye, a Field Linguistic whose research focus is on documenting endangered indigenous languages, with a special focus on indigenous sign languages in Nigeria. The organization consists of 16 active members and an Advisory board drawn from academic linguists and public school teachers and a computer scientist and other community activists focused on the empowering Deaf children in Nigeria.

Founded in 2018, S-DELI has embarked in six major Deaf student related projects involving state governments and other government parastatals in Nigeria. The most prominent recent project was the Indigenous Nigerian Sign Language Documentation Project (INSLDP) where the a 12-person team from the Kaduna State government, was trained for two days on sign language documentation. Dr Emma Asonye along with the President of the Nigerian National Association of the Deaf (NNAD) were among the trainers at the event. The role of S-DELI was to logistically support the project, providing accommodation (or lodging apartments) for the duration of the project, transportation to and from the collection sites, recording and storage equipment, paraphernalia such as t-shirts, etc. The S-DELI team facilitated any specific regional logistics requirements related to the project. Findings from the project will be used to in Dr. Asonye's research in indigenous sign linguistics. Other projects implemented by S-DELI involve community outreach projects such as the Beauty Beyond Speech, a campaign launched in 2017against all forms of sexual and domestic abuse against deaf girls in Africa, while advocating for their safety and education.

We expect to engage with S-DELI in a similar capacity as with the INSLDP project.

Project Tasks and Timeline

Project Timeline

4.3 Link fromResearch to Practice

Current State of the Technical Application

• Sign-to-text (S2T): For this branch of the project, we have currently implemented an extensive sign language translation application, the only known continuous ASL translation project in the literature in [Ananthanarayana et al., 2021b]. For this work, the current BLEU1 score was 22.39 and BLEU4 was 12.25. While these numbers appear low for real-life deployment, the limitation is due to the limited sizes of existing annotated continuous ASL datasets. When a similar algorithm was applied on the annotated German sign language dataset, RWTH PHOENIX-Weather2014T benchmark dataset [Necati Cihan Camg¨oz, 2018], which is about 8x larger than our ASLing dataset, the resulting BLEU scores were 5x and 3x, respectively. A larger training dataset will significantly improve the current S2T metrics.
• Text-to-sign (T2S): For this significantly more challenging branch, we have implemented a commercially available speechto-text product for spoken English speech transcription. We are currently able to successfully produce up to 2000 words from the ASL word-level dataset [Li et al., 2020], where these words overlap and are a significantly larger superset of the Swadesh list 10 . Speaking users are restricted to single words in this first iteration, due to the limitation of the technical solution which is currently only able to produce single words. The usability testing will determine whether the application can still be deployed in spite of this limitation.
• Avatar development for sign production: Animating avatars for sign language comprehensibility was studied in detail by [Kipp et al., 2011], where they describe methods for the creation and evaluation of a signing avatar. They described techniques useful for enhancing comprehensibility in a short development time and also argue for future research to focus on non-manual aspects and prosody, in order to reach the comprehensibility levels of human

10 The Swadesh list in classical linguistics, is a compilation of potentially universal concepts useful for learning and describing a language.

• signers. We will continue to iterate our signing avatar development, in a similar progression as laid out in the paper. Currently, we have a multi-racial, multi-gender, single word signing avatar, that displays facial expressions, though not yet coordinated with the manual signs. Feedback from the usability study will guide the next iterations of the avatar development.
• Usability test: A first usability test will be conducted once IRB approvals are obtained both at UB and RIT, for the series of human studies, including the study in Nigeria. The co-design usability study will include Deaf students, their hearing, non-signing peers, ASL interpreters, Deaf student instructors, and the computing graduate students involved in developing the solution. The goal of the studies will be to determine the aspects of the application that is ready for deployment while identifying others that we consider to be showstoppers. A showstopper is a userrequirement identified during testing without which the application can be successfully deployed. Some attributes to be tested will include the comprehensibility of the translation (either way), the appearance of the avatar, its scale and realism while signing, the ease of operating the tablet to be used, the optimal distance of a signer from the tablet camera, etc. Investigators Peiris and Gamta at RIT will lead the co-design efforts in Rochester, NY.
• First implantation test: The first experiment to be conducted from Kaduna, Nigeria, which we refer to as the implantation test will involve the collection of about 200 signed phrases from the local signers in Nigeria. This set of continuously signed phrases should capture the universal concepts for the regional sign language, and will be compiled by our collaborating Nigerian linguists including Dr. Asonye. After incorporating the feedback from the usability test, we will test the regional data set to observe the resulting BLEU1-4 scores. They should be in a similar range as the US-based ASL translation scores. Similarly, we will obtain a set of signed word videos of 200 words similarly identified by the Nigerian linguists. We will expect to obtain an accuracy of greater than 75% on the 200 regional words. We anticipate that this implantation test will highlight some deficits in the US-trained system which will have to be addressed before a full at-home deployment is conducted.
• Tool deployment: The tool will be deployed with the assistance of S-DELI and DSDC in Nigeria. From the DSDC, we will identify 20 families of Deaf children 3-7 year old in the early stages of language usage, where their vocabulary is still limited, and enroll their parents/caretakers in the study. Currently, DSDC conducts weekly sign language tutorials (sign speed and accuracy, sign recognition rates, etc) on Saturdays, for students' parents. We will leverage this established communication protocol to deploy the tool and train its users. Surveys will be given to measure the quantity and quality of communication taking place at home with the Deaf child. Ongoing feedback from the users will be incorporated into the research and development process.

Budgetary Requirements

Programmer to complete interface and GUI (Hourly at $25/hour for 40 hours) - $1000

Student researchers to develop the algorithms (1 grad students at $50k/year for 3 years); Hourly students for 500 hours of work per year at $16/hour; for 3 years - $174,000

Research Administrator in Nigeria for $40k/year - $120,000

ASL Consultant in US $12k/year - $36,000

Computational linguistic research and data gathering $25k/year - $75,000

• Data collection costs - videos, surveys, interviews, room rentals, etc. at $3,000 per year - $9,000
• Tablets for 20 families at $125 per tablet + participatory incentive costs $5 per family per quarter for 3 years - $3,700

Travel cost for 2 investigators to Nigeria for 5 days every year at $2500 per person per year - $15,000

TOTAL = $433,700

Note: Researchers' salaries and fringe benefits not included

Potential Problems

1. The most prevalent limitation of the project that we anticipate is from the automated interpreting avatar, which is limited in its performance. Sign language consists of manuals and nonmanuals, but the proposed interpretation tool is currently only taking manuals into consideration. Manuals are the linguistic features for sign language, which include the hands - its shape, location, orientation and movement [Neff et al., 2008]. Nonmanuals are the linguistic features which include facial expression, gaze and torso movements and these may stretch over several signs. The absence of nonmanuals for this deployment could present a risk in the adoption of the technology. To mitigate this, we have employed facial expressions in the avatars although they are not yet coordinated with the manual signs.
2. Although ASL is the foundational sign language used in teaching students in schools across Nigeria, there are regional "dialects" of ASL, as well as indigenous sign languages used in the home. Hence, attempting to introduce ASL initially trained on American subjects could be a limitation of the project - the language usage, word orderings and sign vocabularies could potentially be different. To mitigate this, as the system continues to be iteratively updated using data obtained from Nigeria, the effects of training originally only on the US data could be reduced over time.
3. The commercially available speech-to-text and text-to-speech converters are trained and designed to work with voice types primarily fromUS and Europe. It is not clear how readily these systems will be able to understand the accents and other language features in Nigeria. Although the initial sets of users will be family members who speak English, they do so with regional intonations and slang. Similarly, it is not yet known, how readily the regional participants will understand the spoken version of the signed phrases that were translated to text and then converted to speech again by commercial speech converters. To mitigate this, we will explore speech-text converters that can train on the users' voices in their initial stages.
4. The system is designed to translate spoken English to ASL and vice versa. But many DHH student's families, who are the target audience of the interpretation tool, are not native English speakers. Hausa is the main language spoken in Kaduna, the city where DSDC is located. Deploying the tool in homes where English is not spoken conversationally would be a limitation of the project. This can be mitigated by including an English to Hausa and Hausa to English speech translator, such as found in Google Translator. To initially work around this issue, our first cohort of families will be ones where the parents speak English, even if not as a native language.

5 Project Evaluation

Over the course of the three years of the project duration, the research team will continue to improve and deploy the tool to meet the defined usability standards in terms of it effectiveness, efficiency and user satisfaction.

The research staff will be trained to employ a basic general language assessment test, the Kendall P-Levels conversation proficiency test 11 [French, 1999]. This will be used assess how well the family members are improving in their sign language development. The test assigns one of eight grades from level P-Level 0+ (primary forms of communication include informal gestures, facial expressions, and differentiated cries), through P-Level 1 (the person calls attention to physical needs and expresses personal reactions. The person uses a similar behaviors as 0+, but can now imitate signs produced by others, though not initiating the signs themselves yet) . . . to P-Level 7 (the person can say what she has in mind to say without circling around it, can provide abstract, detailed information such as the rules of a complex game and can use other words to say the same thing so the other person understands, if needed).

Ethical Considerations

All human research protocols to be implemented in this work will be required to be approved by Institutional Review Board (IRB) before the research can go forward. All research administrators, both within and outside the US will be required to have appropriate certifications to carry out the research work.

During the field evaluation, although the control group will not receive access to the tool at the onset of the project, they will still be invited to participate in the hour-long ASL weekend tutorials offered on the DSDC premises. Families will not be kept as control groups for longer than one year. Lastly, since we plan to collect data from families of children with disabilities, we will doubly ensure that their identities are protected in any video recordings obtained.