Bridge to Non-Barrier Communication: Gloss-Prompted Fine-Grained Cued Speech Gesture Generation with Diffusion Model

1 Introduction

According to the World Health Organization (WHO), more than 5% of the global population (466 million) suffers from the hearing loss. As a predominant communication method for hearing-impaired people, Lip reading [Puviarasan and Palanivel, 2011; Fernandez-Lopez et al., 2017] has a major defect of visual confusion. For instance, it struggles to differentiate pronunciations with similar labial shapes, such as [u] and [y], posing challenges for hearing-impaired individuals in accessing spoken language through conventional education.

∗Work done during internship at Tencent AI Lab.

†Corresponding Author: avrillliu@hkust-gz.edu.cn.

To tackle the limitations of lip reading, and to improve the reading skills of individuals with hearing impairments, in 1967, Cornett introduced the Cued Speech (CS) system [Cornett, 1967], which employs several hand codings (i.e., finger shapes and hand positions) to complement lip reading, providing a clear visual representation of all phonemes in spoken language [Puviarasan and Palanivel, 2011; Fernandez-Lopez et al., 2017]. For instance, in Mandarin Chinese CS (MCCS) [Liu and Feng, 2019] (see Fig. 1(a)), it utilizes five hand positions for encoding vowel groups and eight finger shapes for encoding consonant groups. With CS, individuals with hearing impairments can differentiate sounds that might appear similar when observed on lips by incorporating hand information. Another widely adopted communication method is Sign Language (SL) [Stokoe, 2005; Liddell and Johnson, 1989; Timothy, 2003]. It is crucial to emphasize that CS is not a visual language like SL; instead, it is a coding system of spoken language [Cornett, 1967]. In addition, studies indicate that CS can be learned much more quickly than SL [Reynolds, 2007]. Given that CS can effectively promote non-barrier communication, audio/text to CS gestures video generation draws researchers' attention. It should be noted that comparing to text, CS is more friendly and more easily adopted by the hearing impaired who are illiterate [Cox et al., 2002; Power et al., 2007].

The multi-modal CS gesture generation is a challenging task for the following reasons: 1) high requirement for finegrained and accurate gesture generation, as shown in Figure 1(a), where nuances in the hand's position and fingers' shape lead to quite different semantic meanings; 2) the limited size of CS datasets and expensive annotation cost of complicated fine-grained CS gestures. To address these challenges, we design a novel Gloss-Prompted Diffusion-based CS Gesture generation framework (GlossDiff). Specifically, we first propose a CS gloss, which is a direct motion instruction for bridging the gap between spoken language and CS gestures. It is automatically generated by LLM based on the encoding rule of CS in Figure 1(a). As shown in Figure 1(b), when expressing the word "tree", which is pronounced as "/ù/ /u/" in Chinese, we generate the intermediate instruction text (i.e., gloss) to describe the process of using CS gestures to express this word, i.e., Stretch out two fingers apart without closing them and place the hand near the neck. Besides, we design a Gloss-Prompted Diffusion Model that can generate accurate hand and finger movements.

Moreover, rhythm is a critical paralinguistic information in spoken language. As a coding system for spoken languages, we suggest natural rhythm dynamics should also be considered as a very important feature for CS's complete semantic expression. More specifically, the rhythm here refers to the ability to generate multi-modal CS speech gesture movements (i.e., hand and finger movements), which match the phoneme durations and utterance prosody of speech. Unfortunately, previous works have not pay enough attention to this. To this end, we propose an Audio-driven Rhythmic Module (ARM) that considers the overall rhythm of the CS movements aligning with speech signals. We leverage the large-scale WavLM [Chen et al., 2022] to extract audio features, which we demonstrate outperforming traditional MFCC features.

We summarized our contributions as follows: 1) A novel GlossDiff framework that simultaneously generates finegrained hand position, finger movements, and lip reading in CS. Specifically, we introduce a CS gloss, which establishes a direct link between text/audio and CS hand movements, enabling more specific prompts for an accurate fine-grained CS gesture generation. 2) A new module ARM that improves the overall rhythm of the CS movements. 3) Publication of the first multi-cuer large-scale Mandarin Chinese CS (MCCS) dataset, which contains four cuers 1 and 4000 CS videos. 4) Extensive experiments conducted on the MCCS dataset show the proposed GlossDiff achieves SOTA performance under different metrics. The qualitative and ablation studies, as well as user studies, further verify the effectiveness of the proposed model.

1 The people who perform CS are called the cuer

2 RelatedWork

2.1 Cued Speech Generation

In prior work, early attempts at CS gesture generation [Duchnowski et al., 1998; Bailly et al., 2008] are mainly rulebased. Notably, in [Duchnowski et al., 1998], specific keywords were manually selected, along with low-context sentences [Rothauser, 1969], and manual templates for corresponding hand gestures were predefined. This processing involved CS recognition, followed by mapping the recognized text to the hand templates. However, this method relied heavily on hand-crafted designs, which constrained the expressiveness of CS gestures and increased the required manual effort. In [Bailly et al., 2008], a post-processing algorithm was introduced to refine synthesized hand gestures, including adjustments for hand rotation and translation. However, this approach required prior human knowledge to adapt the algorithm to new images, resulting in limited robustness. To the best of our knowledge, there is still a gap in research about end-to-end deep learning-based CS gesture generation.

2.2 Co-speech and Sign Language Generation

The generation of Co-speech gestures involves generating body movements corresponding to audio input. Previous studies mainly developed large speech-gesture datasets to learn how speech audio maps to human skeletons using deep learning, as in [Ao et al., 2022]. To make gestures more expressive, some methods use Generative Adversarial Networks (GANs) for more realistic results [Ginosar et al., 2019; Youngwoo et al., 2020]. Recently, diffusion models like DiffGesture [Zhu et al., 2023], effectively links audio and gestures while keeping time consistency, allowing for highquality Co-speech gestures. However, Co-speech gesture generation focuses on fluency and style rather than gesture fine-grained accuracy. Existing methods cannot generate accurate subtle CS hand gestures.

In the literature, there are several Sign Language (SL) generation methods: 1) The Neural Machine Translation approach from [Stoll et al., 2020] sees SL generation as translation, using neural models to process SL text. 2) The Motion Graph method in [Stoll et al., 2020] uses motion graphics to make a directed graph from motion capture data for SL creation. 3) Conditional generation methods, like GANs and VAEs, are also used for SL gestures. 4) Some researches have introduced transformer-based models for SL, as mentioned in [Ben et al., 2020]. Despite these advancements in SL gesture generation, applying these methods to CS gestures has limitations. Firstly, CS gesture generation necessitates more precise methods to achieve complex finegrained gesture generation, while SL gesture generation is more coarse-grained. Secondly, SL gesture is not related with lip-reading, thus cannot match the speech rhythm and gesture-speech asynchrony characteristics [Liu et al., 2021; Liu and Liu, 2023] in CS gesture generation.

2.3 NeRF and Diffusion-based Gesture Generation

NeRF is a novel technique in 3D modeling, which effectively creates highly detailed and photo-realistic static scenes from 2D images. Its application has been extended to generating life-like talking head models [Guo et al., 2021], demonstrating NeRF's capability in handling subtle facial movements and expressions. However, the application of NeRF in fullbody gesture generation is relatively limited. Additionally, the high requirements for data and computation further constrain its use in CS gesture generation.

Currently, in the field of human gesture generation, diffusion models [Ho et al., 2020] are predominantly used in two main applications: generating comparably large body movements (e.g., human walking) [Tevet et al., 2022b; Zhang et al., 2022; Zhao et al., 2023; Ao et al., 2023] and generating poses in Co-speech scenarios [Ji et al., 2023; Zhi et al., 2023; Yang et al., 2023]. However, the existing approaches lack the capability to tackle fine-grained gesture generation. Additionally, they primarily focus on body poses without lip movements. Lastly, their diffusion models require extensive training data, which is not feasible given the limited dataset in our CS scenarios.

3 Method

In this section, we provide a comprehensive description of our proposed method, GlossDiff, designed for rhythm-aware CS gesture generation, which seamlessly integrates domainspecific knowledge for CS generation. As shown in Figure 2, our GlossDiff framework consists of three primary components: the knowledge infusion module, the rhythmic module, and the Diffusion-based generation module.

3.1 ProblemFormulation

Automatic CS gesture generation involves generating the corresponding landmarks sequence of CS gesture M ∗ , given an audio signal A and the text T . In the task of automatic multimodal CS gesture generation, the combined features of A, T , and the generated rhythmic information are input into the CS gesture generator. The final CS gestures (M ∗ ) including lips, fingers, and hand positions are obtained by minimizing:

where L represents the frame count of the current CS video. The ground truth CS gesture landmarks Mi in the i-th frame of the CS video is obtained by the Expose method [Choutas et al., 2020]. M ∗ i = ˆMi+˜ Mi, where ˆM = GD(T,A) are generated semantic gesture landmarks representing the corresponding generated gesture in the i-th frame. GD is the diffusionbased semantic gesture generator. Additionally, ˜ M = GR(A) is the rhythmic information derived from the correspondinng audio speech, with GR as the rhythm generator.

3.2 Knowledge InfusionModule

The primary objective of the knowledge infusion module is to transform spoken language text T (i.e., the speech transcription) into direct text instructions (i.e., gloss, see Figure 1(b)), which describe the corresponding fine-grained CS motions. To achieve this, we leverage the LLM, i.e., ChatGPT4 [OpenAI and et.al, 2023], the prompt engineering approach to infuse the encoding rules of Chinese CS [Liu and Feng, 2019] into our framework by the following:

where P is our designed prompt based on CS domain knowledge (i.e., prior transformation rules of CS based on [Liu and Feng, 2019]), and T is the input text. Ultimately, this process enables the transformation of our indirectly semantic-related text into directly semantic-related gloss.

3.3 Diffusion-based GenerationModule

Gloss-based Motion CLIP Fine-tuning

MotionCLIP [Tevet et al., 2022a] is a multimodal large-scale model specifically designed for generating general motion gestures. To obtain an accurate feature embedding of CS gloss, we leverage the MotionCLIP as our pre-trained model, and fine-tune it using the generated CS gloss (introduced in Subsection 3.2) and the paired CS gestures.

As for the fine-tuning stage, we adopt CLIP-style contrastive learning [Radford et al., 2021] to fine-tune the encoders with CS data. Given a batch of pairs containing CS gesture motion and gloss embeddings, denoted as B = {(zm i , zg i )}Bi=1 , where B is the batch size. Em and Eg are the corresponding MotionCLIP encoder for both motion sequence and gloss. Z m = Em(M), Z g = Eg(g). The goal of the training is to maximize the similarity between paired zm i and zg i of in the batch while minimizing the similarity of the incorrect pairs (zm i , zg j )i̸=j . A symmetric cross entropy (CE) loss LCE is optimized over these similarity scores. Formally, the loss is:

where y specifies the true correspondence between the gestures zm i and gloss zg j in the training batch B. If they are paired, y = 1, otherwise, y = 0. p is defined as:

where η is the temperature of softmax, and pg (zg j ) follow the same computations.

Gloss-Prompted Diffusion Model

To generate CS gesture video, we propose a Gloss-Prompted Diffusion Model. More precisely, the semantic hand gesture generator GD is designed based on the latent diffusion model [Rombach et al., 2022], which applys diffusion and denoising steps in a pre-trained latent space. The latent diffusion model is trained with the standard noise estimation loss [Ho et al., 2020] defined as:

where Zn is the latent CS gesture at each time step n. A is audio speech, and g is the generated gloss. ϵ is the ground truth noise and ϵθ is the noise predicted by latent diffusion model, where θ is the parameters of latent diffusion model.

To inject the information of the gloss prompts into the diffusion network, we employ an adaptive instance normalization (AdaIN) layer [Huang and Belongie, 2017]. Specifically, we leverage the fine-tuned MotionCLIP gloss encoder Eg to convert the gloss prompt into a gloss embedding zg . Then, we learn a MLP network to map the gloss embedding zg to parameters that modify the per-channel mean and variance of the AdaIn layer.

To train our Gloss-Prompted Diffusion Model, we employ classifier-free guidance as detailed in [Ho and Salimans, 2022]. Specifically, during training, we enable the diffusion model GD to master both the semantic conditional and unconditional distributions by randomly configuring g = ∅. This action effectively deactivates the AdaIN layer with a probability of p during the training phase, which is set to 10% [Tevet et al., 2022b]. During inference, the anticipated noise is calculated using:

After obtaining the predicted noise ϵ∗n , the model operates in a reverse step-wise manner over N time steps, updating a latent gesture sequence Zn at each time step n. It begins by generating a sequence of latent codes ZN ∼ N(0, I) and subsequently calculates a series of denoised sequences Zn through the iterative removal of the estimated noise ϵ∗n from Zn (n = N - 1, . . . , 0). Z0 is the final generated CS gesture latent embedding through N reverse diffusion steps. Z0 is fed into a Transformer-based decoder [Petrovich et al., 2021] to generate semantic CS gesture motion ˆM.

3.4 Audio-driven Rhythmic Module

In CS gesture generation, it's not just the accurate positioning of the gesture that matters; the natural rhythm of gesture motion plays a crucial role. We believe that the audio speech signal contains not only the semantic information but also the rhythmic dynamics of CS, which significantly contributes to achieving visual and auditory coherence.

To address this, we introduce a novel Audio-driven Rhythmic Module (ARM), designed to capture the rhythmic dynamics of gestures. This module employs three convolution layers as a rhythmic dynamics generator GR, further aligning the motion dynamics with the CS rhythm.

Existing research (e.g., WavLm and AudioLDM) [Lebourdais et al., 2022; Liu et al., 2023] have shown that compared with MFCC features, audio features extracted by the large pre-trained model have a stronger expressive capability and can avoid information loss. Without loss of generality, in this work, we use the encoder of WavLM, denoted as EA to extract audio features to prevent information loss, thereby preserving richer and higher-dimensional rhythmic information.

To handle the lip-hand synchronization issue [Liu et al., 2021] in CS, we reformulate the task as one of determining the motion magnitude for each frame within consecutive motion sequences. Unlike methods that attempt to enforce perfect alignment between generated gestures and speech, our approach implicitly learns how to produce asynchronous gestures that correspond to the input speech. Rather than directly controlling the gestures of each individual frame, we focus on regulating the overall rhythm of a motion sequence.

The loss function for the ARM is defined as:

where ¯M represents the average motion within the set of generatedmotionsM. The difference betweenM and ¯M quantifies the magnitude of hand and finger movement. The purpose of Lrhythm is to ensure that the generated ˜ M = GR(EA(A)) maintains the natural offset relative to the mean gesture. EA is the encoder of WavLM. This offset helps in generating motion dynamics for a natural, non-mechanical movement without disrupting the semantics of the CS gesture. We demonstrate the efficacy regarding rhythm quality and naturalness with quantitative result in Sec. 4.3, as well as qualitative result of in Sec. 4.4.

Novel Quantitative Rhythmic Metrics

In this work, for the first time, rhythm is investigated as an important paralinguistic feature to improve CS' communication efficacy. To capture the unique asynchronous dynamics between lip and hand movements in CS scenarios, we propose a novel metric, Gesture Audio Difference (GAD), to evaluate the rhythmic synchronization of the generated gestures. This metric is defined as follows:

where M and A represent the CS gesture and audio speech, respectively. The term N denotes the number of annotated temporal segments, which are equal for both speech and gesture. The variable Ui refers to the middle time instant of a segment, indicating a specific moment when a gesture or speech occurs. The function 1 is an indicator function, mapping elements within the subset (satisfying ||U Mi -U Ai ||1 < τ) to one, and all other elements to zero.

Taking the asynchrony between audio speech and CS hand movements into consideration, we introduce a threshold τ, which ensures their alignment and is empirically determined based on a statistical study of the hand preceding time [Liu et al., 2020].

3.5 Training of GlossDiff Framework

We employ a semantic loss to ascertain the semantic accuracy of the final generated gestures. To be specific,

where cos(·, ·) represents the cosine distance, while Z0 and Z ∗ 0 denote the final generated CS gesture latent embedding and the ground truth CS gesture motions, respectively.

Following the existing training procedure for denoising diffusion models, we optimize the following loss:

where α is the weight of Lnoise (in Equation (5)), β is the weight of Lsemantic (in Equation (9)), and γ is the weight of Lrhythm (in Equation (7)).

4 Experiments

4.1 MCCS Datatset

Previously, only two CS datasets were available for public access: One was in French 2 [Liu et al., 2018], consisting of recordings of a single cuer delivering 238 sentences; The other was in British English 3 [Liu et al., 2019], similarly featuring a single cuer reciting 97 sentences. To remedy the scarcity of Chinese CS data, we built in this work, for the first time, a large-scale Mandarin Chinese CS dataset that includes contributions from four CS cuers, called MCCS.

We first select 1000 text sentences following the below principles: (1) They cover common scenarios in daily life, including colloquial dialogues, more formal words, as well as written words. (2) The materials aim to cover possible syllable combinations. All in all, our text album covers 23 main topics, 72 subtopics, and the most commonly used 399 Mandarin syllables. It comprises a total of 1000 sentences, 10,482 words with an average of 10.5 words per sentence. The shortest sentence contains 4 words, while the longest has 25 words. Then, we recorded CS videos for each of the four cuers performing the 1000 sentences, resulting 4000 sentences in total.

All videos are recorded using either a camera or a mobile phone in landscape mode, The four cuers have received systematic training to ensure they can perform Mandarin Chinese CS smoothly and accurately. Note that our dataset has been collected with the explicit consent of the individuals involved and is eligible for open source.

4.2 Experimental Setup

During the training phase, we pre-train the motion clip first and then follow an end-to-end pipeline to train the latent diffusion model. The experiments are implemented using PyTorch, with four A6000 GPU cards for model training. During the inference phase, we use the latent diffusion model to generate CS gestures. The training and test data are randomly split as 4 : 1. The number of diffusion steps is 1000, and the training batch size is 128. The weight of loss items is set to α = 1, β = 0.2 and γ = 0.1.

Evaluation Metrics

The conventional evaluation metrics of the generated gestures contain three classes: Percentage of Correct Keypoint (PCK) [Yi and Deva, 2013], Fr´echet Gesture Distance (FGD) [Youngwoo et al., 2020], Mean Absolute Joint Errors

2 https://zenodo.org/record/5554849#.ZBBCvOxBx8Y

3 https://zenodo.org/record/3464212#.ZBBAJuxBx8Y

(MAJE) [Youngwoo et al., 2020], and Mean Acceleration Difference (MAD) [Youngwoo et al., 2020]. In addition, to further measure the unique asynchronous dynamics between lip and hand movements in CS scenarios, we use the novel metric GAD as described in Sec.3.4 to evaluate the rhythmic synchronization of the generated gestures.

4.3 Quantitative Result and Analysis

Comparison with SOTA

We compare our approach with four recent gesture synthesis methods, i.e., Speech2Gesture [Ginosar et al., 2019], Gestures from Trimodal Context (GTC) [Youngwoo et al., 2020], HA2G [Bhattacharya et al., 2021], DiffGesture [Zhu et al., 2023]. We take DiffGesture as the SOTA method among these approaches, since it achieves the best result on the TED Gesture datasets [Youngwoo et al., 2019].

Table 1 provides a detailed comparison among our methods and the previous methods on the MCCS datasets. Our method GlossDiff gives the best results in PCK, MAJE, MAD, and GAD metrics, most of which have a wide superiority leap comparing to the reference systems. The results demonstrate a higher quality of fine-grained gesture generation by our proposed system. The only exception is one FGD score that slightly trails the SOTA method, while it surpasses all other reference methods. Notably, our method's PCK values are significantly higher than other methods, showing its effectiveness in fine-grained generation. Moreover, our method excels in rhythm performance, achieving the highest GAD values. This superiority on GAD metrics demonstrates that our method can effectively capture the rhythm in CS gesture.

Ablation Study

We provide the ablation study for three modules in Table 1. The term "Gloss-prompt" indicates the integration of a Gloss Knowledge Infusion Module. "WavLM" refers to using features extracted from the pre-trained large-scale speech model wavLM instead of conventional MFCC. "Gloss-CLIP" denotes the incorporation of Gloss-based Motion CLIP Finetuning. We can observe that the absence of any module leads to a decline in performance metrics, demonstrating the efficacy of each module in our framework. Specially, the absence of the Gloss-prompt and Gloss-CLIP modules results in a decrease in PCK by 1.85% and 1.26%, respectively, highlighting their critical role in fine-grained generation.

4.4 Qualitative Result and Analysis

Visualization of Generated Fine-grained CS Gesture

Figure 3 shows fine-grained hand gestures generated with gloss prompts, where each row shows the detailed gloss of different body parts and their gesture sequences. We used arrows to indicate lip movement trends, red circles for finger shape transformations, and red stars for hand position shifts, including their movement directions. The first row in the figure shows the lips' contour expanding as the gloss input. The second row emphasizes finger shape changing aligned with detailed finger gloss. In the third row, there are subtle hand position shifts, marked by red stars moving from near the mouth to the chin area, showing our method's effectiveness in using detailed gloss to guide CS gesture generation.

Distribution of Fine-grained Gesture Feature

To visualize the generated CS gesture in the feature space, we used t-SNE [van der Maaten and Hinton, 2008] for dimension reduction. We uniformly select frames from the generated CS sequences and extract the hand gesture features corresponding to the text. Recall that, as depicted in Figure 1, the MCCS incorporates 8 distinct finger shapes to signify the 24 consonants of the Chinese language, along with 5 hand positions to denote the 16 vowels. In the left part of Figure 4, the 8 distinct clusters are separate, with each cluster corresponding to a set of finger shapes (where each color represents a different consonant group). Some clusters that are very close in distance have similar finger shapes, such as shape8 and shape6, as well as shape2 and shape7. This visualization validates the effectiveness of our method in capturing the fine-grained semantics of CS hand and finger shapes. On the right side of Figure 4, We can find different hand positions have differences in features, but there is more overlap among the clusters, which means they are not as distinctly differentiated in feature-level as finger shapes.

Visualization of Generated CS Gestures

Figure 5 compares the visualization results of our method with the SOTA method, DiffGesture. This comparison includes the gestures' corresponding audio, text, and ground truth video frames. We highlighted corresponding phonemes in red and used red stars and circles to indicate hand locations and finger shapes, respectively.

Our method shows a noticeable improvement in gesture accuracy, particularly in fine-grained details. For example, our index finger shape is more precise than the SOTA method, as seen in the first column. In the second column, our method accurately places the hand beside the face, unlike the SOTA method's placement beside the eye. The fourth column illustrates our method's superior precision in thumb position and overall gesture alignment with the ground truth, showing greater adherence to CS rules and enhanced detail accuracy.

User Study

We conduct a user study to evaluate CS gestures generated by our method compared with SOTA and the ground truth. This study involved 10 groups of videos, each with a groundtruth CS gesture video, videos generated by the current SOTA method (DiffGesture) and our method (GlossDiff). All videos were randomly shuffled. Ten subjects trained in CS were asked to rate the CS gesture videos from three perspectives: accuracy, rhythm quality, and naturalness, each with a score ranging from 0 to 10 (the higher the better). We calculated average scores and confidence intervals for each case.

It is shown in Figure 6 that our method surpassed the current SOTA DiffGesture in all three metrics, getting closer to the ground truth. This demonstrates our method's ability to produce more accurate and natural CS gestures, especially in rhythm quality, attributed to the proposed ARM. Our approach notably outperforms the DiffGesture in accuracy, proving its effectiveness in fine-grained gesture generation.

5 Conclusion

We introduced a novel GlossDiff framework that effectively generates fine-grained CS gesture sequences. We have proposed a gloss knowledge infusion module and an audio rhythm module for an accurate and natural CS gesture video generation. Additionally, we contributed the first large-scale MCCS dataset. Extensive experiments on MCCS demonstrate our approach's efficacy, surpassing current SOTA methods. Qualitative experiments and ablation studies validated our system's overall effectiveness as well as each individual module's. Future work aims to infuse CS video generation with prosody and emotion. The Automatic Prompt Engineering (APE) is also a promising direction to improve gloss quality.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62101351), Guangzhou Municipal Science and Technology Project: Basic and Applied Basic research projects (No. 2024A04J4232) and Tencent AI Lab Rhino-Bird Program (No. RBFR2023014).