Contrastive Learning for Sign Language Recognition and Translation

1 Introduction

Sign language research is committed to transferring sign video into natural language. Considering the unique grammatical rules and linguistic characteristics of sign language, current researchers mainly focus on two tasks: Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT). CSLR aims at recognizing continuous signs as the corresponding gloss sequence, and it is a weakly labeled recognition task (i.e., source and target sequences have unequal lengths but the same order). Differently, SLT aims at translating sign language into spoken language, and it is a sequence-to-sequence (seq2seq) translation task (i.e., source and target sequences have unequal lengths and different orders). Due to the difference between these two tasks, researchers often adopt Connectionist Temporal Classification (CTC) loss [Graves et al., 2006] to address CSLR [Zhang et al., 2019] while adopting seq2seq architecture [Sutskever et al., 2014] to address SLT [Jin and Zhao, 2021].

∗Yafeng Yin is the corresponding author.

Most current end-to-end sign language processing models can be simplified as Figure 1, and it consists of a visual module which mainly includes 1D, 2D, 3D CNN layers for mapping sign video into visual features, a temporal module which mainly includes RNN-like/transformer layers for exploring global context features. Besides, a unique CTC module is adopted to get the gloss sequence for CSLR, or a unique semantic module which mainly is RNN-like/transformer decoder is adopted to get the spoken sentence for SLT. However, there are two widely existing problems in this architecture: (1) For CSLR, CTC loss will bring the spike phenomenon [Graves et al., 2006], i.e., the model tends to classify frames as 'blank' tokens rather than meaningful gloss tokens, leading to the fact that only a few frames are useful for training visual module. Consequently, the representational ability of visual module is not fully exploited and further visual module may not provide efficient features during testing. (2) For SLT, teacher forcing (often used to train seq2seq model) leads to exposure bias which causes distributional shifts during testing [He et al., 2021], i.e., the model in training stage predicts the next word based on ground-truth words, but the model in testing stage needs to make the next prediction based on previous words generated by itself. Thus the probability distribution generated by model in testing will gradually deviate from that in training, which leads to performance degradation. Besides, video-gloss or video-sentence pairs in current sign language datasets are scarce to fully train each module in above architecture, which further exacerbates these two problems.

To tackle CTC spike phenomenon which may weaken visual representational ability, Hao et al. and Min et al. explicitly mentioned this issue and added extra constraints through knowledge distillation to enhance visual representation. Most other work implicitly tackled this issue by manually enhancing visual representational ability. They intended to design various visual modules by injecting extra clues, including local areas [Zhou et al., 2020], skeletons [Gan et al., 2021] and depth images [Tang et al., 2021]. As for exposure bias caused by teacher forcing, there are many attempts in Natural Language Processing (NLP) area while little work has paid attention in SLT. Besides, when considering data scarcity problem in CLSR and SLT, methods like back-translation [Zhou et al., 2021a], cross modality augmentation [Pu et al., 2020] and pre-trained models [Chen et al., 2022] had been proposed.

In this paper, we provide a simple unified framework by adopting contrastive learning to alleviate CTC spike phenomenon and exposure bias problem. The main intuitions are: (1) Although video-gloss or video-sentence pairs are limited and labeling each frame for fully training visual module is unrealistic, frames provided by datasets are rather a lot. Thus we design a visual contrastive loss to minimize feature distance of different augmented samples of frames in one sign video. Then the visual module can improve its representation by utilizing numerous unlabeled frames in an unsupervised way. (2) Distributional shift caused by teacher forcing can be alleviated by exposing model to its own mistakes. Thus we design a semantic contrastive loss by comparing semantic features of different inputs (ground-truth sequence and the predicted sequence). We make the following contributions:

• To alleviate CTC spike phenomenon in CSLR and enhance visual-level feature representation, we propose a visual contrastive loss to explore features by utilizing numerous unlabeled frames.
• To alleviate exposure bias in SLT and improve semanticlevel error tolerance, we propose a semantic contrastive loss to expose model to its own translation mistakes by re-inputting predicted sequence into semantic module.
• Two new metrics, i.e., Blank Rate and Consecutive Wrong Word Rate are proposed to reflect our improvement on two problems.
• The extensive experiments demonstrate the superiority of proposed losses and our framework achieves state-ofthe-art performance on CSLR and SLT.

2 RelatedWork

In this section, we review the related work on end-to-end CSLR, end-to-end SLT, and contrastive learning.

CSLR and SLT. CSLR aims at recognizing a sequence of signs to corresponding gloss sequence [Koller et al., 2017] while SLT aims to translate sign language to spoken language [Camgoz et al., 2018]. The unique grammatical rules of sign language lead to some differences between frameworks of two tasks . Nevertheless, both two frameworks contain a visual module for encoding sign videos, and a temporal module for exploring context features. In regard to the decoder module, CSLR models often adopt CTC module [Koller et al., 2019] which requires that source and target sequences have the same order. Differently, SLT [Camgoz et al., 2018] is limited to seq2seq architecture and often adopts a RNN-like/transformer decoder. However, it is worth mentioning that CSLR is not limited to CTC decoder and there is also some work choosing seq2seq architecture for CSLR [Huang et al., 2018]. Recently, Camgoz et al. [Camgoz et al., 2020b] combined CTC decoder and transformer decoder in one network, to achieve joint CSLR and SLT.

To alleviate CTC spike phenomenon explicitly or implicitly, existing work focused on designing various visual modules to enhance visual representation. Extra clues including face, hands [Camgoz et al., 2020a], skeletons [Gan et al., 2021] and depth images [Tang et al., 2021] were added in visual encoder. Besides, some work turned its attention to obtaining more training samples by back translation [Zhou et al., 2021a], cross modality augmentation [Pu et al., 2020] and pre-trained models [Chen et al., 2022]. While others were committed to adopting iterative training [Cui et al., 2017] or knowledge distillation [Hao et al., 2021; Min et al., 2021] to enhance representational ability. As for exposure bias problem [Feng et al., 2021], in NLP field, researchers utilized techniques from generative adversarial networks [Subramanian et al., 2017] and reinforcement learning [Shi et al., 2018], then proposed many training algorithms to replace teacher forcing training. However, in SLT, little work had paid attention to exposure bias and only beam search [Graves, 2012] which is the most related algorithm has been used in SLT [Camgoz et al., 2020a] . Besides, unlike NLP tasks that have millions of sentence pairs, SLT datasets only have thousands of sentences, which further exacerbates exposure bias problem. In this paper, to tackle CTC spike phenomenon and exposure bias problem, we provide a simple unified framework for CSLR and SLT, and introduce contrastive learning at visual and semantic levels.

Contrastive learning. As one of the popular unsupervised learning algorithms, contrastive learning [Hadsell et al., 2006] is trained by minimizing the feature distance of different augmented views of the same input ('positive pairs'), and maximizing feature distance of views from different inputs ('negative pairs'). Methods like MOCO [He et al., 2020], SimCLR [Chen et al., 2020] usually adopted Siamese network [Chen and He, 2021], i.e., a weight-sharing network applied on two or more inputs, and they heavily relied on negative sample pairs to prevent collapsing problem (i.e., all outputs of network 'collapse' to a constant).

However, it may be hard to construct negative samples for sign videos. The reason can be included as: (i) In frame level, frames corresponding to transition actions or blur motions have high similarity and may appear in most sign videos, thus not all frames in one video are meaningful and many of them can not be labeled as specific classes. (ii) In video level, label sequences of different sign videos may overlap thus sign videos are not exactly 'negative' to each other (e.g., label 'bab' and label 'cca' have an overlap of token 'a'). Recently, BYOL [Grill et al., 2020] proved that negative sample pairs are not necessary by adopting a momentum encoder. Moreover, SimSiam [Chen and He, 2021] further deleted the momentum encoder while using a stop-gradient operation. That is to say, we can design contrastive losses by stop-gradient operation without constructing negative pairs.

3 Method

In this section, we first give an overall look of our framework. Then, we simply describe CTC spike phenomenon in CSLR and propose a visual contrastive loss which enhances visual representation. After that, we describe the exposure bias problem in SLT and propose a semantic contrastive loss which enhances semantic-level error tolerance. Finally, we design joint loss to train our model for both CSLR and SLT.

3.1 Overall Framework

For a sign video f={fi}ni=1 with n frames, the goal of CSLR is to learn a mapping from f to gloss sequence g={gi}ϑi=1 with ϑ glosses, and the goal of SLT is to translate f to spoken sentence w={wi}ti=1 with t words. As shown in Figure 1, the process of CSLR and SLT can be simplified as follows.

Visual feature extraction. The visual module VE processes frames to get visual features v=VE(f) with m vectors, where v={vi∈Rdv }mi=1 (m≤n) and dv is dimension of vi.

Temporal feature extraction. The temporal module T E extracts the global context information from v to get u=T E(v), where u={ui∈Rdt }mi=1 and dt is dimension of ui.

Recognition and translation. For CSLR, a fully connected layer with weight Wr and bias br, and a softmax layer SM are added to get the final probability matrix of recognized gloss sequence: R=SM(Wr∗u+br). For SLT, a RNN-like or transformer decoder SE is needed to obtain the translation probability matrix of the ith step: Yi=SM(Wy ∗

SE (u, {yj }i-1 j=0 ) + by) where Wy, by are the weight and bias of the fully connected layer in semantic module.

3.2 Visual-Level Contrastive Learning for CTC Spike Phenomenon

CTC spike phenomenon. In CSLR, CTC loss tackles the temporal classification problem for unsegmented sign video, by maximizing probability of all possible paths which align frames with labeled gloss sequence g in probability matrix R. Ideally, it is necessary to label each frame with one specific class. However, considering the meaningless frames and same glosses occurring consecutively, CTC introduces 'blank' token and further defines a many-to-one map B which simply removes all repeated tokens and blanks from the path (e.g., in Figure 2, B(PathB)=B(a-aa-)=aa). Thus CTC loss can be formalized as follows:

where p(π)=∏mi=1 Rπi i , π denotes one of alignment paths (i.e., B(π)=g) and Rπi i is the probability of the ith token πi.

However, CTC loss will lead to spike phenomenon. As shown in Figure 2, the model tends to classify frames as 'blank' tokens when it cannot confidently distinguish gloss boundaries in video, because predicting 'blank' tokens is a much safer choice for minimizing loss LCTC [Min et al., 2021]. For example, B(PathA)=B(aaabb)=ab and B(PathC)=B(-a--b)=ab are both right paths for label sequence 'ab', while CTC 'prefers' P athC than P athA, which causes that only a few frames contribute to final results. Due to that most frames are classified as 'blank' tokens during the training, the visual module is not fully trained and cannot provide effective sign features on the test set.

Visual-level contrastive learning. To tackle spike phenomenon, our intuitive thought is to enhance visual representational ability by training the visual module with more data. However, unlike image classification task (1.28 million labeled images used by ResNet) or NLP task (4.5 million sentence pairs used by Transformer), the data pairs (videogloss pairs or video-sentence pairs) in current sign language tasks are rather limited, and increasing samples by labeling each frame manually to train visual module separately is unrealistic. But unlabeled frames provided by sign videos are

𝒚𝒚𝟓𝟓

𝒚𝒚𝟒𝟒

plentiful, e.g., 963,664 frames provided by 5672 samples in Phoenix14 train set [Koller et al., 2015], and 827,354 frames provided by 7096 samples in Phoenix14T train set [Koller et al., 2019]. Thus, we introduce visual-level contrastive learning [Chen and He, 2021] by comparing visual features of dif-

ferent augmented views of one sign video, so that the visual module can further explore features from numerous unlabeled frames in an unsupervised way.

To construct the visual contrastive loss, we first need to generate two augmented videos from a sign video f={fi}ni=1 with n unlabeled frames. As shown in Figure 3, we introduce two different augmentations: soft augmentation SA and hard augmentation HA to get augmented videos f s and f h ,

Then, visual module VE maps two augmented videos into visual feature vectors in frame level vs={vs i }mi=1 and vh={vh i }mi=1 , as follows.

Next, as shown in Figure 3, a projection MLP head PROvi and a prediction MLP head PREvi are attached to map visual features to the space where the contrastive loss is applied.

After getting ps and zh , our similarity function S calculates negative cosine similarity of each psi and zh i in frame level, as follows. Here, ∥·∥2 is ℓ2-norm.

When using S to calculate similarity of ps and zh , we get can the averaged loss of all frames, as follows.

Finally, following SimSiam [Chen and He, 2021], we further get ph=PREvi(zh ) in a symmetrized way and define a symmetrized visual contrastive loss LV iCo as follows,

where detach(·) function denotes a stop-gradient operation which means that zh and zs are treated as a constant instead of a variable with gradient.

3.3 Semantic-Level Contrastive Learning for Exposure Bias

Exposure bias. In SLT, seq2seq architecture (more precisely auto-regressive architecture) is commonly trained via Maximum Likelihood Estimation (MLE). As shown in Figure 4, taking the RNN-like decoder as an example, the MLEbased training is optimized by maximizing the likelihood of the next word yi based on its previous ground-truth word wi-1 (called teacher forcing) [He et al., 2021], as follows. Here, θ denotes the model parameters, θ∗ denotes updated parameters, and hi-1 is the (i-1)th hidden state of semantic module and u denotes the temporal features.

While during the inference stage, the model needs to predict Yyi i (i.e., probability of word yi) based on prefix word yi-1 generated by model itself.

Due to the exposure to ground-truth data during training, the model may over-rely on right prefix words. While during inference, the wrong predicted word yi-1 could exacerbate error propagation along the generated sequence, and the target probability distributions generated by the model will be incrementally distorted (called exposure bias). Exposure bias will lead to huge performance degradation especially when the sequence distribution of training samples is insufficient to cover the sequence distribution of testing samples.

Semantic-level contrastive learning. To tackle the exposure bias problem, the most straight forward solution is scheduled sampling [Bengio et al., 2015]. Scheduled sampling decides whether to use ground-truth word wi-1 or predicted word yi-1 with a probability of α at the ith step during training. In this way, the model can weaken the dependence on ground-truth words by exposing model to its own mistakes. However, it is non-trivial to apply scheduled sampling directly for transformer model, since that different from RNN-like decoder, transformer generates the ith word based on the whole prefix sequence {yj}i-1 j=0 instead of the last word yi-1. Besides, scheduled sampling training is a sequential process along time thus hinders parallel training. In this paper, following the same idea of exposing the model to its own mistakes, we introduce semantic-level contrastive learning by comparing semantic features of different inputs (the groundtruth sequence input and the predicted sequence input) in semantic module (i.e., decoder module) to alleviate exposure bias. When compared with scheduled sampling, our method can be used for both RNN-like decoder and transformer decoder, and easily trained in parallel.

To construct the semantic contrastive loss, we first need to get translated sentence y based on ground-truth sentence w. Specifically, for temporal features u={ui}mi=1 , the semantic module SE takes u and w as inputs to get semantic features (i.e., hidden states) hw ,hw = SE(u, w) (11)

Then, a fully connected layer and a softmax function SM are adopted to get the possibility matrix Y of translated sentence and further get the translated sentence y through a argmax operation,

After that, as shown in Figure 5, we re-input the predicted sentence y into semantic module to get another semantic features hy , as follows. Here, re-inputting predicted sequence y can be treated as re-inputting augmented ground-truth w, and can also be regarded as a form of scheduled sampling.

Next, similar to visual module in Figure 3, a projection MLP head PROse and a prediction MLP head PREse are added to get ρw and ξw . Besides, the similarity function S is adopted to calculate the distance between ρw and ξ g .

Finally, we also get ξ y =PREse(ρy ), and define a symmetrized semantic contrastive loss LSeCo as follows,

3.4 Joint Loss for CSLR and SLT

To optimize our model with proposed contrastive losses, we need to infer our model MO twice based on different inputs. One contains soft augmented video fs and labeled word sequence w. The other one contains hard augmented video f h and predicted word sequence y, where y=argmax(Y).

After getting (ps , zs , ph , zh ), (ξw , ρw , ρy , ξ y ), we can calculate contrastive losses LV iCo, LSeCo according Equation 8, Equation 16, respectively. Meanwhile, we get two recognition probability matrices R, R′ and two translation probability matrices Y,Y′ . To achieve the goal of CSLR and SLT jointly, we employ CTC loss LCTC and cross entropy loss LCE to train our model, as follows,

Finally, we get the joint loss L with a balance weight α, where α is a hyperparameter, as follows,

4 Experiment

4.1 Datasets

We test our model on public sign language datasets that currently are often used. (1) Phoenix14T [Camgoz et al., 2018] contains 7096, 519 and 642 samples from 9 signers for training, validation, and testing respectively. The Phoenix14T also has two-stage annotations: sign gloss annotations with a vocabulary of 1066 different signs for CSLR and German translation annotations with a vocabulary of 2877 different words for SLT. (2) The CSL-daily dataset [Zhou et al., 2021a] contains 18401, 1077 and 1176 labeled videos from 10 signers for training, validation and testing respectively, and it has both gloss annotations with a vocabulary of 2000 glosses for CSLR and Chinese translation annotations with a vocabulary of 2343 words for SLT. For CSL-daily dataset, there are a few differences between official data and our received data, as shown in Table 1. (3) Phoenix14 [Koller et al., 2015] contains 5672, 540, 629 samples from 9 signers for training, validation, and testing respectively and it has a vocabulary of 1295 glosses for CSLR only.

4.2 Experimental Setting

Here, we describe the baseline settings for our architecture.

Data preprocessing. For data preprocessing, we follow the existing work [Chen and He, 2021] and propose two data augmentations: (1) soft data augmentation: resizing frames to 256×256 pixels, random cropping frames to 224×224 pixels, random horizontal flipping with a probability of 0.5, random temporal scaling (± 20%). (2) hard data augmentation: soft data augmentation, random Gaussian blurring with a probability of 0.5, color jittering with following parameters {brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1}, random grayscaling with a probability of 0.2.

Architecture setting. Our architecture adopts the components provided by PyTorch 1.11. (1) Basic architecture: The visual module consists of a ResNet18 and two 1D convolutional layers with kernel size 5. The temporal module is a three-layered transformer encoder. The semantic module is a three-layered transformer decoder. The dimension of visual features dv is set to 512 and the hidden dimensions of transformer encoder and decoder are both 1024. (2) Projection MLP: it consists of three minimal units which is a combination of fully-connected layer, Batch Normalization layer and ReLU function (FC-BN-ReLU for short), while ReLU function in the last unit is removed. (3) Prediction MLP: it consists of one FC-BN-ReLU unit and a fully-connected layer. The projection MLP and prediction MLP contain the same components at both visual and semantic levels, and the hidden dimensions of MLPs are both 4096.

Label:

ModelA:

ModelB:

Label:

nord

nord nord nord

nord

WERA = WERB = 1/8 = 12.5%

BRA = 1/12 ≈ 8.3% BRB = 3/12 ≈ 25%

morgen

morgen

morgen achtzehn achtzehn achtzehn achtzehn

-

-

-

dann sechs zwanzig zur region

***

sechs

*** sechs

-

sechs zwanzig zwanzig zur zur region region

BLEU1A = 75% BLEU1B = 50%

CWWRA = 0/12 = 0% CWWRB = 5/12 ≈ 41.7%

morgen temperaturen von achtzehn grad an der küste bis sechsundzwanzig im breisgau

ModelA: morgen temperaturen von achtzehn *** an der küste *** sechsundzwanzig *** breisgau Figure 6: An example of BR, CWWR calculation."***" denotes wrong predicted token and "-" denotes blank token.

ModelB:

morgen temperaturen von achtzehn ***

BLEU1A = 75% BLEU1B = 50%

Training setting. To train our model, we use the following settings for CSLR and SLT. We adopt Adam optimizer with a weight decay of 0.0001 to train our model for 70 epochs on 2 GeForce RTX 3090 GPUs. The initial learning rate is 0.0001 with a decay factor of 0.5 and the batch size is set to 6.

Evaluation setting. To evaluate our model, we adopt Word Error Rate (WER) as CSLR performance metric, while adopting ROUGE-L F1-Score [Lin, 2004] and BLEU-1,2,3,4 [Papineni et al., 2002] as SLT performance metrics. Those metrics have been often used to measure the quality of recognition and translation in the existing work.

To verify whether our method can effectively deal with CTC spike phenomenon and exposure bias problem, we propose two new metrics: Blank Rate (BR) and Continuous wrong Word Rate (CWWR). For CSLR (as the top part shown in Figure 6), suppose both model A and model B have the same WER (12.5%), while model A predicts fewer 'blank' tokens than model B. To measure how many 'blank' tokens are in recognized sequence, we introduce the BR metric. BR is an indicator that directly reflects the severity of spike phenomenon and can be formulated as follows,

where #Blank denotes the number of 'blank' tokens in recognized sequence and #TotalR denotes the total number of tokens in recognized sequence.

For SLT (as the bottom part shown in Figure 6), model B is easier to make wrong predictions given wrong prefix words, compared with model A. Therefore we introduce the CWWR metric which measures how many consecutive wrong words (≥2) are in translated sequence. CWWR is an indicator that directly reflects how much the model relies on previously predicted words in translation and can be formulated as follows,

***

***

küste *** sechsundzwanzig **

*

***

where #CW is number of consecutive wrong words and #TotalT is total number of words in translated sequence.

4.3 Ablation Study

In this section, we perform ablation study to verify the effectiveness of the proposed model. We mainly conduct experiments on Phoenix14T by following previous works [Camgoz et al., 2018] for a fair comparison.

Effects of the beam width. Beam search is the most common algorithm used to alleviate exposure bias problem [Zhou et al., 2021a]. Here, to set a proper beam width for beam search, we test recognition performance and translation performance under different beam widths ranging from 1 to 9 and 10 to 50. According to Figure 7, we set beam width of CTC decoder to 6 for lower WER in CSLR, and set beam width to 2 for higher BLEU-4 in SLT.

Effects of weights α. To balance the different losses in Equation 21, we test model performance under different α ranging from 0.1 to 2. According to Table 2, the model achieves the best performance when α is set to 1. Thus we set α to 1 in this paper.

Effects of training strategies. In the early training stage where word predictions may be noisy, the model directly trained with contrastive losses can be difficult to converge. To verify whether training strategies affect model perfor-

mance, we compare different training strategies and show the ROUGE scores on Phoenix14T dev set. As shown in Figure 8, as for the fine-tuning strategy, the model converges much more stable in the early stage, while the model needs more epochs for fine-tuning. As for our training strategy, during 1 to 5 epochs of the training stage, the training process is unstable and the ROUGE oscillates around 0. But the model can quickly converges after 10 epochs. Thus the model is trained with our training strategy in this paper.

Effects of proposed contrastive losses. As shown in Table 3, results demonstrate that the proposed losses contribute to higher performance for CSLR and SLT. Specially, visual contrastive loss LV iCo improves the recognition performance (WER drops by 2.8%), which verifies that visual representation ability can be enhanced with proposed LV iCo. The semantic contrastive loss LSeCo improves the translation performance (ROUGE increases by 5.65%) which indicates that the model can benefit by exposing model to its own mistakes.

To further verify whether the proposed losses can tackle CTC spike phenomenon and exposure bias problem, we show results on proposed BR and CWWR in Table 3. From this table, we get the following conclusions: (1) CTC spike phenomenon does widely exist in CSLR task, and up to 67.72% of tokens are predicted as 'blank'. (2) 'Blank' token still exists with proposed visual contrastive loss, but LV iCo can effectively alleviate CTC spike phenomenon, i.e., BR drops by 10.25% when adopting LV iCo. (3) As for translation, exposure bias problem also exists with proposed LSeCo. However, CWWR drops by 3.88% when LSeCo is adopted, which verifies the effectiveness of semantic contrastive loss.

Figure 10: Effects of different padding rates.

in der

Figure 11: Qualitative analysis on Phoenix14T test set. sinken2 means two repeated 'sinken' tokens. 'Baseline': only using Lsoft.

in der nacht sinken die temperaturen auf vierzehn bis sieben grad in der nacht sinken die temperaturen zurück auf sieben grad Heute der nacht sieben die temperaturen auf sieben bis sieben grad Trans [36, 98, 37, 107, 5, 94, 190, 107, 88, 107, 90, 2 label [1, 65, 98, 37, 348, 5, 94, 190, 342, 88, 107, 90, 2 Explore random masking in visual level. Inspired by MAE [He et al., 2022], we add random masking in hard augmentation HA to further explore the possible improvement of visual representation ability. As shown in Figure 9, as the mask rate increases, BR decreases and CSLR performance further improves (i.e., WER decreases). However, when mask rate further increases, it is difficult for the visual module to capture valid features through visual contrastive loss since that sign-related features could be masked.

Explore random padding in semantic level. Semantic module benefits from exposing possible errors during training. However, as the model converges, the predicted sentence y tends to be fixed (i.e., almost unchanged), which leads to that the model could not generalize well. Thus we further replace y with ypad to manually add errors in re-input sequence y, where we randomly replace words in y with 'PAD' tokens to get ypad. As shown in Figure 10, as the random padding rate increases, translation performance increases first and then decreases, which demonstrates that manually adding a small proportion of random errors can enhance the generalization ability of semantic module. But when padding rate further increases, the distribution of ypad will deviate far from that of testing sequence, leading to performance degradation.

Qualitative Analysis. To verify whether our model can alleviate CTC spike phenomenon and exposure bias problem, we show a recognized example and translated example in Figure 11. Recognized results and translated results indicate that the proposed contrastive losses can effectively alleviate CTC spike phenomenon and exposure bias problem, further improve model performance with lower BR and CWWR.

4.4 Comparisons

Evaluation on Phoenix14T dataset. As shown in Table 4 and 6, we compare our model with existing models on both CSLR performance and SLT performance. We provide both the performance on validation set (i.e., 'DEV') and test set (i.e., 'TEST'). On CSLR performance, our model achieves a comparable performance (21.8% WER on test set) with the state-of-the-art (SOTA) approaches which inject extra clues like local areas, skeletons or pre-trained models. Furthermore, our model improves the CSLR performance and outperforms the SOTAmodel on test set, i.e., 20.1%WER, when adopting random masking and random padding. On SLT performance, our model still outperforms current models.

Evaluation on CSL-daily dataset. We also provide CSLR and SLT performance comparisons with other methods on CSL-daily dataset. As shown in Table 5 and Table 7. The WER of our model on dev and test set is 26.0% and 25.3%, respectively, and our model outperforms existing models by a large margin on CSLR performance. Furthermore, the BLEU4 of our model on the dev, test set is 21.79%, 21.81%, which outperforms current models.

Evaluation on Phoenix14 dataset. We compare our model with existing models of CSLR performance on Phoenix14 dataset. As shown in Table 8, our model which does not use any extra clues or pre-trained models, achieves SOTA results compared with models trained with extra clues or pre-trained models.

5 Conclusion

In this paper, we introduce contrastive learning to tackle two problems that widely exist in sign language tasks: CTC spike phenomenon in CSLR task and exposure bias in SLT task. Specifically, to tackle CTC spike phenomenon, we design a visual contrastive loss to enhance visual module by learning features from numerous unlabeled frames. To tackle exposure bias problem, we design a semantic contrastive loss to enhance semantic module by exposing the model to its own mistakes. Besides, we propose two new metrics i.e., Blank Rate and Consecutive Wrong Word Rate to directly reflect our improvement on two problems. The experimental results on public sign language datasets demonstrate the effectiveness of our method.

Acknowledgments

This work is supported by Jiangsu Provincial Key Research and Development Program under Grant BE2020001-4; National Natural Science Foundation of China under Grant Nos. 62172208, 62272216, 61832008. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.