Connectionist Temporal Modeling of Video and Language: a Joint Model for Translation and Sign Labeling

1 Introduction

This paper addresses problems associated with sign video interpretation, which is related to topics in the computer vision and machine learning fields, i.e., gesture recognition [Joshi et al., 2017], action detection and location [Nguyen et al., 2018], human behavior analysis [Kacem et al., 2018], and video understanding [Jelodar et al., 2018]. Vision-based sign interpretation originates from isolated sign recognition [Wu et al., 2016]; however, researchers are paying more and more attention to continuous Sign Language Translation (SLT) [Koller et al., 2016a; Cui et al., 2017]. Essentially, SLT aims to bridge the semantic gap between vision and language under complicated sign linguistics. Thus, we consider the SLT task, with a particular focus on online SLT.

SLT is challenging due to the following. (1) Visual hints under sign linguistics are latent and obscure, i.e., facial expressions, lip languages, flexible signs of locality, and even some specific adjectives and adverbs. For example, the adverb "fast" is represented by increasing the speed of signing[Neverova et al., 2016]. (2) SLT involves additional challenges related to hybrid semantics learning under vision understanding, sign recognition, and natural language translation. How to jointly learn an excellent visual representation, encode complicated sign linguistics, and decode grammatical sentences in a unified model framework remains a difficult problem to solve [Graves et al., 2006]. (3) Sign videos have sentence-level annotations, rather than the exact temporal location of each sign action. It is a weakly-supervised sequence-to-sequence problem [Pu et al., 2018]. (4) Accurate online video translation is also difficult. Some previous studies first encode the entire video and decode textual semantics, e.g., for video captioning [Venugopalan et al., 2015]. To improve the translation performance, offline algorithms have been used to fine-tune the model parameters [Koller et al., 2016a; Cui et al., 2017]. Our objective is end-to-end training for online translation with no additional supervision.

In this paper, we propose a joint model for online translation and sign labeling, which we refer to as a Connectionist Temporal Modeling (CTM) network. As shown in Figure 1, the pre-trained deep models ResNet-18 [He et al., 2016] and ResNet-3D [Hara et al., 2017] are used to extract 2D and 3D CNN features, respectively. Here a 3D CNN learns the spatiotemporal hints in the short-term clips, while 2D CNN features remain visual details at the frame-level. To align and fuse these features under different granularities, a Temporal Convolution Pyramid (TCP) module is designed to compress the 2D features by convoluting several adjacent features. As shown in Figure 2, there are three-stage convolutional operations, and each temporal convolution span involves two nonoverlapping time steps in the TCP. Thus, the long frame-level 2D features (eight time steps) are transformed into a compact clip-level 3D feature (one time step). In fact, the TCP is a "pseudo" 3D feature extraction operation, i.e., 2D spatial +1D temporal convolutional operations. Finally, we adopt MultiLayer Perceptron (MLP) to fuse pseudo 3D and original 3D features.

Next, three modules in the CTM framework, i.e., Connectionist Temporal TRanslation (CTTR), Feature CLaSsification (FCLS), and Feature CORrelation (FCOR), receive aforementioned fused features for long-term sequential learning. As shown in Figure 3, the first CTTR module performs sentence translation, while the FCLS and FCOR modules measure sign labelling at the word-level. In particular, the CTTR module provides a translated sentence with the maximum probability under multiple decomposed paths of sign word labeling. It addresses the task in a weakly-supervised learning manner. Since videos are annotated at the sentencelevel, CTTR is the primary contributive module in the longterm sequential learning.

To further address the challenges associated with weakly supervision learning, we integrate the pseudo-supervised learning into the deep CTM model. In our solution, Lcttr, Lfcls, and Lfcor involve the objective functions of above three deep modules, respectively. Using Lcttr, we calculate a sign labeling path with maximum probability corresponding to the sentence label and consider it as sequential sign labels of features, i.e., pseudo labels. Based on the pseudo labels, Lfcls measures the entropy regularization on feature classification, and Lfcor is a triple loss relative to feature correlation. Lfcor models the feature similarity under the same sign class or different sign classes, i.e., similarity difference among positive or negative feature samples. Finally, Lcttr, Lfcls, and Lfcor are combined as the objective function of the overall framework. By minimizing the joint loss, the model is pushed to learn more temporal cues. The primary contributions of this study can be summarized as follows:

• Learning a good clip representation facilitates the acquisition of the short-term temporal correlation. The TCP is conducted on 2D CNN features to realize (2D+1D)=pseudo 3D′ CNN. We align the 3D′ features with the original 3D features and fuse them.
• We propose a connectionist decoding scheme for longterm sequential learning, where the decoder embeds the dynamic programming optimization into end-to-end deep learning. It learns the connectionist mapping among features, sign words, and the generated sentence.
• Pseudo supervision cues are utilized for online learning. We design a joint objective function to measure sentence

translation, feature correlation, and entropy regularization based on pseudo labels, where pseudo labels denote the sign word labels obtained from previous connectionist decoding.

2 RelatedWork

2.1 Sequence-to-sequence Learning

Learning temporal cues in videos is very important. In this paper, we do not discuss traditional temporal models, such as Dynamic Time Warping (DTW) [Lin et al., 2014] and Hidden Markov Models (HMM) [Guo et al., 2017], in detail. For SLT, both DTW and HMM require significant training time. DTW implements template matching on the entire dataset and HMM represents the transformation among a large number of latent states. With the rapid development of deep learning, various RNN deformations have become more prevalent and effective in sequential learning. Liu et al. proposed an LSTM-based end-to-end neural network that was effective for isolated sign recognition [Liu et al., 2016]. Given the ability of CNNs to extract features and RNNs to perform sequential learning, many hybrid models were emerged, i.e., temporal convolution and bidirectional RNNs [Pigou et al., 2018], Recurrent 3D CNNs [Lefebvre et al., 2015], and depth DNN embedded with HMM [Wu et al., 2016]. These hybrid methods can model the spatiotemporal variations simultaneously. Note that we also exploit the merits of CNNs and RNNs in our proposed model.

2.2 Weakly-supervised Learning in Videos

As SLT is a typical weakly-supervised task, some researchers have focused on action location and word alignment. To address word alignment, Koller et al. embedded a deep CNN into the hybrid HMM framework [Koller et al., 2017]. Cui et al. proposed an LSTM and Connectionist Temporal Classification (CTC) framework to address gloss-level classification [Cui et al., 2017]. In contrast, the encoder-decoder framework, which is widely used in neural machine translation and video captioning, is prevalent for weakly-supervised sequential learning. Guo et al. proposed an encoder-decoder framework with variable-length clip mining for a Chinese sign language translation [Guo et al., 2018]. A similar study [Cihan Camgoz et al., 2018] proposed an attentionbased encoder-decoder network comprising two specialized RNN modules. However, these proposed methods decoded word by word after encoding all visual content. They do not apply to online SLT. Online sequential action recognition [Liu et al., 2018] is closely related to our study. However, the task assumed strict supervision cues, i.e., exact start and end time labels for each action. Our task differs in that it involves sentence-level labels without exact temporal cues at the word-level. Compared to the problem that Liu et al. [Liu et al., 2018] addressed, online SLT has fewer temporal cues.

In order to enhance temporal cues, some studies have adopted multiple iterations of the EM algorithm to perform offline fine-tuning. Koller et al. trained a frame-level classifier on extra sign language dataset by embedding a CNN in an iterative EM algorithm [Koller et al., 2016a]. In [Koller et al., 2016b], the proposed model embedded a CNN into a HMM, which feeds the output of the CNN to a Bayesian model in the HMM for continuous SLT. Cui et al. designed a three-stage optimization process, i.e., feature extraction, word alignment, and sequence learning [Cui et al., 2017]. In [Pu et al., 2018], authors alternately optimized the feature extractor and the sequence learning model using dilated convolution operations. Differing from these offline iterations, the proposed model introduces pseudo supervision cues into an end-to-end framework for online sequential learning.

3 ProposedMethod

The general framework of our model is shown in Figure 1. Given a video V = {vn}Nn=1 , two feature streams, i.e., framelevel features f2d = (f1, · · · , fN) and clip-level features f3d = (clip1, · · · , clipN), are extracted, and then a sequence of gloss labels Y = (y1, · · · , ym) is output. We discuss each module in detail in the following.

3.1 Clip Feature Learning in Videos

Sequential learning always suffers gradient attenuation along temporal transitions. Learning a good clip representation is to acquire the compact short-term temporal correlation. In this section, we discuss the extraction of 2D and 3D CNN features, the alignment of 2D and 3D, and feature fusion. The advantage of this clip representation is that both 3D and temporal 2D CNNs are leveraged to learn discriminant features.

TCP on 2D Features

Motivated by the △t-gram language model used in natural language processing tasks, in the Temporal Convolution Pyramid (TCP) module, each layer (temporal convolution layer; TCOV) implements the embedding of n-item adjacent features. It calculates local convolution in the short-term temporal view. In other words, the TCP is a pseudo 3D feature learning operation, i.e., the combing 2D spatial and 1D temporal convolutions.

Given the original 2D frame features {f2d} ∈ Rd0×N , an l-layer TCP transforms them to pseudo clip features {f ′3d } ∈

Rdl× N ∏li=1 (si) . The TCP calculation is expressed as follows:

where Φi indicates the parameter of the i-layer TCOVi in the TCP, and Φi =(chi, di, △ti, si, padi) denotes the format of a convolutional parameter (number of channels, height, width, stride, and padding). di ×△ti denotes the convolutional kernel size, and si is the sliding window along the temporal dimension. Here, di+1 = chi, i.e., the kernel size for the (i+1)- th layer is set to the output dimension of the i-th layer.

As shown in Figure 2, there is a three-layer TCP with (△ti = 2)-gram and non-overlapping (si = 2). Thus, the frame-level 2D features (eight time steps) are transformed into a short clip-level 3D feature (one time step). The TCP gradually condenses temporal cues via contiguous △t-items.

2D & 3D Feature Alignment and Fusion

Give the temporal dimension of the original 3D features, a compact pseudo 3D feature is obtained using the TCP; thus, the model can align pseudo 3D feature f ′ 3d to the original 3D features f3d, and fuse them. Here, the entire clip representation enforces feature fusion [(2D+1D ≈ 3D′ )+3D]. The fusion scheme using the MLP is formulated as follows:

where ⊕ represents the concatenation operation on vectors.

3.2 Connectionist Temporal Translation

After feature fusion, we tackle long-term sequential learning. This subsection elaborates on the Connectionist Temporal TRanslation (CTTR) module in Figure 3. The other two modules are introduced in Section 3.3.

Temporal Encoding

The BGRU excels at modeling forward and backward contexts; thus, it is robust and effective for sequential action recognition. We employ the BGRU, expressed as follows, as the basic encoding RNN unit.

Then, based on the output of BGRU, we employ a fullyconnected layer FC to embed the output into non-normalized categorical probabilities with K sign classes as follows:

where ϕsoftmax is the softmax function, P is the probability score matric, pn ∈ RK is the categorical probability vector of the n-th clip, and K equals the vocabulary size plus 1 (a new introduced blank symbol ' ').

Decoding Optimization

Let word alignment π be a sequence of words including ordinary words and the blank symbol ' ', i.e., π = {πn}Nn=1 , where πn denotes the n-th word of π. The probability of π is given by the product of probabilities as follows:

where pπn n denotes the probability of word πn at time step n. Motivated by dynamic programming optimization [Graves et al., 2006], we define a two-stage greedy strategy on a long word label sequence. It outputs a short sentence by removing the blank label in the first stage and deletes continuous repetitions in the second stage. Note that a target sequence Y may correspond to multiple different alignments {π}; thus, the CTTR module defines a many-to-one mapping as B, which removes all blanks and repeated labels from the alignments. As shwon in Figure 3, B implements the transformation between sign labeling and sentence translation. Therefore, in the training process, the CTTR decoder objective function is defined as follows:

where B-1 (Y)={π | B (π)=Y} is the set of all alignments. In the testing stage, we output the word alignment using the maximum function on score matrix P = {pn}Nn=1 and transform it to a sentence Y={yn}mn=1 .

3.3 Joint Loss Optimization

To further enhance the temporal cues in videos, we introduce the concept of pseudo-supervision. Note that pseudo labels denote the sign labeling path π = {πn}Nn=1 with the maximum product of probabilities P πmax (Section 3.2). Here, π={π1, · · · , πn} is taken as the available pseudoword labels of the sequential features. Based on these labels, the FCLS

module evaluates the feature classification entropy. In addition, the FCOR module measures the inter- and intra- similarities of different feature groups under K sign classes. Both modules contribute to inhibit overfitting of the CTTR module.

Here, based on training set S (all video samples and annotations), we create set Mof clip-level fused features and set T of all pseudo triplets in all training epochs. The objective function of the entire training framework is given as follows:

where Lcttr is already given in formula (6), while Lfcls and Lfcor are expressed in following formulae 8 and 11.

Cross-entropy Loss

Based on fused features {fn} obtained in subsection 3.1, we realize the FCLS module via Fully Connected layer (FC), Batch Normalization (BN), and softmax operations to obtain predicted probabilities at each clip-level time step. Here, we define m as a feature sample in M and πm is its pseudo label. If the sequential features are reliable, the new predicted probability of m is close to its pseudo label πm. Note that we adopt cross entropy loss to measure the predicted probabilities and pseudo labels as follows:

where ykm obtained by the CTTR module indicates whether sample m belongs to class πm (value of 1 or 0), and pkm is the new predicted probability calculated by the FCLS module.

Triplet Loss

The triplet loss is designed to ensure that features with the same label have close embeddings together, while features with different labels are distant in the embedding space. We adopt the FC and BN operations in the FCOR module to model the new feature embeddings of {fn}.

Based on the sign labeling path π with the maximum product of probabilities P πmax , we split these embedding features into different groups. In summary, features are divided into positive pairs with the same word label; otherwise, features are divided into the negative pairs. We denote it as set T = {(e+, e+), (e+, e-), (e-, e+)}. For the video example shown in Figure 4, e1 and e6 are (e+, e+), while e1 and e2 are (e+, e-)/(e-, e+). In the correlation matrix, the colored squares indicate positive pairs, and the gray squares indicate negative pairs. Here, squares with snowflake points represent pairs containing a blank label ' ' with no word meaning. Note that we do not consider self-pairs along the diagonal line and squares with snowflake points (blank label ' ') in the matrix.

The distance measurement for positive and negative feature pairs must satisfy the following constraint:

where s(a, b) is similarity score between features a and b, L2 represents L2-normalization, and parameter α controls similarity intensity during training. Here, we seek similarity scores for positive pairs that are greater than both negative pairs and the margin α.

Thus, we set triplet loss Lfcor to calculate all triplets {(e+, e+), (e+, e-), (e-, e+)} in dataset T . The similarity measurement s(a, b) by L2-normalization is symmetrical; thus, normal triplet loss is formalized as follows:

where (e+, e-) = (e-, e+) is denoted tneg, and (e+, e+) is denoted tpos in T .

As shown in Figure 4, positive pairs in a video are much less than negative pairs, no more than in the whole dataset. Thus, we adjust the calculation of triplet loss. For each training batch, we randomly sample the same number of negative pairs as positive pairs in formula (10). Then, we evaluate positive pairs individually; the same evaluation is performed for negative pairs. Finally, we adopt the triplet loss Lfcor as follows:

where β controls similarity intensity during training.

4 Experiment

4.1 Dataset and Evaluation

We experiment on two benchmarks, i.e., RWTH-PHOENIXWeather [Koller et al., 2015] and USTC-CSL [Huang et al., 2018]. The PHOENIX dataset includes 6841 Germany sign videos played by nine signers. The dataset is split into three independent parts, i.e., "TRAIN", "VAL" and "TEST". The CSL dataset includes 5000 Chinese sign videos annotated by 50 signers. Under the "Split II" constraint [Guo et al., 2018], it selects video samples of 94 sentences for training. The remaining unseen six sentences are used for testing.

To evaluate performance, Word Error Rate (WER) [Koller et al., 2016a] measures the minimum number of insertion, replacement and deletion operations converting generated sequence L′ to ground truth L. Here, ins and del denotes the proportions of insertion and deletion operations divided by length |L|, respectively.

4.2 Implementation

Videos in the RWTH-PHOENIX-Weather dataset are under pixels 210×260, which covers the human body. In the USTCCSL dataset, each frame has 1280 × 720 pixels, which contains significant amounts of redundant blank background. We segment a local region with 210×260 pixels to cover the entire human body. Then, all images in both two datasets are resized to 224 × 224 pixels, and we combine the adjacent eight frames with a clip of four frames of overlap for ResNet-3D. We feed the same data as the ResNet-18 input.

We adopt the ADAM optimizer and set the batch size to 40 and initial learning rate of the overall network to 1 × 10-4 . Here, the learning rate is gradually reduced by 1/10 every 20 epochs. As shown in Table 1. We set parameters d0=d1=d2=d3=512-dim in the TCP. Note that each di can differ. We also use ReLU and Dropout operations after each TCOV layer in the TCP to avoid over-fitting. We set the Dropout parameter to 0.2. Moreover, the initial learning rate of the TCP is 5 × 10-4 . β = 0.5 was found to provide the best experimental performance.

4.3 Model Validation

We test different features to reflect the impacts of the TCP and feature fusion modules. As shown in Table 2, the del rate of f2d reaches 55.1%, which is far worse than other features. It indicates that as f2d becomes overly long, it generates many redundant incoherent word labels along the temporal dimension. By introducing the TCP on f2d, the del rate significantly improves the effectiveness by dropping approximately 50% of the words. Directly adding f3d ′ and f3d improves performance; however, the improvement is insufficient. By implementing MLP fusion on f3d ′ and f3d, the WER of the fused features is reduced from 42.2% to 41.0% and 42.2% to 41.3% on the VAL and TEST sets, respectively.

We verify the joint loss optimization, i.e., the effectiveness of pseudo-supervision optimization. The CTM framework primarily conducts end-to-end weakly supervised learning with Lcttr. By introducing Lfcls, the performance on both the VAL and TEST sets are improved by 1.1% (Table 4). With the auxiliary of Lfcor, the model learns the similarities and differences among clip-level features, which further reduces the WER value on the VAL and TEST sets by 1.0% and 1.5%, respectively. When Lcttr, Lfcls, and Lfcor are used simultaneously, the proposed approach achieves the best results. A comparison of the obtained results relative to different module settings is shown in Figure 5.

4.4 Main Comparison

Here, we compare the proposed approach to the state of the art methods. As shown in Table 3, there are two obvious conclusions. (1) Extra visual hints are widely used to improve performance in previous studies, such as introducing visual representations of human body parts (hand and face) and pose trajectory. Moreover, 1M-Hands [Koller et al., 2016a] imported a pre-trained sign vocabulary, and CNN-Hybrid [Koller et al., 2016b] utilized an initialized word alignment. In contrast, the proposed model has no additional initialization and extra input data. (2) Most approaches adopt additional offline optimizations, such as 1Mio-H, CNN-Hybrid, Staged-Opt [Cui et al., 2017], and Dilated-CNN [Pu et al., 2018]. Without offline iterations, the proposed CTM demonstrates much better performance than HOG-3D [Koller et al., 2015], CMLLR [Koller et al., 2015], SubUNets [Camgoz et al., 2017], Staged-Opt-init and Dilated-CNN-init. Note that Staged-Opt and Dilated-CNN show excellent performance with offline iterations. Their initial WER value reduce rapidly to 46.9% and 59.7% on the TEST, and these values are much lower than those obtained by the proposed CTMmodel. Note that besides time consuming, offline iteration has another weakness, i.e., it is always trained repeatedly using fixed datasets, which is not applicable to dataset extension. The proposed CTMmodel does not suffer this limitation, and still achieves comparable performance without extra supervision and offline iterations.

In addition, as shown in Table 5, this proposed model also achieves the best performance compared to other methods on the USTC-CSL dataset. The proposed model demonstrates the performance gains of 2.2∼5.1%. More importantly, both S2VT and HLSTM involves a encoding-decoding architecture for sentence generation, and this architecture decodes sentences after encoding the entire video. In contrast, the proposed CTM model exploits online connectionist temporal encoding; thus, the proposed approach is more flexibility relative to online SLT.

5 Conclusion

The paper proposes a temporal convolution pyramid module to learn the short-term temporal correlation, and a connectionist decoding scheme for long-term sequential learning. In addition, we design a joint objective function to optimize online learning under pseudo supervision. The proposed CTM model represents an end-to-end deep network for online translation and sign labeling. The experimental results demonstrate that the proposed approach achieves results that comparable to state of the art methods.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under grants 61725203, 61732008, and 61876058.