HOI-aware Adaptive Network for Weakly-supervised Action Segmentation

1 Introduction

Action segmentation aims to predict the action for every frame in the video. While previous methods have achieved remarkable performance in the fully-supervised setting [Kuehne et al., 2016; Lea et al., 2017; Rohrbach et al., 2012; Singh et al., 2016; Yi et al., 2021; Park et al., 2022], framewise annotation still requires huge labor costs and is hard to obtain. Therefore, action segmentation with weaker forms of supervision gradually gains its popularity in recent years. In particular, transcript supervision [Bojanowski et al., 2014; Kuehne et al., 2017; Huang et al., 2016; Ding and Xu, 2018; Li et al., 2019a; Lu and Elhamifar, 2021] provides an ordered list of actions occurring in the video without the starting and ending time, which significantly reduces the annotation costs and improves the applicability to a rapidly-growing number of videos on the Internet.

∗Corresponding author

To learn from the transcript, previous approaches mainly follow the "generating-matching" pipeline [Richard et al., 2018; Li et al., 2019a; Lu and Elhamifar, 2021]. With the given training videos, they first apply a temporal encoder to generate framewise action probabilities, and then match the predicted probabilities sequence with the transcript based on Viterbi decoding or dynamic time warping. However, to estimate the action probability of frame t during the generating step, most existing approaches only take a fixed number of neighbor frames around it [Richard et al., 2018; Li et al., 2019a], and feed such video clip features into an RNN-based [Chung et al., 2014] architecture. In this case, the temporal encoder would fail to distinguish the attribute of similar actions such as pouring coffee and pouring juice, which may lead to counter-intuitive results of pouring coffee in a juice-making video. Although tremendous efforts have been made to remedy such ambiguity in the matching step, the results are still unsatisfying due to the inherent defect in the previous generating process.

In this paper, we address the ambiguity problem by designing an adaptive weakly-supervised action segmentation framework called AdaAct. Different from previous methods which take a series of fixed-length video clips as input successively (as shown in Figure 1 (a)), we exploit rich contextual information from temporally global but spatially local human-object interactions (HOI) throughout the whole video. Such HOI sequence further instructs the network as prior knowledge, where our temporal encoder can be dynamically adapted to it at the test time. As illustrated in Figure 1 (b), our method first extracts key interactions with objects at different video timestamps, such as the knife, orange, and squeezer. The obtained HOI sequence is further incorporated into the temporal encoder, thereby the network parameters of the encoder dynamically change with the HOI information on the fly. More specifically, we design a three-step video HOI encoder with the "extracting-selecting-integrating" process. We first apply a pre-trained HOI detector to extract positive interaction bounding boxes for the whole video and design a simple selecting algorithm to pick the most representative ones from them. Then, we explore the relations between these key HOI boxes and integrate them into a single feature vector via a transformer-based network. To dynamically adapt the network parameters, we propose a two-branch HyperNetwork that simultaneously learns HOI-dependent and HOIindependent knowledge. Our HOI-independent branch aims to unearth the general characteristics of instructional videos by iteratively updating a learnable embedding list throughout the training process. Such transferable information will later be encoded as a part of instruction used in the temporal encoder during the test time. On the other hand, our HOI-dependent branch takes the encoded feature vector from the video HOI detector as input and adapts the temporal encoder to the given HOI knowledge occurring in the video. Finally, late fusion is utilized to merge the knowledge from two branches, resulting in more precise action segmentation.

We summarize our key contributions as follows:

1. To our best knowledge, this is the first work to learn an adaptive temporal encoder for weakly-supervised action segmentation, where the parameters of the network are dynamically adapted according to the input video on the fly.
2. We propose to exploit temporally global but spatially local HOI information in weakly-supervised action segmentation, which provides essential contextual information to address the ambiguity problem of similar actions.
3. We validate our method on two challenging datasets, Breakfast and 50Salads, and achieve state-of-the-art results for both weakly-supervised action segmentation and alignment tasks.

2 RelatedWork

Fully-supervised action segmentation. Fully-supervised action segmentation methods learn action segments under the guidance of framewise annotations. Earlier attempts [Karaman et al., 2014; Rohrbach et al., 2012] applied action classification on the sliding window, followed by non-maximum suppression to filter out redundant predictions. However, these approaches failed to model the temporal dependency between action sequences. Kuehne et al. [2016] tackled this problem via the hidden Markov model, while Pirsiavash et al. [2014] applied context-free grammar to capture temporal structure. In recent years, various network architectures were proposed for learning long-range dependency. Lea et al. [2017] introduced an encoder-decoder architecture for action segmentation and detection. Lei et al. [2018] further applied deformable convolutions and residual stream. Farruha et al. [2019] and Li et al. [2020] introduced dilated temporal convolution and proposed a multi-stage temporal convolutional network, while various methods improved the multistage network using graph-based temporal reasoning [Huang et al., 2020] or boundary-aware cascade network [Wang et al., 2020b]. With the success of transformer-based models in computer vision, Yi et al. [2021] first introduced the transformer into the action segmentation task. Different from previous methods, Li et al. [2022] reformulated the action labels as text prompts and paired them with corresponding video clips, and co-trained the text encoder and the video encoder through a contrastive approach.

Weakly-supervised action segmentation. Many of the weakly-supervised methods utilize transcripts as supervision during training. Huang et al. [2016] first introduced the connectionist temporal classification framework to evaluate all possible matching between the videos and transcripts. Ding et al. [2018] started from the initial uniformmapping of the action transcript, and iteratively refined the transcript during the training procedure. However, these methods fail to achieve end-to-end training. Richard et al. [2018] instead generated pseudo frame labels using the Viterbi algorithm and trained a classifier based on framewise cross-entropy loss. Li et al. [2019a] further extended the NN-Viterbi [Richard et al., 2018] by introducing a new constrained discriminative forward loss, which maximized the energy difference between valid and invalid segmentation of training videos. In D3TW, Chang et al. [2019] first applied a discriminative model for solving the degenerate sequence problem. As these methods have to search all the transcripts during testing and thus suffer from long inference time, Souri et al. [2021] proposed MuCon, a two-branch network that predicted both transcript and framewise label of action segmentation, and designed the mutual loss to ensure the consistency of representations. In recent years, different weakly-supervised settings besides the transcripts have been studied. Fayyaz et al. [2020] and Li et al. [2020] reduced the supervision level, assuming only the unordered list of actions is available for each training video. Inspired by the point supervision in semantic segmentation [Bearman et al., 2016], Li et al. [2021] trained a segmentation model using timestamps annotations, in which case only one arbitrary frame is annotated for each action. As these methods use different kinds of supervision for training, we do not directly compare them with our approach.

Human object interaction. The existing HOI detection can be mainly categorized into single-stage approaches [Liao et al., 2020; Wang et al., 2020a; Kim et al., 2020; Chen et al., 2021] and two-stage approaches [Li et al., 2019b; Zhang et al., 2021; Zhou and Chi, 2019; Zhou et al., 2020; Ulutan et al., 2020]. Single-stage approaches integrate bounding boxes detection and interaction recognition into a single model. Liao et al. [2020] and Wang et al. [2020a] first simultaneously generated bounding box candidates and interactions, and then outputted final predictions after the matching step. Chen et al. [2021] instead reformed the HOI detection as an adaptive set prediction problem. Compared with one-stage methods, two-stage approaches first detect humans and objects following the object detection pipeline and then apply an interaction model to analyze the relations of the bounding boxes. Qi et al. [2018] and Zhang et al. [2021] modeled the relations using graph neural network. Fang et al. [2018] emphasized the importance of human-part knowledge in HOI detection. Although different methods have been proposed in the image domain, research on video-level HOI detection is still under-exploited.

3 Methodology

Our goal is to address the weakly-supervised action segmentation problem under transcript supervision. Formally, we define each video with its supervision as a tuple {v, xT1 , aO1 , lO 1 }, where v represents the video as a stack of raw frames, xT1 = [x1, ..., xT ] denotes the unsupervised framewise features with length T , aO1 = [a1, ..., aO] indicates the transcript, an ordered list of O actions occurred in the video, and lO 1 = [l1, ..., lO] records the number of frames for each of the corresponding actions. Every action ao belongs to the set of A action classes, namely ao ∈ A = {1, ..., A}. During the inference, the objective is to predict the optimal action list ˆa and corresponding length ˆl based on the framewise features x of the video v.

In this paper, we propose an adaptive network named AdaAct that utilizes video-level HOI to distinguish similar actions. As shown in Figure 2, our method mainly consists of a video HOI encoder and an adaptive temporal encoder. For the video HOI encoder, it first takes the input video and extracts all the valid interactions, then selects top-K interactions by removing redundant and low-score detection. These interactions are finally integrated as HOI-dependent knowledge s. For the adaptive temporal encoder, it incorporates HOI-dependent knowledge s with HOI-independent knowledge z via a two-branch HOI-aware HyperNetwork [Ha et al., 2016], which predicts the network parameters of the temporal encoder. In the following, we describe the video HOI encoder and the adaptive temporal encoder in detail, as well as the training strategy to learn these two models.

3.1 Video HOI Encoder

The goal of our video HOI encoder is to model the dependencies between key HOI through the whole video and encode them as HOI-dependent knowledge s. It mainly contains three levels from the bottom to the top: extracting, selecting, and integrating.

Extracting

We take the video as input at the first level. Since the majority of HOI detection methods are developed only for image scenarios, we pre-process the video by down-sampling and extracting the raw frames under 15 FPS. After that, we employ the detector on every frame iteratively following the temporal order. To avoid introducing additional computation cost, here we follow the 100 Days of Hands [Shan et al., 2020] with its weight frozen during training and testing. The model outputs the predictions as tuple {bh, bo, c, t}, where bh and bo represent the bounding boxes of hands and object, c ∈ [0, 1] denotes the interaction confidence score, and t indicates the timestamp of the frame.

Selecting

Inspired by the non-maximum suppression (NMS) [Neubeck and Van Gool, 2006] used for filtering proposals in object detection, we propose a video-NMS algorithm to select top-K object bounding boxes from the predictions pool. Different from the traditional NMS algorithm that filters the proposals only by the intersection over union (IoU), our method also adds the temporal constraint, so that the duplicates of highestscore proposals are removed based on IoU and time interval. After that, K tuple predictions with the highest score are selected and ranked by timestamp order for the next step.

Integrating

In the integrating step, we propose a ViT-based network to generate HOI-dependent knowledge s. To handle all the K object bounding boxes bo, we use a frozen ResNet50 [He et al., 2016] and project them into a sequence of HOI embeddings [e1, ..., eK]. Following ViT's design, we append a learnable embedding etoken before the sequence, the state of which serves as the HOI-dependent knowledge s at the transformer output. We also add the 1D learnable position embeddings P = [ptoken, p1, ..., pK] to the HOI embeddings and feed the resulting sequence into the ViT network.

Given the input E0 = [etoken, e1, ...eK] + P, the network conducts the following procedures for layer n from 1 to N:

where MSA stands for the multi-head self-attention module, MLP represents multi-layer perceptron and LN denotes LayerNorm. The obtained HOI-dependent knowledge s is then merged with HOI-independent knowledge z, which will be further explained in the following section.

3.2 Adaptive Temporal Encoder

For a fair comparison, we apply the GRU followed by a linear layer as the temporal encoder backbone in consistence with the previous methods [Richard et al., 2018; Li et al., 2019a; Lu and Elhamifar, 2021]. To instruct the temporal encoder with video-level knowledge, we employ the two-branch HOIaware HyperNetwork [Ha et al., 2016], a sub-network used to learn parameters for the temporal encoder in the action predicting process. Specifically, for the linear layer in the temporal encoder, its weights and bias are separately generated by feeding learnable embedding z into the HOI-independent branch and s into the HOI-dependent branch. The pipeline can be written as follows:

where Hi, Hd represent the independent and dependent branches for weights generation, and H′ i , H′d are for bias. Finally, we apply the late fusion module F to integrate information from the two branches. Instead of fixing the network during test time in typical deep learning networks, our method adaptively adjusts the network parameters by incorporating different video-level prior knowledge into framewise action prediction, thus eliminating the potential ambiguity occurring between similar actions.

Multi-head HOI-independent Branch

Since the weight and bias can be considered as matrices with different dimensions, here we use the weight generation HyperNetwork as the example. We suppose the weight parameters generated from the HOI-independent branch are stored in matrix W z ∈ RCout×A, where Cout represents the frame representation dimension after processed by GRU. Therefore, the HOI-independent branch can be written as below:

Instead of formulating the HOI-independent knowledge as a single vector, we initialize the embedding list z = [z1, ..., zm], zi ∈ RD . These vectors are fed into the twolayer linear network Hi, yielding m different vectors with the same length Cout×A m . Finally, the outputs are reshaped and concatenated together as the W z . To ensure the correctness of dimension, Cout must be divisible by m. Formally, the network processes the following procedures:

where φ(·) represents the reshape operation. Compared with the original multi-head mechanism that uses different linear layers to project the same input, our method initializes a list of vectors and keeps the same network parameters.

Multi-head HOI-dependent Branch

Similar to the HOI-independent branch, we maintain the embedding list with the same size and separately sum them with the HOI-dependent knowledge s. The resulting vectors are projected by the two-layer linear network Hd, followed by the reshaping and concatenation to get the matrix W s .

Finally, we generate the weight of the linear layer by element-wise multiplying W z and W s :

3.3 Transcript Decoding and Training

We formulate the action segmentation problem as finding the most likely labeling based on the video features. Specifically, the optimal (ˆaO1 , ˆlO 1 ) can be obtained as follows:

In the above formula, p (xt | a) can be further transformed:

where p (a | xt) is modeled by the output of our adaptive temporal encoder. For the modeling of p (lo | ao) and p (aO1 ), the same settings with previous work [Richard et al., 2018] are utilized for the fair comparison.

We apply the constrained discriminative forward loss proposed by [Li et al., 2019a] for the network training, and provide detailed comparisons with the baseline method in the following section. It is worth noting that our method shows great flexibility and can be plugged into different existing methods.

4 Experiments

We validate our proposed method by comparing it with several state-of-the-art weakly-supervised action segmentation approaches, and discuss the effectiveness of each component in the following ablation studies.

4.1 Experimental Setup

Datasets. We conduct our experiments on two real-world instructional video datasets: Breakfast [Kuehne et al., 2014] and 50Salads [Stein and McKenna, 2013]. The Breakfast dataset contains more than 1.7k videos of people performing 10 different cooking activities, such as preparing juice or preparing salad. The cooking activities are comprised of 48 fine-frained actions. Each video has 6.9 action segments on average, and the length of the video varies from several seconds to a few minutes. The 50Salads dataset has 50 long videos with 17 different action classes. On average, each video contains 20 action instances.

Evaluation metrics. We use the following four metrics for evaluation. (1) Mean over frame accuracy (MoF) is defined as the number of correctly predicted frames divided by the total number of frames. (2) Mean over frame accuracy without background (MoF-BG) removes the background frames when calculating MoF, thus eliminating the drawback when video contains long periods of irrelevant information. (3) Intersection over union (IoU) is calculated as |GT ∩ correct| / |GT ∪ correct|, where GT stands for the ground truth frames and correct denotes the correctly classified frames. (4) Intersection over detection (IoD) is defined as |GT ∩ correct| / |GT |.

Implementation details. For the video HOI encoder, we follow the same HOI detector pre-trained on the 100K dataset as mentioned in 100 Days of Hands [Shan et al., 2020]. We set 0.5 as the HOI detection threshold and pick K = 10 bounding boxes after the selection process. For the ViT network, we replace the image patching and linear projection steps with the ResNet50 backbone, leading to 10 × 2048 input size. We use D = 128 for the dimension of both HOI-dependent and HOI-independent knowledge, and set the multi-head number as 8. For the adaptive temporal encoder, we use the 64-hidden unit GRU. We maintain the learning rate of 0.01 with 12500 epochs through the training process.

4.2 Experimental Results

We report the experimental results for two tasks, namely action segmentation where only the video is available during the inference, and action alignment where both video and transcript are provided.

Quantitative Results

We quantitatively compare our method with prior works in this section. Table 1 reports the action segmentation results on two instructional video datasets under four evaluation metrics, where the best results are indicated in bold. We can observe that by introducing HOI-aware knowledge, our method exceeds state-of-the-art methods by 1.4% MoF and 1.2% MoF-BG on the Breakfast dataset, and 0.9% MoF and 0.5% MoF-BG on the 50Salads dataset. This validates that when only video is given during testing, our method learns rich video-level knowledge and instructs the decision-making of the temporal encoder, leading to significant performance improvement on both datasets.

Table 2 shows the action alignment results following the same metrics in Table 1. Notice that in this setting the transcript is available during inference, thus providing stronger video-level knowledge compared with learned HOI-aware knowledge in our method. Despite this, our method still outperforms existing approaches on both datasets and achieves +1.8% MoF and 1.2% MoF-BG improvement on the 50Salads, which proves that the HOI-aware knowledge also helps to refine the starts and ends of predicted actions in the video.

We also report the per-activity MoF results on the Breakfast dataset under the action segmentation setting. In Table 3, we can observe that our methods outperforms the baseline approaches for most of the cooking activities as expected. For those activities containing similar actions, such as pouring cereals in "cereals", pouring coffee in "coffee", and pouring milk in "pancake", our method achieves a large performance gain (+4.3%MoF in "cereals", +13.7%MoF in "coffee" and +15.9%MoF in "pancake"). This validates the effectiveness of our method in distinguishing ambiguous actions among different cooking activities.

However, we notice that our method still suffers from low performance in tea-making and egg-making ("fried-egg" and "scrambled-egg") videos. Since our method applies the constrained discriminative forward loss and follows the temporal encoder architecture in CDFL, our method is inevitably affected by its performance. From this perspective, we still outperforms CDFL for +0.9% in "tea" and +1.3% in "scrambledegg". We also investigate the reason for performance dropping in the "fried-egg" activity. By visualizing the selected HOI bounding boxes in Figure 3, we observe that our HOI extractor tends to capture the spatula, while limited egg information is selected due to the long distance between itself and human hands. Despite the fact that people in the videos directly interact with the spatula for a long period of time, with the absence of egg detection, such HOI still could not provide enough information to instruct the temporal encoder. Therefore, the performance of our method instead degrades due to the noisy HOI-aware knowledge.

Qualitative Results

Figure 4 shows the action segmentation results of two videos on the Breakfast dataset. For the coffee-making video on the top, existing methods make the wrong predictions due to the high similarity of different pouring actions. In contrast, our method encodes strong semantic information in the HOI-aware knowledge, thus correctly classifying all the actions contained in the video. In the bottom juice-making case, when all the methods successfully capture the actions through the video, our method also shows higher accuracy in detecting the boundaries among different actions.

We also visualize how HOI detection helps to eliminate ambiguity in predicting action probability. For the cerealsmaking video in Figure 5, our HOI detector precisely captures the interactions with a cereal bag and a milk box at different timestamps. Without applying HOI-aware knowledge, the existing method suffers from low confidence when distinguishing "pour cereals", "pour water" and "pour milk" in the first frame and makes the wrong prediction in the second. In contrast, our method both makes the correct predictions and widens the probability gaps among similar actions. In the last two frames, our method also greatly improves the "pour milk" confidence, demonstrating that HOIaware knowledge provides strong instruction for a better action probability estimation.

4.3 Ablation Studies

We examine different components of our method and report the results in Table 4. The full model with the best performance is provided at the bottom for comparison. Introducing HOI-dependent knowledge leads to the most significant improvement of MoF by 3.7% and HOI-independent knowledge contributes to 0.9% MoF improvement, which demonstrates that both sources of knowledge are necessary for the HOIaware understanding. In addition, applying the multi-head mechanism further achieves +1.8% MoF.

Table 5 shows how different dimensions of HOIdependent/independent knowledge affect the action segmentation. As expected, either too small or large size would cause the performance to drop. The highest accuracy is achieved with 128 dimensions.

5 Conclusion

In this paper, we have proposed AdaAct, an HOI-aware adaptive network for video action segmentation under transcript supervision. Our method exploits essential contextual information from temporally global but spatially local humanobject interactions, and dynamically adapts its network parameters according to the videos on the fly. AdaAct achieves state-of-the-art results on two instructional video datasets for both action segmentation and alignment tasks, and especially shows strong capability in distinguishing similar actions.

Acknowledgements

This research is supported in part by the National Research Foundation of Singapore under the NRF Medium Sized Centre Scheme (CARTIN), and in part by the National Natural Science Foundation of China under Grant 62206147 and Grant 62206153. Any opinions, findings and conclusions expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and National Natural Science Foundation, China.