GUIDE: A Guideline-Guided Dataset for Instructional Video Comprehension

1 Introduction

Instructional videos guide learners how to accomplish multistep tasks such as cooking, making up and embroidering, repairing, or creating new objects. Recently, numerous instructional video datasets have been proposed [Zala et al., 2023; Tang et al., 2019; Zhou et al., 2018; Zhukov et al., 2019]. As shown in Figure 1 (a), these datasets solely focus on finegrained annotations, leading to trivial and unsystematic step captions, making it difficult to provide clear tutorial guidance. Moreover, while many instructional videos pertain to the same task, there are significant differences in the details and sequence of their steps, which increases the difficulty for beginners to learn. If there exists a model capable of analyzing various videos of the same task and organizing steps into a hierarchical structured tutorial, it will accelerate learning progression for beginners.

∗ Corresponding Author.

Our inspiration comes from two aspects. First, according to educational psychology [Bolkan et al., 2020], learners are always confused when learning an unfamiliar and challenging task because they lack relevant experience. Although previous datasets provide specific steps, their complexity would exceed learners' cognitive load. Thus, a clear guideline can help learners understand the task more efficiently. Second, planning a procedure from an instructional video is to complete a specific guideline (i.e., a procedure matches a clear intention). Since a guideline usually involves specific steps, they can be used to support the procedure generating (shown in Figure 1 (b)).

To support these, we introduce GUIDE, a Guideline-Guided dataset for instructional video comprehension. We propose a three-stage dataset construction pipeline on instructional video

…

…

Figure 2: Overview of the GUIDE dataset. The GUIDE consists of 560 task queries, each containing an average of 6.2 task-related videos. These instructional videos are divided into specific steps with timestamps and text descriptions (yellow area). Additionally, each task contains a set of guideline steps representing a common pattern shared by all task-related videos (purple area).

from video platform 1 , collecting high-quality annotations. The GUIDE contains three annotations (shown in Figure 2): (1) 560 queries: each query represents an instructional task and contains an average of 6.2 task-related videos (total of 3.5K videos), (2) 15K step segments: each video is divided into an average of 4.3 specific step segments with corresponding timestamps and text description, and, (3) 560 guidelines: each instructional task contains a set of guideline steps that represent a common pattern of the task. Moreover, each specific step has its corresponding guideline step.

In GUIDE, we propose three challenging sub-tasks for instructional video analysis. (1) Step Captioning: models have to generate captions for specific steps from videos. (2) Guideline Summarization: models have to mine the common pattern in task-related videos and summarize a guideline from them. (3) Guideline-Guided Captioning: models have to generate captions for specific steps under the guide of guidelines. We benchmark various video foundation models and language foundation models (utilize video transcription instead of visual information), including VideoChat [Li et al., 2023b], VideoLLaMA [Zhang et al., 2023], mPLUG-Owl [Ye et al., 2023], GPT-3.5-turbo [OpenAI, 2022], GPT-4 [OpenAI, 2023], Vicuna [Chiang et al., 2023] and Flan-T5 [Chung et al., 2022]. We also evaluate the performance of humans on the GUIDE for a better comparison.

The experimental results demonstrate that both video and language foundation models are struggle in all three sub-tasks. We initially observed that the specific steps generated under the ground-truth guideline guide are clearer and more accurate, indicating that the accurate guideline is helpful for generating instructional steps. Then, we explore the source of the ability to mine the guideline from multiple videos with different training settings. The results show that the single-video understanding ability is the basis of learning multiple videos, indicating that more pre-training and fine-tuning data is necessary. Subsequently, we investigate the bottleneck of video foundation models. The model demonstrates significant performance degradation compared to their text-only counterparts, indicating that more specialized visual encoders and visuallanguage bridges are needed to represent temporal procedures better. Finally, we perform a human evaluation demonstrating our dataset's promising applications in real-world scenarios. To summarize our contributions:

1 Chinese short video platform: Kuaishou.

• We introduce a novel guideline-guided instructional video comprehension dataset GUIDE, containing task-level guideline annotations and video-level systematic specific step annotations.
• We design three challenging sub-tasks in GUIDE for instructional video analysis, namely step captioning, guideline summarization and guideline-guided captioning.
• We benchmark various foundation models and conduct extensive analyses to provide detailed insights.

2 RelatedWork

2.1 Instructional Video Comprehension

Understanding instructional videos presents a significant challenge, primarily due to their inherent procedural temporal structure. Recently, many researchers have studied the analysis of instructional videos [Dvornik et al., 2023; Yang et al., 2023; Huang et al., 2018]. For instance, Yang et al. [2023] propose an end-to-end framework that augments a language model to predict timestamps and descriptions of steps seamlessly. Gu et al. [2023] propose a two-stream transformer, which constructs a video scene graph [Liang et al., 2023] for video captioning by retrieving additional knowledge. However, the steps generated by these methods are trivial and unsystematic due to the lack of guidelines, making it difficult for people to learn. While some approaches [Han et al., 2020; He et al., 2023] extract the guideline by analyzing correlations between instructional videos, they do not explore the help of the guideline for generating step captions.

2.2 Instructional Video Datasets

Existing instructional video datasets can be categorized into two types: action detection datasets [Zhukov et al., 2019; Tang et al., 2019; Kuehne et al., 2014] and step caption datasets [Zhou et al., 2018; Zala et al., 2023; Damen et al., 2018]. The former are predominantly employed for video segmentation and action recognition tasks, while the latter are used for video segmentation and step captioning tasks. For instance, COIN [Tang et al., 2019] predefines many actions and assigns these actions to instructional videos to describe procedural processes. HIREST [Zala et al., 2023] segments each video based on instructional steps and manually annotates captions for the steps. However, these datasets primarily focus on fine-grained annotations, leading to trivial and unsystematic step captions, making it difficult to provide clear tutorial guidance. Furthermore, many instructional videos related to the same tasks exhibit significant differences in specific procedures, increasing the difficulty for beginners to learn. In this paper, Our GUIDE dataset provides guideline annotations, representing the common pattern across multiple task-related videos. In addition, we annotate guideline-guided specific steps to improve the systematic nature of the data, which reduces the learning difficulty.

3 Dataset

3.1 Overview

In this section, we introduce our instructional video comprehension dataset, GUIDE. Initially, we describe the three-stage dataset construction pipeline. Subsequently, we provide an overview of the data statistics. Lastly, we introduce the three novel sub-tasks we proposed to comprehensively evaluate foundation models based on our dataset.

3.2 Dataset Construction Pipeline

GUIDE dataset construction pipeline contains three stages: video collection, automatic annotation, and manual annotation. In the Appendix A.1, we provide more details for each stage.

Video Collection In this stage, we aim to collect a large number of high-quality instructional videos. To ensure the widely-used of the GUIDE, we collect videos from 560 different instructional tasks across the 8 most common domains in our daily life. We require annotators to collect videos containing explicit instructional steps and clearly defined time boundaries between these steps. To further enhance the practicality of the dataset, we also require the collected videos that include detailed video subtitles, i.e., each step accompanied by corresponding voice explanations. More details are provided in Appendix A.1.

Automatic Annotation As illustrated in Figure 3, the automatic annotation framework contains two stages: Specific Steps Generation and Guideline Steps Generation.

In the specific steps generation, we utilize the SPGENERATOR module, comprising Whisper [Radford et al., 2023] and GPT-3.5-turbo [OpenAI, 2022], to automatically generate the specific steps for each video. Given an instructional task query Q, which contains n related videos {v1, v2, ..., vn}. We first use Whisper to generate video subtitles as AQ = {a1, a2, ..., an}. Subsequently, we feed these subtitles along with their occurrence time and our carefully crafted prompt to GPT-3.5-turbo, enabling it to generate specific steps s and corresponding timestamps t for each video. Our SP-GENERATOR can be formulated as:

where SPsteps = {[s, t]1 , [s, t]2 , ..., [s, t]n }. More details are provided in Appendix A.1.

In the guideline steps generation, we aim to extract a set of guideline steps for each task. Actually, extracting a shared guideline from all task-related videos is challenging due to the coarse granularity of task queries in the video database. Specifically, despite many videos sharing the same query, they exhibit significant variations in specific content. For instance, numerous videos fall under the task query 'Making Crayfish', but variations in ingredients and procedures lead to diverse methods, such as 'Spicy and Numbing Crayfish', 'Fragrant and Spicy Crayfish' and 'Garlic Crayfish'. We try to involve annotators in clustering videos based on the video content during the video collection stage, with the objective of identifying a single cluster that best represents the task query. However, manually clustering a large number of videos is challenging, and there are significant differences in subjective interpretations among annotators, making it difficult to establish clear clustering rules.

Hence, we utilize the GL-GENERATOR module, comprising GPT-3.5-turbo [OpenAI, 2022] and GPT-4 [OpenAI, 2023], to cluster the task-related videos and generate corresponding guideline steps automatically. We feed SPstep and crafted prompts into GPT-3.5-turbo to cluster the videos based on the content of the specific steps and their sequential order. Then, we retain only the cluster SPstep∗ with the maximum number, which includes m videos:

where SPstep∗ = {[s, t]1 , [s, t]2 , ..., [s, t]m}. Finally, we utilize GPT-4 to generate a set of guideline steps GLstep for the current instructional task, as we find during the testing process that GPT-4 is capable of generating more comprehensive and common guideline steps compared to GPT-3.5-turbo. Our GL-GENERATOR can be formulated as:

More details are provided in Appendix A.1.

Manual Annotation The results of automatic annotation cannot be regarded as the final annotations. GPT-3.5-turbo generates timestamps for each specific step based on the video subtitles' occurrence time. However, these timestamps are inaccurate because the steps in the video and the voice explanation may not coincide, and due to the lack of information in the subtitles, the specific steps may not be complete. In addition, despite providing detailed prompts for GPT-4, it still uncontrollably generates overly broad or excessively complex guideline steps. Thus, we employ manual annotation to solve these issues.

Initially, we employ an expert in each domain (e.g., chef, dancer, etc.) to adjust all guideline steps, aiming to achieve consistent granularity across all of them. Then, we require the annotators to refine the specific steps generated by GPT-3.5turbo and annotate the timestamps of steps by watching videos. It is essential that the refined specific steps contain explicit descriptions of the procedures. Furthermore, each specific step is also required to be annotated with its corresponding guideline step.

3.3 Dataset Analysis

Task Category Distribution GUIDE dataset consists of 560 tasks from 8 common domains in daily life. As shown in Figure 4. The top three most frequent domains are 'Food', 'Cosmetic' and 'Craft'.

Dataset Statistics We collect a total of 3.5K instructional videos containing 560 different common tasks in daily life. The average video duration is 103 seconds, totalling 101 hours. Each task contains an average of 6.2 task-related videos and a predefined guideline shared across all task-related videos, resulting in 560 guidelines. On average, each guideline consists of 3.7 guideline steps, yielding a total of 2.1K guideline steps with an average length of 2.9 words per guideline step. Videos are split into multiple segments based on instructional steps, with an average of 4.3 specific steps per video, totalling 15K specific steps. Each specific step is annotated with a start-end timestamp and a step caption, with an average length of 6.5 words per caption.

Comparisons to Other Datasets Table 1 compares our GUIDE dataset to other instructional video datasets. GUIDE contains numerous open domain instructional tasks videos, each with annotated specific step captions written by annotators. While HIREST [Zala et al., 2023] also provides manually written step captions for open-domain videos, its steps are trivial and unsystematic. In contrast, each instructional task in GUIDE is annotated with a guideline, representing a common pattern shared by all task-related videos. On this basis, we annotate systematic specific steps to provide a clear tutorial. Moreover, among all datasets containing handwritten step caption, GUIDE has the largest scale.

3.4 Task Definition

Step Captioning The step captioning task evaluates the models' capabilities to understand the procedural temporal knowledge of the instructional video. In this task, models have to generate a set of instructional step captions.

Guideline Summarization The guideline summarization task evaluates the models' capabilities to analyze correlations across videos. In this task, models have to mine the common pattern in task-related videos and summarize a guideline from them.

Guideline-Guided Captioning To explore the impact of guidelines on step captioning, we propose the guidelineguided captioning task. In this task, models have to generate specific step captions under the guide of guideline.

4 Experiments

4.1 Baselines

Video FoundationModels We evaluate three video foundation models on GUIDE: VideoChat [Li et al., 2023b], VideoLLaMA [Zhang et al., 2023] and mPLUG-Owl [Ye et al., 2023]. VideoChat is instantiated using BLIP-2 [Li et al., 2023a] and Vicuna-7B [Chiang et al., 2023], and combines pretrained ViT-G [Dosovitskiy et al., 2021] and GMHRA [Wang et al., 2022]. Video-LLaMA comprises a pre-trained ViT-G, an audio encoder, Imagebind [Girdhar et al., 2023], and Vicuna7B. mPLUG-Owl consists of a pre-trained ViT-L [Dosovitskiy et al., 2021], a visual abstractor, and LLaMA-7B [Touvron et al., 2023].

Language FoundationModels We evaluate four language foundation models on GUIDE: GPT-3.5-turbo [OpenAI, 2022], GPT-4 [OpenAI, 2023], Flan-T5-XXL [Chung et al., 2022] and Vicuna-13B [Chiang et al., 2023]. GPT-3.5-turbo and GPT-4 are language models with powerful performance proposed by OpenAI. Viucuna is a decoder-only architecture and Flan-T5 is an encoder-decoder architecture.

Human Performance To evaluate the gap between the foundation models' comprehension and human understanding of videos, we ask three people (they are familiar with the program but not seen ground-truth annotations) to accomplish this.

4.2 EvaluationMetrics

Following previous work [Iashin and Rahtu, 2020; Zala et al., 2023], we use METEOR [Banerjee and Lavie, 2005], CIDEr [Vedantam et al., 2015] and SPICE [Anderson et al., 2016] metrics for evaluating models. Additionally, we evaluate the models in two modes for step captioning: Entire Video Captioning (EVC): given an entire video, generate a set of step descriptions. Video Segment Captioning (VSC): given a ground-truth video segment of the step, generate a text description. More details are in the Appendix A.2.

4.3 Implementation Details

In video segment captioning (VSC), we divide the video into multiple segments based on ground-truth step timestamps. We uniformly sample 8 frames for each segment and feed them to models. In entire video captioning (EVC), we uniformly sample 32 frames for each video and feed them to models. In guideline summarization, we modify the input format of the video foundation model to enable simultaneous processing of multiple videos. We uniformly sample 32 frames from each video as input. More details are in the Appendix A.3.

4.4 Main Results

The main results are demonstrated in Table 2. We will summarize different findings in the following:

Step Captioning We observe from the experimental results that video foundation models demonstrate better performances on video segment captioning (VSC) than entire video captioning (EVC). This indicates that while the models can comprehend a specific step, they struggle to understand the entire instructional procedure. One possible explanation is that instructional videos are highly procedural, but the models' pre-training data mainly comprises videos that describe individual events. Conversely, language foundation models are not competent for VSC. This is primarily due to the absence of step-descriptive subtitles in step segments, which hinders models from generating step descriptions based on subtitles.

Guideline Summarization We observe from the experimental results that video foundation models are markedly trailing in this task. Moreover, even the strong GPT-4 demonstrates a substantial performance gap compared to human beings in this task. This indicates that these foundation models struggle to mine the correlation across multiple instructional videos.

Guideline-Guided Captioning By comparing the results of guideline-guided captioning and step captioning, we observe that both video and language foundation models perform much better with the guide of guidelines. This demonstrates the helpfulness of the guideline in generating specific steps.

4.5 Analysis

Importance of Accurate Guideline As shown in Table 3, we use the ground-truth and predicted guideline to guide the generation of specific steps respectively. The results show that there is significant improvement with the ground-truth guideline, indicating that the guideline is helpful to generate specific steps. Moreover, the results using the predicted guideline show a substantial decrease. This further emphasizes the importance of accurate guidelines.

Video Correlation Analysis Capability Mining a common guideline from multiple task-related videos depends on the model's capability to analyze correlations across multiple videos. To investigate the source of this ability, we fine-tune VideoChat (VideoChat has the best fine-tuning performance compared to Video-LLaMA and mPLUG-Owl) under three conditions: (1) single-video: fine-tuning model with single videos along with specific steps (VideoChatS), (2) multiplevideo: fine-tuning model with multiple task-relevant videos along with guideline steps (VideoChatM), (3) single-video + multiple-video: fine-tuning model under multiple-video setting based on VideoChatS (VideoChatS+M).

As shown in Table 4, we observe that the performance of VideoChatS and VideoChatM are superior to 5-shot VideoChat, and the performance of VideoChatM is better than VideoChatS.

However, VideoChatS+M shows significant improvement compared to VideoChatM. This indicates that the models' ability to analyze correlations across task-related videos is contingent upon their ability to comprehend single-video. Moreover, models only with single-video comprehension capabilities are incapable of multiple-video comprehension.

Bottleneck of Video FoundationModels To investigate the limitations of video foundation models on instructional video comprehension, we conduct experiments using the mPLUGOwl on step captioning (EVC). We explore three different settings: (1) giving mPLUG-Owl both video and audio (by using video subtitles to simulate audio), (2) giving mPLUGOwl only video, and (3) giving mPLUG-Owl only audio.

The experimental results are shown in Table 5. Surprisingly, mPLUG-Owl shows a significant improvement given only audio compared to when given only video. Additionally, we observe a substantial drop in the model's performance when adding video after only providing audio. We hypothesize that much irrelevant information is mixed in during the visual feature extraction process, which hinders the model's understanding of instructional videos. This indicates that more specialized visual encoders and visual-language bridges are needed to represent temporal procedures better.

Human Evaluation of FoundationModels To better evaluate the applicability of the GUIDE in real-world scenarios, we follow the distribution of the dataset and randomly select 53 instructional tasks (286 videos) for huaman evaluation. We compare the results of VideoChat, mPLUG-owl, and GPT-4 on step captioning (EVC) and guideline-guided captioning. The human evaluators are required to rate the output based on whether the steps are clear and easy to learn. We implemented a three-level rating system to categorize the quality of outputs. A means 'steps are very easy to learn', B means 'steps are slightly hard to learn', and C means 'steps are very hard to learn'. As shown in Figure 6, guideline-guided captioning has better results compared to step captioning, indicating that the guideline helps models generate clearer and easier-to-learn instructional steps.

4.6 Case Study

In Figure 5, we list an example generated by foundation models and ground-truth annotation for step captioning (EVC), guideline summarization and guideline-guided captioning task on videos associated with the 'Eyebrow Tutorial'. In the step captioning (EVC), the specific steps generated by VideoChat are very inaccurate. The powerful GPT-4 also has some missing and redundant steps. In the guideline summarization, both GPT-4 and VideoChat are unsuccessful in summarizing the accurate guideline. In the guideline-guided captioning, the VideoChat results show a significant improvement in clarity and accuracy, and the specific steps generated by GPT-4 are essentially entirely correct.

5 Conclusion

In this paper, we introduce a guideline-guided dataset (GUIDE) for instructional video comprehension and propose three subtasks based on the GUIDE. We evaluate various foundation models on our dataset. Experimental results show that the accurate guideline is beneficial for generating clear, easierto-learn and systemic specific steps. With different training settings, we found that the key to extracting an accurate guideline from multiple videos is single-video understanding ability, indicating more pre-training and fine-tuning data are necessary. Moreover, we observed in the ablation study that the bottleneck for video foundation models is the visual modality. We believe more specialized visual encoders and visual-language bridges are needed to to represent instructional procedures better. Finally, we perform a human evaluation demonstrating our dataset's promising applications in real-world scenarios.

Ethics Concerns

The videos are sourced from an open-source video platform that we cooperate with. Users have agreed to transfer the copyright to the platform when uploading videos, which does not involve privacy issues. Moreover, videos have been subject to strict ethical review (non-ethical content and sensitive information) by the platform.

Acknowledgements

We thank anonymous reviewers for their insightful feedback that helped improve the paper. The research in this article is supported by the National Key Research and Development Project (2021YFF0901602), the National Science Foundation of China (U22B2059, 62276083), and Shenzhen Foundational Research Funding (JCYJ20200109113441941), Major Key Project of PCL (PCL2021A06).