Weakly-Supervised Movie Trailer Generation Driven by Multi-Modal Semantic Consistency

1 Introduction

Trailers serve as a powerful tool to showcase a movie's highlights and stimulate audience interest. Unlike video summarization [Narasimhan et al., 2021; Wang et al., 2023; Narasimhan et al., 2022], which selects video clips in their original chronological sequence in the video to provide an overview, trailer generation requires not only the selection of key movie shots but also their rearrangement to maintain suspense and avoid revealing the storyline [Thompson, 1999; Hauge, 2017]. At the same time, a well-produced trailer should ensure multi-modal semantic consistency for its visual, textual, and acoustic information. For example, when a movie shot depicts a heavy storm, the music often features dense drum beats, and the narration may vividly describe the disaster, together creating a tense atmosphere. Therefore, generating a high-quality trailer requires the expertise of the editors and is time-consuming and expensive.

† Corresponding Author.

Currently, various efforts have been made to generate movie trailers automatically. Existing trailer generation methods either summarize empirical movie shot editing patterns and rules from official trailers or require manually defined fine-grained labels to learn a trailer generation model in a supervised way. However, due to the diversity and complexity of trailers and the scarcity of large-scale labeled movietrailer datasets, these methods are highly prone to overfitting and often suffer from poor generalization performance. In addition, most existing methods rely solely on single-modality data, such as muted movies and trailers (visual modality) or movie metadata (textual modality), thereby failing to satisfy the requirements of real-world commercial applications. Although some methods try to integrate multi-modal data, they fail to sufficiently explore the semantic consistency across multiple modalities within high-quality movie trailers. As a result, the trailers generated by the existing methods remain far from satisfactory.

To address the aforementioned problems, we propose MMSC, a novel automated trailer generation framework driven by Multi-Modal Semantic Consistency. As illustrated in Figure 1, given a movie with metadata and a piece of music, we formulate the trailer generation task as selecting and sorting key movie shots according to the music and the metadata (e.g., category tags and plot keywords of movies). A multi-modal model is designed to achieve this task and is trained in a weakly-supervised way on a collection of movietrailer pairs. In particular, during training, the model first derives the trailerness pseudo-scores and emotion pseudoscores for movie shots, based on the input movie and music. The trailerness pseudo-score measures the likelihood of each shot being selected for the trailer, while the emotion pseudoscore reflects the emotional intensity of each movie shot. Accordingly, we train the model under the supervision of the two pseudo-scores, encouraging the model to select and sort the movie shots that are semantically consistent with those in official trailers. In addition, given the generated trailer sequence (i.e., the sorted key shots), we leverage a large language model [DeepSeek-AI et al., 2024] to analyze movie subtitles and extract key sentences as the trailer narration, thereby improving the quality of the trailer.

In summary, our MMSC-based method utilizes multimodal information to train a trailer generation model, ensuring semantic consistency across visual, textual, and acoustic modalities when generating trailers. Guided by the trailerness and emotion pseudo-scores, the proposed model generates semantically-meaningful trailers that effectively capture the styles and themes of the movies while aligning the emotion intensities with the given music. Moreover, our method is weakly supervised, learning from movie-trailer pairs without additional fine-grained annotations.

2 RelatedWork

Recently, many learning-based trailer generation methods have been proposed to achieve better performance. For example, some methods leverage movie subtitles [Hesham et al., 2018] and movie plot summaries [Gaikwad et al., 2021] as labels, selecting key movie shots that best match the labels. The work in [Papalampidi et al., 2023] assigns each movie shot with a turning point label and sequentially selects the movie shots based on their labels when generating trailers. Besides using textual labels, some methods use frames [Liu and Jiang, 2015] or shots [Wang et al., 2020] of official trailers as positive samples, training classifiers to identify useful frames or shots from input movies and stitching them as trailers. The work in [Argaw et al., 2024] transforms the feature sequences of movies into those of trailers. More recently, some methods attempt to leverage multi-modal information for trailer generation. For example, the inverse partial optimal transport (IPOT) method in [Wang et al., 2024] selects and sorts key movie shots via aligning the visual features of the movie shots with the acoustic features of music. The work in [Liu et al., 2023] trains a trailer generation model using manually labeled emotional categories for each movie shot and selects key shots by maximizing the emotional alignment among visual, textual, and audio features.

However, the above learning-based methods require annotations for movies at the shot and even frame levels, which is expensive and time-consuming. The scarcity of such finegrained labeled data makes these methods suffer a high risk of over-fitting. Although some attempts have been made to generate trailers using unsupervised learning methods, e.g., the anomaly detection-based movie-to-trailer (M2T) method in [Rehusevych, 2019], these methods often lead to suboptimal performance due to the lack of annotations. In addition, most existing methods mainly rely on information from one or two modalities for trailer generation, and they seldom consider the multi-modal semantic consistency within high-quality trailers. Our work overcomes the limitations of the existing methods, achieving a weakly-supervised learning framework for trailer generation under the guidance of multimodal semantic consistency.

3 ProposedMethod

Given a movie M, a piece of music A, and the movie's metadata W, we aim to learn a multi-modal model to generate a trailer T based on the movie, the music, and the metadata jointly. Here, the trailer T = {V,A} consists of the video V extracted from the movie and the input music. We use the camera shot boundary detector TransNet-v2 [Soucek and Lokoc, 2024] to segment each movie and its corresponding trailer into shot sequences, i.e., M = {mi}Ii=1 and V = {vj}Jj=1 , where mi and vj denote the visual shots of the movie and the trailer, respectively. For the music, we first apply the Ultimate Vocal Remover (UVR) tool 1 to remove its vocals and then segment it into music shots that are aligned with the trailer shots, denoted as A = {aj}Jj=1 . 2 The movie metadata consists of K tokens that indicate the movie's category labels and plot keywords, denoted as W = {wk}Kk=1 .

As illustrated in Figure 1, we design a multi-modal model, which takes {M,A,W} as input and generates V via selecting and sorting key movie shots based on the multi-modal semantic features of the input. The model is learned in a weakly-supervised learning framework, which does not require any fine-grained annotations of movies.

3.1 Model Architecture

As shown in Figure 1(a), our proposed model consists of three parts, i.e., a multi-modal encoder for data representation and two predictors for shot selection and sorting.

Multi-modal Encoding

We first apply the pre-trained ImageBind [Girdhar et al., 2023] to derive the initial visual and textual features, respectively, i.e., M = fv(M) = [mi] ∈ RI×D , V = fv(V) = [vj] ∈ RJ×D , and W = ft(W) = [wk] ∈ RK×D . Note that, the movie metadata used in this study includes movie labels and movie plot keywords, which reflect the core themes and style of the movie and are important considerations in trailer design. Therefore, the text-vision encoder merges the information of the metadata into the visual representation of each movie shot, enhancing the semantics of the visual representation accordingly. In particular, given M and W, we deploy a cross-attention module between the source and the query modalities at the very first layers of the encoder, i.e.,

where {W m i ,W w i ∈ RD×D }2i=1 , σ(·) denotes the softmax operation, andM t ∈ RI×D is the metadata-dependent visual representations of movie shots.

Furthermore, taking the metadata-dependent visual representations as input, we apply one more self-attention layer to capture the temporal correlations among the movie shots, i.e.,

1 https://github.com/Anjok07/ultimatevocalremovergui.

2 Here, we can use existing music segmentation tools such as Ruptures [Truong et al., 2020] to obtain the music shots.

where {W m i ∈ RD×D }4i=2 , MLPM : RD ↦→ Rd is a multilayer perceptron (MLP), and M s = [msi ] ∈ RI×d is the contextualized visual features.

The music shots can be processed similarly. In particular, we apply the pre-trained CLAP [Wu et al., 2023] to derive the initial acoustic features for the music shots, i.e., A = fa(A) = [aj] ∈ RJ×D′ , and obtain its contextualized acoustic features by a self-attention module, i.e.,

where {W a i ∈ RD′×D′ }3i=1 , MLPA : RD′ ↦→Rd , and As = [asj ] ∈ RJ×d denotes the contextualized acoustic features of music shots.

Predictors for Shot Selection and Sorting

As aforementioned, our model selects and sorts movie shots conditioned on the given music. Therefore, we need to consider the inter-modal interaction between different modalities. In particular, we apply a cross-attention module to fuse the contextualized visual and acoustic features and get the final movie shot representation as

where W m 5 , {W a i }5i=4 ∈ Rd×d , and M c is the final movie shot representations, which are aligned cross-modal features for key shot selection and emotional prediction.

Given the final movie shot representations Mc , we use two MLPs to predict the normalized trailerness score and emotion score for each movie shot, respectively, i.e.,

Each ˆ ti indicates the predicted probability that i-th movie shot is in the trailer and ˆ ei indicates the predicted emotional intensity of i-th movie shot. The two predicted scores will be used for selection and sorting during the inference stage.

3.2 Weakly-supervised Learning Framework

We design two informative signals to supervise the training of the model, ensuring the trailerness and emotion scores derived in Eq. (5) to guide the selection and sorting of movie shots effectively. As illustrated in Figure 1(b), given a movie M = {mi}Ii=1 and its corresponding trailer T = {V,A}, where V = {vj}Jj=1 and A = {aj}Jj=1 , we calculate two pseudo-scores based on the initial features and treat them as the training labels of our model. Because the labels are derived from the input data themselves, we essentially train our model in a weakly-supervised learning framework.

Trailerness Pseudo-Score

For each movie, we assign a trailerness pseudo-score ti to the i-th movie shot based on the similarity between the movie shot and the shots in the corresponding trailer. Specifically, we first utilize the cosine similarity to compare each movie shot feature mi with all trailer shots V = [vj] belong to the same movie, i.e.,

Then, we take the maximum value in the set {si,j}Jj=1 as the trailerness pseudo-score of the movie shot mi, i.e.,

A high trailerness pseudo-score ti indicates that the movie shot mi is highly similar to a trailer shot and should be selected, whereas a low pseudo-score suggests that it is dissimilar to any trailer shot and should not be included in the trailer.

Emotion Pseudo-Score

The video and the music of the official trailer are carefully designed to express emotions consistently [Xu et al., 2015]. For example, music with a fast tempo is often paired with thrilling chase and fighting scenes, while slow and smooth music tends to complement scenes with a warm and soothing atmosphere. Therefore, besides the trailernesss pseudo-score, we further propose the emotion pseudo-score for each movie shot, quantifying its emotional intensity.

In particular, given a music shot, we calculate the average energy coefficient of its Mel Frequency Cepstral Coefficients (MFCCs), i.e., for each music shot aj, we have

where MFCC(·) denotes extracting Nj energy coefficients, Norm(·) denotes min-max normalization, and Average(·) denotes the average operation. Note that, MFCC is commonly used to extract features from music, where the energy coefficient reflects core attributes of the music signal, such as timbre, intensity, frequency distribution, and dynamic variations. Music with high energy typically sounds loud and has a large dynamic range, a broad frequency spectrum, and strong rhythm and beats. Such music often conveys intense emotions and high vitality. Accordingly, a high emotion pseudoscore indicates an intense emotional expression in the music shot, while a low score suggests relatively mild emotions.

We transfer the emotion pseudo-scores from the music shots to the movie shots: given a movie shot, we first find the most similar trailer shot and then assign the emotion pseudoscore of the corresponding music shot to the movie shot, i.e., for i = 1, 2, ..., I,

Learning with Multi-modal Semantic Consistency

Given the above pseudo-scores and the predictions in Eq. (5), we learn the proposed model to fit the pseudo-scores, i.e.,

Here, the first loss pursues the trailerness consistency between a movie and its trailer based on their visual modality. The second loss pursues the emotion consistency between a movie and its trailer's music, which leverages the visual and acoustic modalities jointly. By minimizing the two losses jointly, we learn the proposed model, ensuring its prediction results to be consistent with the semantics within the corresponding trailers.

3.3 Trailer Generation

As illustrated in Figure 1(c), given a well-trained model, we can generate a trailer based on a movie, a piece of music, and the metadata of the movie through the following steps.

Selecting Key Movie Shots

First, given the movie shots M = [mi]Ii=1 , the music shots A = [aj]Jj=1 , and the metadata W = [wk]Kk=1 , the model predicts the trailerness scores {ˆ ti}Ii=1 and the emotion scores {ˆ ei}Ii=1 for the movie shots. Accordingly, we select the movie shots corresponding to the J highest trailerness scores to construct the target trailer. The set of the movie shots, denoted as MS = {mi}i∈S, where S = arg Top-Ji∈{1,...,I} ˆ ti , covers the key shots that are likely to appear in the trailer.

Sorting Selected Movie Shots

As mentioned earlier, to maintain emotion consistency, the order of selected shots is crucial for constructing a highquality trailer and should be aligned with the music shots. To achieve this aim, we sort the selected movie shots based on their emotion scores. In particular, given the music shots, we calculate their energy coefficients {cj}Jj=1 via Eq. (8), leading to a sequence reflecting the temporal dynamics of the emotions behind the music. We sort the selected shots based on their emotion scores {ˆ ei}i∈S, ensuring that the order of the sorted emotion scores matches with that of {cj}Jj=1 .

Post-processing

Besides selecting and sorting movie shots, we further design three post-processing steps to make the generated trailer more realistic. Firstly, given the sorted key movie shots, we follow the previous work [Wang et al., 2024] and align the duration of each shot with its corresponding music shot, as shown in Figure 2(a). For movie shots that exceed the duration of the corresponding music shots, we trim the excess parts of the movie shots. For movie shots that are shorter than the corresponding music shots, we use the adjacent shot from the original movie with higher trailerness score to fill the duration gap. Secondly, we use DeepSeek-V3 [DeepSeek-AI et al., 2024], a pre-trained large language model (LLM), to analyze and select the movie's subtitles. As shown in Figure 2(b), the LLM takes the movie's subtitles with timestamps and some instructional prompts as input and selects some subtitles as the narrations of the generated trailer. Based on the timestamps of the subtitles, we can automatically extract the corresponding audio from the movie. Finally, we determine the positions of the selected narrations automatically. As shown in Figure 2(c), we utilize MiniCPM-V 2.6 [Yao et al., 2024], a multi-modal LLM for video captioning, to generate a onesentence description for each shot of the generated trailer. We extract the textual features of the shot descriptions and the selected narrations and calculate their pairwise similarities. Accordingly, we associate each narration with a shot by maximizing the sum of the similarities between all narrations and the shot descriptions under the constraint that the narrations do not overlap. This problem can be solved efficiently using dynamic programming (DP) [Bellman, 1966]. After adjusting the durations of the shots and merging the audio of the narrations into the music, we derive the final generated trailer.

4 Experiments

4.1 Implementation Details

Dataset

We construct a dataset of 500 movies and 922 trailers spanning 18 genres and 30 years based on IMDb [IMDb, 2025] tags and years. The movie labels and movie plot keywords of each movie are collected from IMDb. Following the work in [Wang et al., 2024], we resize movies in the dataset to 320p for learning convenience and efficiency and remove human vocals from the audio of trailers using UVR. The entire dataset is divided into training, validation, and test sets in a ratio of 85:5:10. The test set contains 74 movie-trailer pairs, denoted as Test-74. In addition, previous trailer generation methods [Ganh¨or, 2014; Rehusevych, 2019; Irie et al., 2010; Xu et al., 2015; Wang et al., 2024] are tested on an independent test set with eight movies. Therefore, besides our dataset, we also use the eight movies as our test data, denoted as Test-8. The movies in Test-8 do not appear in the training and validation sets of our dataset.

Baselines

We compare our MMSC-based method with state-of-theart trailer generation methods, including V2T [Irie et al., 2010], M2T [Rehusevych, 2019], PPBVAM [Xu et al., 2015], TGT [Argaw et al., 2024], and IPOT [Wang et al., 2024]. When evaluating the capability of shot selection, we also choose state-of-the-art video summarization methods as baselines, including a commercial video summarization software Muvee [Ganh¨or, 2014], VASNet [Fajtl et al., 2018], CLIPIt [Narasimhan et al., 2021], and OTVS [Wang et al., 2023].

Evaluation Metrics

Following the work in [Wang et al., 2024], we employ Precision, Recall, and F1-score (F1) to evaluate the performance of shot selection given the shots in the ground truth trailers (i.e., official trailers). Following the work in [Argaw et al., 2024; Liu et al., 2024], we employ Levenshtein distance (LD) and Pairwise agreement accuracy (AA) to evaluate the consistency between the order of the correctly selected movie shots with the order of ground truth trailer shots.

Model Architecture and Hyperparameter Settings

Each attention layer in our multi-modal model consists of two Transformer encoders with four attention heads. Each MLP in our model consists of two linear layers connected by a GELU function. The Adam [Kingma and Ba, 2015] optimizer is used during training, with an initial learning rate of 1e-5 and a cosine warm-up scheduler with β1=0.9 and β2=0.999. Our model is trained for 1,500 epochs with a batch size of 10, on a single NVIDIA RTX 3090.

4.2 Objective Evaluation

In Table 1, we compare the performance of our method and baselines on movie shot selection and sorting. IPOT [Wang et al., 2024] introduces a constraint in shot selection to avoid spoilers, that is, limiting the selection in the first 90% of the movie shots, i.e., IPOT (90%). However, to better evaluate the model's selection capability, we perform selection and ranking across all movie shots as other baselines. To ensure a fair comparison, we also provide the results of IPOT's selection across all movie shots in Table 1. TGT [Argaw et al., 2024] uses the encoder-decoder architecture to generate the trailer sequence. Since there is no official trailer as input to limit the length of the trailer sequence generated by the TGT decoder during the inference phase, we set the length of the decoded sequence to no more than 150. From the results, we can see that our method achieves the best performance on most selection and sorting metrics. Especially on Test-74, our proposed method outperforms the state-of-the-art method IPOT by a large margin, i.e., 5.93%, 6.80%, and 6.34% on the Precision, Recall, and F1-score, respectively. Besides, the performance on two sorting metrics also improves by 5% and 4% in AA metric over the state-of-the-art method on the two test sets, respectively.

Since closely adjacent shots in a movie often share similar content, following prior work [Argaw et al., 2024], we also relax the evaluation criteria by expanding the ground truth to include the R shots before and after each official trailer shot in the original movie. This prevents overly strict evaluation from overlooking reasonable shots and underestimating our model's performance. In Table 2, this expansion leads to improved model performance across all shot selection metrics, indicating our model's ability to generalize to shot variations.

Figure 3 provides an example comparing the trailer generated by our method with the official trailer. After selecting the appropriate movie shots for the trailer, our method calculates the energy coefficient for each shot of the given music, as visualized in the last row of Figure 3. In our generated trailer, shots with higher emotion scores correspond to the music shots with higher energy. This is attributed to emotion consistency loss, which adjusts the order of the selected movie shots to ensure that the emotion scores of the movie shots align with the energy coefficients of the music shots.

4.3 Subjective Evaluation

Since the goal of producing trailers is to attract audiences, we conduct user studies to evaluate how well the trailers generated by our method and baselines perform compared to professionally edited trailers. We invite 30 volunteers (15 females and 15 males) to assess the quality of the trailers generated by different methods from four aspects: Theme&Style: "How well does the trailer convey the theme and style as the movie?", Rhythm: "How well do the visuals match the rhythm of the music?", Attractiveness: "How attractive is the trailer?", Appropriateness:"How close is the trailer to a real trailer?". Following the previous work [Xu et al., 2015; Wang et al., 2024], we resize all trailers to the same resolution, anonymize the names of the trailer generation methods, and upload them to the website for volunteers to rate. We also provide the official trailer (RT) as the reference standard with the highest score (set to seven). According to the average and median scores of the four aspects shown in Figure 4, our method significantly surpasses all baseline methods.

4.4 Ablation Study

Movie Metadata

We study the importance of metadata guidance by training our model with textual content at different granularity levels.

The movie labels represent the narrative style of the movie, providing coarse-grained categorical information. Movie plot keywords capture the core themes and key visual elements of the movie, providing fine-grained content descriptions. In Table 3, training the model without any metadata leads to severe performance degradation across all metrics, while applying metadata at both granularity levels contributes to model performance. Compared to abstract labels, the more specific plot keywords can better guide the model to learn key events that are more likely to be selected by professional editors.

Network Components

In Table 4, we analyze the impact of different network components by training the model with the single-modality context encoder or without any context encoders. The results show that both context encoders enhance overall model performance by integrating contextual information and improving feature discrimination between similar movie shots or similar music shots. Notably, the models corresponding to the suboptimal results differ between the two test sets. We speculate that this may result from Test-8, which has fewer trailers and is more sensitive to atypical trailers, leading to inconsistencies with the larger test set Test-74.

Emotion Consistency Loss

To investigate the effectiveness of emotion consistency loss in shot sorting, we compare shot sequences obtained by sorting randomly (i.e., Random), sorting according to the chronological order in the movie (i.e., Chronological), and sorting based on the emotion consistency (i.e., Proposed) against the official trailer sequence. We propose three metrics to measure the detailed order consistency, namely, Ascending consistency (AC), Descending consistency (DC), and the harmonic mean (HM) calculated based on these two metrics, i.e., HM = 2×AC×DC AC+DC . Note that for any pair of two different shots (mi, mj), where i < j, with their shot indices (ai, aj) in the generated trailer, there are two possible order relationships of these two shots in the ground truth trailer, including ascending order ai < aj and descending order ai > aj. The AC metric computes the number of correctly predicted pairwise ascending orders in the generated trailer among all possible pairwise ascending orders. The DC metric does the same for descending orders. As shown in Table 5, since we calculate metrics on the shot sequence of correctly selected shots by our model and all pairwise orders in the chronological sorted sequence are in ascending order, DC and AC of the chronologically sorted sequence are always fixed at 0 and 1, respectively. Compared with the randomly sorted and the chronologically sorted sequences, the sequence generated based on emotion consistency achieves the best performance in DC and HM, demonstrating that our proposed strategy effectively captures reverse-order editing techniques used in official trailers to prevent spoilers and enhance suspense.

5 Conclusion

In this work, we propose a weakly-supervised framework for trailer generation, driven by multi-modal semantic consistency. Our method leverages data from all modalities to generate movie trailers that contain all modality elements. The entire process is fully automated, eliminating the need for additional fine-grained annotation. Experiments demonstrate that our MMSC-based method outperforms state-of-the-art trailer generation and video summarization methods on both objective and subjective evaluation metrics.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (62102031), and the foundation of Key Laboratory of Artificial Intelligence, Ministry of Education, Shanghai, P.R. China (AI202409).