FunCineForge: A Unified Dataset Pipeline and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes

1 Introduction

Movie Dubbing [Chen et al., 2022], also known as VisualVoice-Cloning (V2C), is a task that synthesizes speech for video clips from scripts using target timbres. This technique has significant potential in television and film production, digital media, and video editing. Unlike Text-to-Speech

∗Corresponding author.

(TTS), movie dubbing requires incorporating visual information from character faces to precisely control the duration, emotion, and prosody of the synthesized speech.

Existing movie dubbing methods can be broadly categorized into three research paradigms. The first category focuses on improving the synchronization between the synthesized speech and lip movements by learning visual features from the speaker's lip region. For example, a text-video aligner is introduced in NeuralDubber [Hu et al., 2021], a duration aligner is introduced in HPMDubber [Cong et al., 2023], and a phoneme-guided lip aligner is introduced in StyleDubber [Cong et al., 2024b]. These methods aim to explicitly control phoneme-level speech duration by mapping textual phonemes to video frames. However, their alignment performance heavily relies on the presence of clear and visible lip regions, making them effective only for simple, frontal, and high-resolution monologue scenes. In live-action television, complex cinematic scenes such as facial occlusion, frequent shot changes, multiple coexisting speakers, large speaker-camera distances, and low-resolution frames are common, which prevents reliable audio-visual alignment.

The second category focuses on enhancing dubbing quality and speaker similarity. For example, Speaker2Dubber [Zhang et al., 2024] improves synthesis quality by via prosodyenhanced acoustic pretraining. FlowDubber [Cong et al., 2025] introduces dual contrastive learning and LLM-based flow matching guidance, enhancing speech quality. ProDubber [Zhang et al., 2025b] proposes prosody-enhanced acoustic pretraining and acoustic-disentangled prosody adaptation to improve prosody modeling and expressiveness. EmoDubber [Cong et al., 2024a] employs a flow-based user emotion controlling with positive and negative guidance mechanisms to manipulate dubbing emotions. Authentic-Dubber [Liu et al., 2025] enhances emotional retrieval and matching by constructing a multimodal reference footage library and employing emotion-similarity-based retrieval augmentation. To better clone the reference speaker timbre, most above works adopt acoustic encoders to extract speaker embeddings, while Lee et al. [2023] learn speaker identity from facial features to capture speaker paralinguistic information. However, these methods heavily rely on clear, single-speaker faces and consistent timbre. In the presence of shot-level facial transitions, speaker timbre switching, or intermediate silent segments, alignment becomes difficult, and both dubbing quality and speaker similarity degrade substantially.

The third category leverages Multimodal Large Language Models (MLLMs) to fuse information from multiple modalities for movie dubbing. DeepDubber-V1 [Zheng et al., 2025] utilizes multimodal Chain-of-Thought (CoT) reasoning on visual inputs to infer dubbing styles and speaker attributes, generating visual-derived clues that serve as instructions for the dubbing MLLM. InstructDubber [Zhang et al., 2025a] enables controllable dubbing by conditioning the model on textual prompts that describe speaker identity and emotional clues as an additional modality. These methods primarily rely on natural language instructions to control speech style and emotion, offering a more intuitive and interpretable paradigm for movie dubbing. However, the effectiveness of MLLMs critically depends on large-scale datasets, whereas existing datasets are limited in scale and lack rich annotations, which constrains their effectiveness in real-world scenarios.

To address these issues, we propose FunCineForge, a unified framework for zero-shot movie dubbing that integrates an automated data production pipeline with an MLLM-based dubbing model for diverse cinematic scenes, as illustrated in Fig. 1. The pipeline combines the advantages of lightweight specialized models and general-purpose MLLMs, innovatively adopting a "specialized model prediction + multimodal CoT correction" strategy to refine both transcription and speaker diarization, significantly reducing word error rates and speaker identification errors. Based on this pipeline, we construct CineDub, a large-scale dubbing dataset comprising two language subsets, CineDub-CN and CineDub- EN, with over 9,000 hours of speech in total. Building upon CineDub, we develop an MLLM-based dubbing model that, for the first time, explicitly introduces a temporal modality via a Timestamp-Speaker tokenizer, enabling precise control over temporal alignment. Specifically, the model jointly models when speech occurs, who is speaking, and what is being spoken, while striving for fine-grained lipspeech alignment within speech-active intervals. It is further equipped with an improved flow matching module that supports flexible speaker switching. As a result, the proposed dubbing model synthesizes highly natural speech with precise temporal alignment and lip sync, effectively handling complex cinematic scenes and speaker switching. We provide a project website 1 to showcase synthesized demos and CineDub dataset samples, and the code repository 2 has been open-sourced.

2 RelatedWorks

2.1 Dubbing Datasets

Dubbing datasets used for model training, including V2CAnimation [Chen et al., 2022], Chem [R et al., 2020], and GRID [Cooke et al., 2006], are limited in scale. Even the largest one, V2C-Animation, contains only 10,217 videoaudio-text clips with an average duration of 2.4 seconds, totaling 6.8 hours of data. These datasets heavily rely on manual annotation or crowd-sourced services, are difficult to scale, and are limited to monologue scene due to excessively short clips. The scarcity of data and limited, inconsistent annotations also hinder the effectiveness of MLLMs in complex cinematic scenes, highlighting the urgent need for a fully automated pipeline to construct large-scale dubbing datasets.

2.2 LLM-based Text-to-Speech

Recent advances in large language models (LLMs) have reshaped the field of TTS. Most LLM-based TTS methods adopt a two-stage paradigm, where an LLM first generates speech semantic tokens from text and an acoustic decoder converts them into acoustic features such as Melspectrograms, achieving significant advantages in synthesizing expressive speech. Representative models include CosyVoice 3 [Du et al., 2025], IndexTTS 2 [Zhou et al., 2025], FireRedTTS 2 [Xie et al., 2025], VibeVoice [Peng et al., 2025] and Spark-TTS [Wang et al., 2025], In parallel, flow matching has emerged as an effective acoustic decoding strategy. Matcha-TTS [Mehta et al., 2023] adopts optimaltransport conditional flow matching for single-speaker TTS, while CosyVoice 2 [Du et al., 2024] demonstrates the effectiveness of integrating flow matching with LLMs. Despite their success, these methods cannot provide the speech with desired emotion and audio-visual sync for movie dubbing.

2.3 Speaker Diarization for Cinematic Scenes

Speaker Diarization task [Hsu et al., 2017] aims to segment an audio recording into regions corresponding to different speakers by estimating both the number of speakers and the start-end timestamps of their speech segments. In complex cinematic scenes with multi-speaker interactions, accurate movie dubbing requires explicit knowledge of which speaker is active at each temporal interval, making robust speaker diarization a crucial prerequisite. Most speaker diarization methods [Zheng et al., 2020; Li et al., 2023] follow a feature extraction and unsupervised clustering pipeline, consisting of: (1) voice activity detection (VAD) to remove non-speech regions; (2) sliding-window extraction of speaker embeddings using models such as Res2Net [Gao et al., 2019] and CAM++ [Wang et al., 2023]; and (3) unsupervised clustering of speech segments. 3D-Speaker [Zheng et al., 2023] further incorporates visual embeddings for visually enhanced clustering, achieving notable performance gains. Despite these advances, speaker diarization in long-form videos remains challenging due to: (1) clustering errors caused by a large number of speakers in long-form videos; (2) failures under overlapping multi-speaker speech; and (3) speaker confusion induced by similar tones. Consequently, effective segmentation plus speaker diarization for long-form cinematic scenes remain open problems.

1 https://funcineforge.github.io/

2 https://github.com/FunAudioLLM/FunCineForge/

3 FunCineForge

3.1 Dataset Pipeline

To enable training of dubbing models in diverse cinematic scenes, FunCineForge dataset pipeline transforms raw longform television sources into structured multimodal data. It is specifically designed to address challenges such as high word error rates in transcription and inaccurate speaker diarization. As shown in Fig. 2, the pipeline comprises several coordinated workflows, summarized in four stages.

Video Preprocessing and Segmentation

For long-form videos, we first perform audio-visual preprocessing and segmentation. To improve vocal clarity and suppress background music, thereby reducing word error rates and diarization error rates, we employ a MelRoFormer [Wang et al., 2024] module to separate vocal and instrumental components, yielding clean vocal tracks. Next, an FSMN-Monophone VAD [Zuo et al., 2023] model optimized for long-duration speech is applied to extract speechactive segments. Based on the timestamps, the original video is subsequently segmented into shorter video-audio clips. Subsequently, an overlapped speech detection module is applied to the vocal tracks to identify and discard clips containing multiple simultaneous speakers, ensuring the reliability of downstream speaker diarization.

Timestamped Transcription and Alignment

The vocal tracks of the segmented clips are transcribed into text with punctuation using automatic speech recognition (ASR) models, and character-level timestamps are predicted via forced alignment. Specifically, we employ FunASR [Gao et al., 2023] for Chinese and Qwen3-ASR [Shi et al., 2026] for English, which exhibit slightly different performance in terms of word error rate and temporal alignment accuracy. To further reduce temporal discrepancies, we apply an additional alignment step using MMS to refine the audio-text sync.

Audio-Visual Speaker Diarization

A visually enhanced speaker diarization framework is adopted to integrate audio and visual embeddings for robust speaker attribution. In the audio modality, CAM++ [Wang et al., 2023] is employed to extract speaker embeddings from clean vocal tracks, which implicitly capture speaker activity information, while the CosyVoice 3 speech tokenizer [Du et al., 2025] encodes audio into speech tokens at 25 Hz. In the visual modality, frame sampling is first performed by selecting one frame every five frames for 25 fps videos. All faces in these sampled frames are detected using a lightweight face detection module 3 to obtain candidate active frames. The detected faces are then scored by TalkNet-ASD 4 module, and the face with the highest score in each active frame is selected as the active speaker. For the selected face, a FANbased [Bulat and Tzimiropoulos, 2017] module is applied to detect 2D facial keypoints, from which lip coordinates are obtained and raw face and mouth regions are extracted. An adaptive face encoder [Huang et al., 2020] is used to extract face embeddings of the active speaker, followed by normalization to suppress expression-related variations and retain identity-related representations. Finally, a audio-visual joint unsupervised clustering strategy [Zheng et al., 2023] is applied, where audio speaker embeddings serve as the primary modality and normalized visual face embeddings act as auxiliary cues, producing robust speaker clustering results formatted as Rich Transcription Time Marked (RTTM) files.

3 https://github.com/Linzaer/Ultra-Light-Fast-Generic-FaceDetector-1MB

4 https://github.com/TaoRuijie/TalkNet-ASD

Multimodal CoT Correction

To enrich each clip with paralinguistic information for controllable dubbing, the pipeline augments video-audio clips with structured character and emotion annotations, which are traditionally costly to obtain manually. Meanwhile, ASR transcripts often contain lexical and punctuation errors, and speaker diarization may suffer from missing or duplicated speaker identities. To address these issues, a multimodal Chain-of-Thought correction strategy is introduced.

Specifically, Gemini-3.1-Pro is employed as a multimodal verifier agent to perform cross-modal reasoning over clean vocal tracks, ASR transcripts, and speaker diarization tuples (start time, speaker ID, end time). It first identifies active speakers over time and corrects transcription errors, and then infers speaker attributes, including gender, age group, identity, timbre, tone, speech rate and emotion. These results are further summarized into structured instructions. To ensure reliability, strict prompt templates are adopted, and a bidirectional verification mechanism is introduced to address occasional hallucination in MLLM outputs. Samples with large discrepancies (e.g., Levenshtein distance > 72%) between corrected and original transcripts are discarded, while low-discrepancy samples retain the corrected transcripts. The threshold is selected based on the empirical distribution of transcript discrepancies. Similarly, samples with inconsistent speaker identities are filtered out, and uncertain attribute predictions are replaced with <unknown> to mask unreliable labels. Finally, the remaining samples are categorized into monologue, narration, dialogue, and multi-speaker scenarios based on facial evidence and the number of active speakers.

3.2 DubbingModel

The architecture of the dubbing model is illustrated in Fig. 3. Given an input video clip, frame-level facial sequences Face ∈ RT of the active speaker are extracted, where T denotes the number of frames. For the video clip containing N speech segments, each segment is associated with a dubbing script Text, an instruction Instruct, and a timestampspeaker tuple T = (start, spk, gender, age, end). In addition, reference speech samples ref M from M speakers (M ≤ N) are provided for timbre cloning. The model synthesizes Mel-spectrograms that are temporally aligned with the silent video and exhibit precise lip sync,

The dubbing model is trained in two stages: MLLM training and flow matching training. After flow matching, the synthesized Mel-spectrograms ˆ Y are converted into waveforms using the widely used HiFiGAN vocoder.

MLLM with Multimodal Alignment

First, the instruction and script are concatenated as "〈|startofprompt|〉Instruct〈|startofprompt|〉Text", and tokenized using a byte pair encoding (BPE)-based tokenizer to obtain the combined text tokens XText.

Face and lip images in Face are extracted from 25-fps videos and uniformly sampled every five frames, resulting in 5-fps visual sequences. These downsampled face and lip frames are efficiently encoded into facial and lip representations using the face encoder [Huang et al., 2020] and a lip encoder [Cong et al., 2023], respectively, without normalization, thereby preserving expression-related variations. The resulting features are then upsampled to match the original video frame rate, producing face embeddings ET Face and lip embeddings ETLip . Both ET Face and ETLip are frame-level onedimensional continuous feature sequences with the same sequence length as the speech tokens XSpeech. These features are inherently sparse, as the speaker's face is not consistently visible across all frames in real-world videos.

To handle frequent shot changes, speaker switching and facial occlusion in complex cinematic scenes, and to achieve accurate lip-speech alignment, we design a multimodal alignment mechanism that combines strong and weak supervision to jointly model when speech occurs, who is speaking, what is being spoken, and fine-grained lip-speech alignment.

(1) The provided timestamp-speaker tuples T N serve as strong supervision, explicitly constraining when speech occurs and who is speaking, and enabling precise temporal alignment. A dedicated frame-index codebook is introduced to encode start-end timestamps. Given that each video clip is limited to 60 seconds and sampled at 25 fps, the maximum number of timestamp tokens is 1500. To jointly model temporal and speaker-related information, a Timestamp-Speaker Tokenizer (TST) is designed to map timestamps and speaker attributes into a discrete token sequence XTS. Specifically, each segment is represented by tokens corresponding to speaker identity token XSpk, gender token XGender for male and female, and XAge corresponding to child, teenager, adult, middle-aged, elderly age groups. Attribute tokens corresponding to <unknown> annotations are masked during training. In addition, a frame-level voice activity indicator is defined as τt = 1 if speech is present at frame t, and τt = 0 otherwise. Let ˆ τt denote the predicted voice activity derived from the synthesized speech tokens. A voice activity loss is constructed over XSpeech as

(2) For speech tokens XSpeech, the supervision focuses on what is being spoken. Accordingly, the speech token loss is defined as a cross-entropy loss under conditions CLM,

(3) Visual features are introduced to enhance the MLLM's understanding of speaker identity and emotional expression, and to provide weak supervision for fine-grained lip-speech alignment when lip movements are clearly observable. Frame-level face embeddings ET Face , temporally aligned with the speech tokens XSpeech, are fed into the MLLM. A contrastive learning objective is introduced between frame-level lip embeddings ETLip and speech token embeddings ET ST = gst(Emb(XSpeech)),

where τt is voice activity indicator activated in speech-active frames, wt down-weights frames with weak lip motion, and γ is a temperature parameter.

Flow Matching with Speaker Switching

The flow matching module is built upon the CosyVoice 3 [Du et al., 2025] backbone to reconstruct Mel-spectrograms from speech tokens. However, the original design conditions only on a single global speaker embedding, limiting its ability to model speaker transitions in dialogue scenarios. To address this limitation, a speaker switching concatenation strategy is introduced, conditioned on M reference speakers ref M . A

universal speaker encoder [Zheng et al., 2023] is employed to extract audio speaker embeddings EM Spk for ref M . Given timestamp-speaker tuples T N , the last silent token preceding each speech segment is identified via bidirectional search. The corresponding speaker embedding ESpk ∈ EM Spk is then inserted immediately after the last silent token and before the onset of speech tokens, which are encoded into ET ST ∗ using a causal upsampling encoder. This design enables explicit speaker switching aligned with timestamp boundaries.

The concatenated feature sequence is then fed into a Diffusion Transformer (DiT) [Peebles and Xie, 2022] as conditional context Cflow for flow matching training,

where Cflow = Concat(ET ST ∗ , EM Spk (T N )), u ∼ U(0, 1), Y1 ∼ N(0, I), and Yu = (1 - u)Y0 + uY1. Within the DiT module, local attention is adopted to restrict the model's receptive field to speech-active segments. During training, the conditioning Cflow is randomly dropped with a fixed probability to enable classifier-free guidance at inference stage.

4 Experiments

4.1 Implementation Details

Data preprocessing. All videos are standardized to MP4 format, with opening and ending credits removed. Video frames are sampled at 25 fps, and audio is resampled to 16 kHz. The speech tokenizer adopts an FSQ codebook of size 6,560, and the frame-index codebook size is 1,500.

Training setup. The MLLM and flow matching modules are trained separately on the CineDub dataset for over 20 epochs. The CosyVoice3-0.5B checkpoint is used for initialization, providing strong speech naturalness. Training is conducted on 8 NVIDIA A100 GPUs, using the AdamW optimizer with a peak learning rate of 1e-4, 2,000 warm-up steps, and a batch size of 2 × 104 tokens.

Baselines and evaluation. For evaluation, we select several open-source SOTA baselines specifically designed for movie dubbing, and use their official checkpoints trained on the V2C-Animation, Chem, or GRID datasets for inference. For instruction-driven methods, DeepDubber-V1 and InstructDubber, we modify their implementations and retrain them on the CineDub dataset to enable comparison under the same data setting.

Cross-dataset analysis. To evaluate dataset quality, the FunCineForge model is additionally trained on the combined V2C-Animation, Chem, and GRID datasets using only monologue scenes, with empty instruction and timestamp-speaker tuples. This setting enables comparison of how CineDub data types and annotation richness improve model performance.

4.2 EvaluationMetrics

We adopt a comprehensive set of objective and subjective metrics to evaluate dataset quality, dubbing quality, lip sync, temporal alignment, speaker timbre similarity, emotional consistency, and instruction-following consistency.

MCD-DTW & MCD-DTW-SL. Mel Cepstral Distortion with Dynamic Time Warping (MCD-DTW) and its variant MCD-DTW-SL [Chen et al., 2022], which incorporates duration-aware weighting, measure the difference between synthesized speech and ground-truth recordings.

CER &WER. Character Error Rate and Word Error Rate are used to measure pronunciation accuracy, using Whisperlarge-v3 5 as the ASR model.

UTMOS. UTMOS 6 is a 5-point speechmean opinion score (MOS) predictor to assess the naturalness of synthesized speech.

LSE-C & LSE-D. To accurately measure lip sync, we use Lip Sync Error Confidence (LSE-C) and Lip Sync Error Distance (LSE-D) [Chung and Zisserman, 2016]. Both metrics are computed based on the pre-trained SyncNet, which is widely adopted in lip-reading task.

DER & SBFE. To evaluate speaker-aware temporal alignment, we adopt Diarization Error Rate (DER) and Speaker Boundary Fidelity Error (SBFE). DER measures the proportion of time that is not correctly attributed to the reference speaker, including missed speech, false alarm, and speaker confusion. Lower DER indicates better overall temporal accuracy.

5 https://huggingface.co/openai/whisper-large-v3

6 https://github.com/tarepan/SpeechMOS

Given ground-truth time intervals Ii = [tstart i , tend i ) from timestamp-speaker tuples T N = {(tstart i , si, gi, ai, tend i )}Ni=1 , and predicted intervals ˆ Ii extracted from synthesized speech via VAD, SBFE quantifies truncation and leakage via a balanced asymmetric measure:

where SBFE ∈ [0, 1], and lower values indicate more accurate alignment and faithful speaker switching.

SSIM. Speaker cosine similarity is used to evaluate speaker timbre similarity between synthesized speech and reference speech within each speaker's active time interval.

ESIM& ES-MOS. Emotional learning depends on the input face images and instructions. We employ emotion2vec 7 to extract emotion vectors from synthesized speech and groundtruth recordings, and compute the cosine similarity (ESIM) between them. We conduct a subjective 5-point MOS on emotional and style consistency (ES-MOS), where human raters evaluate how well the generated speech matches intended emotion and style in the instruction description.

4.3 CineDub Dataset and Quality Evaluation

To train the dubbing model and evaluate the quality of the dubbing dataset produced by our pipeline, we carefully collected over 400 raw television series covering diverse genres, including Chinese dramas, American dramas, and British dramas. Strict selection criteria are applied: non-documentary content, standard pronunciation, prominent vocal tracks, minimal colloquial speech, and unobstructed faces (i.e., no masks or veils). Using the pipeline, over 9,000 hours of effective speech clips are retained, with an average clip length of 10.8 seconds. Non-neutral emotional speech accounts for 38.4% of the dataset, and the average instruction length reaches 55 words with a variance of 16, covering character attributes, tone, speech rate, voice timbre, and emotion, indicating strong diversity and robust multi-scale annotations. Additional dataset statistics are provided in Fig. 4. We select four samples from each television series, corresponding to the four scene categories, to construct the test set. + Table 1 compares the intrinsic quality reported in the GT rows for the CineDub dataset (with and without Multimodal CoT Correction) and the combined V2C-Animation, Chem, and GRID datasets, as well as the performance of the FunCineForge dubbing model trained on these datasets. Results show that models trained on these three small-scale, limited-annotation datasets tend to underfit lip sync, leading to lower alignment accuracy, and produce speech with lower naturalness. In contrast, training on our CineDub dataset significantly improves pronunciation quality and lip sync. Moreover, training on the corrected CineDub dataset leads to clear improvements across all metrics compared with the uncorrected version, especially in pronunciation accuracy and alignment, demonstrating that the proposed bidirectional verification mechanism between lightweight specialized models and general-purpose MLLM significantly reduces word and punctuation errors in the training data, and alleviates duplicated and missing speaker identities in speaker diarization predictions, thereby substantially improving dataset quality.

7 https://github.com/ddlBoJack/emotion2vec

4.4 Scene Analysis and Baseline Comparison

As shown in Table 1, we evaluate the performance of the FunCineForge dubbing model across four scene categories. The results indicate that our model achieves higher speech naturalness and alignment accuracy in monologue and narration scenes.

Specifically, in monologue scenes, alignment relies on temporal constraints and weakly supervised lip movements; in narration scenes, where no active facial frames are present, the model degrades to a time-controllable TTS model. In dialogue and multi-speaker scenes, which are inherently more challenging, the model maintains strong temporal alignment, accurate lip sync, and effective timbre switching. With longer facial sequences and richer instructions, it further exhibits more accurate emotional expression and improved instruction-following capability. However, as the duration of dubbed clips increases and scene complexity grows, temporal alignment and lip-sync performance degrade, and model robustness decreases.

Furthermore, to verify the effectiveness of FunCineForge dubbing model, as shown in Table 2, we conduct comparisons under a unified monologue setting, where all SOTA methods are evaluated on the same test samples. On three opensource dubbing datasets, our model achieves consistently superior performance in terms of speech quality, WER/CER, and alignment accuracy. In addition, we compare our model with two MLLM-based dubbing models trained on the CineDub dataset and evaluate them on the same monologue test set. The results demonstrate that FunCineForge dubbing model achieves significant improvements across all evaluation metrics, highlighting the effectiveness of the proposed model architecture and design.

4.5 Ablation Studies

To analyze the contribution of the key components in our dubbing model, we conduct ablation studies on the CineDub dataset. Results are summarized in Table 3.

Effectiveness of timestamp-speaker tuples T N . Removing the timestamp-speaker tokenizer, the tuple inputs T N and loss LVA, causes clear degradation in temporal alignment. Both SBFE and LSE-D increase substantially, especially in dialogue and multi-speaker scenes, where speaker truncation and leakage frequently occur. This shows that facial features alone are insufficient for long sequences with frequent shot changes and speaker switching, and explicit temporal speaker supervision is essential for complex cinematic scenes.

Effectiveness of lip contrastive loss LLip. When removing LLip while keeping facial embeddings, LSE-C decreases and LSE-D increases, indicating weaker lip sync. Although facial features in the input visual modality provide coarse lip motion cues, the lack of frame-level contrastive supervision leads to reduced fine-grained alignment accuracy.

Effectiveness of speaker switching concatenation (SSC). We remove the SSC strategy and condition the flow matching on a global reference speaker embedding as in CosyVoice3. Speaker similarity drops slightly in monologue and narration scenes, but degrades significantly in dialogue and multispeaker scenes, where the synthesized speech converges toward an averaged timbre. Nevertheless, subjective evaluation shows that speaker attributes such as gender and age remain distinguishable, likely due to the multi-task supervision in the CosyVoice3 speech tokenizer, which includes speaker age and gender prediction, leading to partial retention of speaker information in the speech tokens.

5 Conclusion

In this work, we propose an end-to-end dataset pipeline and an MLLM-based dubbing model designed for live-action cinematic dubbing. We construct the first television dubbing dataset CineDub with rich annotations. Multimodal CoT Correction improves dataset quality, while the four complementary modalities provide rich information for synthesizing expressive speech. The frame-index codebook and dedicated MLLM supervision enable accurate alignment in complex scenes, and the improved flow-matching design supports flexible speaker switching. In future work, we will explore the possibility of unifying speech, music, and audio generation through the understanding of visual information, detailed script content and captions.

Ethical Statement

All third-party tools and models used in this work are sourced from open-source projects and comply with their licenses. For the data involved in the CineDub dataset, we have obtained the necessary usage rights from third-party rights holders under formal licensing agreements. Consequently, we are only able to provide sample data from the dataset under the CC-BY-NC 4.0 license to ensure appropriate copyright protection. This work is conducted solely for academic research purposes, and all demos are provided only for demonstration purposes and contain no sensitive or infringing information. All human participants involved in subjective evaluations are compensated under employment agreements. The experiments and datasets pose no ethical issues.

Acknowledgments

This work was funded by the National Nature Science Foundation of China under Grant U23B2053.