TALC:Time-Aligned Captions for
Multi-Scene Text-to-Video Generation

Recent advances in diffusion-based generative modeling have led to the development of text-to-video (T2V) models that can generate high-quality videos conditioned on a text prompt. Most of these T2V models often produce single-scene video clips that depict an entity performing a particular action (e.g., `a red panda climbing a tree'). However, it is pertinent to generate multi-scene videos since they are ubiquitous in the real-world (e.g., `a red panda climbing a tree' followed by `the red panda sleeps on the top of the tree'). To generate multi-scene videos from the pretrained T2V model, we introduce \textbf{T}ime-\textbf{Al}igned \textbf{C}aptions (\name) framework. Specifically, we enhance the text-conditioning mechanism in the T2V architecture to recognize the temporal alignment between the video scenes and scene descriptions. For instance, we condition the visual features of the earlier and later scenes of the generated video with the representations of the first scene description (e.g., `a red panda climbing a tree') and second scene description (e.g., `the red panda sleeps on the top of the tree'), respectively. As a result, we show that the T2V model can generate multi-scene videos that adhere to the multi-scene text descriptions and be visually consistent (e.g., entity and background). Further, we finetune the pretrained T2V model with multi-scene video-text data using the \name framework. We show that the \name-finetuned model outperforms the baseline methods by 15.5 points in the overall score, which averages visual consistency and text adherence using human evaluation. The project website is \href{https://talc-mst2v.github.io/}{https://talc-mst2v.github.io/}.

Figure 2: The architecture of Time-Aligned Captions (TALC). During the generation process of the video, the initial half of the video frames are conditioned on the embeddings of the description of scene 1, while the subsequent video frames are conditioned on the embeddings of the description of scene 2 (ry2 ).

Existing T2V generative models such as Modelscope and Lumiere are trained with large-scale datasets where each instance consists of a video and a human-written video description. These videos either lack the depiction of multiple events, or the video descriptions only focus on the main event in the video. As a result, the pretrained T2V generative models only synthesize single video scenes depicting individual events.

TALC is a novel and effective framework to generate multi-scene videos from diffusion T2V generative models based on the scene descriptions. TALC leverages the widely adopted text conditioning mechanism and modifies it to be aware of the alignment between text descriptions and individual video scenes.

We illustrate the framework in Figure 2. TALC aims to equip the generative model with the ability to depict all the events in the multi-scene descriptions, the visual consistency is ensured by the temporal modules (attentions and convolution blocks) in the denoiser network. TACL can be applied to the pretrained T2V such as Modelscope and Lumiere during inference.

Figure 3: The architecture of Time-Aligned Captions (TALC). During the generation process of the video, the initial half of the video frames are conditioned on the embeddings of the description of scene 1, while the subsequent video frames are conditioned on the embeddings of the description of scene 2.

Our TALC approach improves multi-scene video generation, but struggles with accurate text adherence due to limited training on relevant data. Multi-scene video-text datasets are scarce and challenging to compile because creating detailed captions requires considerable time and resources. Previous projects like ActivityNet have generated captions for specific video scenes, but these scenes often overlap or are spaced far apart, which is problematic for producing smooth transitions in multi-scene videos. Consequently, there is a lack of continuous, high-quality captions needed for effective training of T2V models.

We develop a comprehensive multi-scene video-text dataset that will enhance the training of our existing T2V models. Using the multimodal model Gemini-Pro-Vision, we aim to create synthetic, high-quality video-text data. We start with a basic video-text dataset and use the PySceneDetect library to identify different scenes within a video. We then select a representative frame from each scene to capture its essence. These frames, along with the overall video caption, are fed into a large multimodal model to generate coherent, scene-specific captions. Using this method, illustrated in Figure 3, we generated 20k video scene-captions. The generated dataset consists of 73% multi-scene videos.

Figure 4: Automatic evaluation results for (a) ModelScope and (b) Lumiere. In (a), we observe that TALC-finetuned ModelScope model achieves the highest overall score, that is the average of the visual consistency and text adherence scores. In (b), we find that TALC framework with the Lumiere base model outperforms merging captions and merging videos on the overall scores. We report the average performance across the diverse multi-scene prompts and the number of generated scenes.

We show the efficacy of TALC in multi-scene video generation by utilizing two models: ModelScope and Lumiere. We show two settings, one where TALC is applied to the base model during inference and one where we finetune Modelscope on the curate multi-scene video data with TALC framework.

TALC outperforms the baselines without any finetuning Figure 4(a) and 4(b) show that the base ModelScope and Lumiere with TALC achieves the highest overall scores in multi-scene video generation, outperforming other methods such as merging captions and merging videos. TALC and merging captions excel in visual consistency, whereas merging videos struggle with maintaining consistent visual elements across scenes. TALC also leads in text adherence, demonstrating its effectiveness in aligning closely with scene-specific descriptions, unlike merging videos which, while high in text adherence, fail to integrate descriptions across multiple scenes effectively. This indicates that TALC is particularly effective in producing coherent and textually consistent multi-scene videos.

Finetuning with TALC achieves the best performance. The ModelScope T2V model was finetuned using the TALC framework and merging captions method to improve performance on multi-scene video-text data. The finetuning led to TALC achieving the highest overall score, maintaining strong visual consistency while significantly enhancing text adherence. Conversely, finetuning with merging captions resulted in a notable decrease in visual consistency. This suggests that finetuning on multi-scene data predominantly benefits text adherence, particularly when using the TALC approach.

We show videos generated using merging captions baseline and TALC for various multi-scene descriptions. Overall, we find that the merging captions baseline generates poor quality videos. This highlights that finetuning a T2V model with multi-scene video-text data by naively merging the scene-specific descriptions in the raw text space leads to undesirable artifacts in the generated video.