External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding

1 Introduction

Large Language Models (LLMs) excel in knowledge, context, multitasking, and high-quality text generation. Existing video content understanding tasks using LLMs [Chiang et al., 2023; Touvron et al., 2023] primarily rely on large pre-trained models trained on progressively larger datasets. These models are typically trained on vast datasets to acquire extensive knowledge and abilities, enabling them to handle various complex video understanding tasks, such as image or video classification [Li et al., 2024; Ma et al., 2024; Meng et al., 2024; Wang et al., 2024; Meng et al., 2025], video object segmentation [Dang et al., 2024a; Dang and Yang, 2021] and scene parsing [Dai et al., 2024]. Despite strong performance in various tasks and benchmarks, their training data's timeliness limits their ability to process the latest knowledge, especially for new objects and dynamic long video content, resulting in poor performance in recognition and content generation.

∗ Corresponding author.

One of the most significant challenges is that model performance often suffers when dealing with rare and ambiguous objects, as well as knowledge appearing frequently in very long videos. Such rare or uncommon objects or concepts are not adequately represented in traditional datasets, and thus may not be accurately recognized or understood by large models during training. In addition, long videos often contain complex scenes and dynamic backgrounds, which can cause existing models to fail to capture certain details or objects [Dang et al., 2023a]. These issues typically require supplementation from external knowledge bases to help bridge the knowledge gap, significantly enhancing the model's understanding of video content.

Moreover, as new concepts continue to emerge, the task of understanding video content is also constantly evolving. The introduction of new elements and the dynamic nature of video content often lead to exponential increases in the computational costs of existing training models, posing more significant challenges for the training and application of largescale language models [Liu et al., 2024]. As the data volume grows, traditional training methods require substantial computing resources and face the challenge of delayed knowledge updates, making it difficult to keep pace with realworld changes on time. Therefore, continuously updating the model's object information and dynamic knowledge at a reasonable computational cost is crucial for achieving efficient video content understanding.

In this paper, instead of relying on larger datasets or more complex network architectures, we build a scalable external text-based object memory and propose an efficient retrievalaugmented method to update this memory, addressing challenges in long-term video understanding. This external memory incorporates a minimal yet highly effective text-to-visual mapping mechanism that seamlessly integrates static and dynamic multimodal information, enabling LLM-based models to align textual and visual content more accurately and efficiently. Our method can access static object attributes, such as color, shape, and location, as well as dynamic video features including object motion trajectories, scene changes, and temporal coherence from the external memory. By leveraging these capabilities, the model effectively handles the complexity and dynamics of long videos, enhancing both efficiency and accuracy in long-term video understanding.

The key contribution of REVU is the construction of two key-value memories, similar to [Vo et al., 2022; Dang et al., 2023b]. The first key-value pair represents the features and names (semantic labels) of static objects, while the second captures the characteristics and names (semantic labels) of dynamic actions. Unlike the previous approaches [Vo et al., 2022; Dang et al., 2024b], which define dependent objects as keys, our method uses the visual characteristics of objects and actions as keys, taking advantage of the rich availability of object images and video data on the Internet.

In summary, our key contributions are as follows:

• We provide an extensible external object-action mem-
• ory with minimal but useful text-visual mapping, which enables LLMs-based models to align textual and visual information to comprehend the open world.
• We propose a highly effective retrieval-augmented method for long-term video understanding, which can adaptively retrieve world knowledge, i.e., static object information from the external object-text memory and dynamic information from action-text memory.
• Our approach achieves state-of-the-art performance on various downstream video tasks, including long-term video understanding and video question answering. Furthermore, our method demonstrates strong generalization capabilities.

2 RelatedWork

2.1 Multimodal Large Language Models

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the integration of diverse modalities, including natural language processing and computer vision. By combining text, image and audio data, MLLMs can perform more complex tasks, including text and image generation, visual question answering, crossmodal retrieval and multimodal sentiment analysis. These models not only enhance unimodal performance but also advance cross-modal understanding and generation. For example, Contrastive Language-Image Pretraining (CLIP) [Radford et al., 2021] improves zero-shot image classification and cross-modal tasks through contrastive learning on largescale image-text pairs. The DALL-E series [Ramesh et al., 2021] demonstrates the capability to generate high-quality images from textual descriptions, thereby expanding the application scope of generative models. Flamingo [Alayrac et al., 2022] enhances multi-round dialogues and complex scene understanding, enabling more natural human-computer interactions. Recent models such as GPT-4 [Achiam et al., 2023] and PaLM-E [Driess et al., 2023] extend multimodal capabilities by incorporating additional modalities and optimizing architectures, further improving generalization and adaptability. These advancements not only push the boundaries of multimodal understanding and generation but also support practical deployments of intelligent systems.

2.2 Long-TermVideo Understanding

Long-term video understanding aims to capture intricate spatiotemporal dynamics and semantic information over extended durations by analyzing prolonged video sequences. Recent advances in deep learning architectures and computational resources have significantly propelled progress in this field. For example, VideoMAE [Tong et al., 2022] employs the Mask Autoencoder (MAE) framework to perform occlusion reconstruction on videos, thus significantly enhancing video representations and improving the performance of downstream tasks. The Action Graph Transformer (AGT) [Nawhal and Mori, 2021] effectively captures complex action relationships by introducing a graph-structured attention mechanism, thereby enabling accurate recognition and prediction of multi-level behavior sequences. VTimeLLM

[Huang et al., 2023] improves video understanding by improving temporal modeling, effectively capturing events, actions, and contextual changes over time.

2.3 MemoryModels in Vision Tasks

Memory models in vision tasks aim to improve the storage, retrieval, and processing of complex visual information by leveraging memory mechanisms. Computer vision tasks such as image classification, object detection, and video understanding require efficient processing of large-scale visual data. To handle long videos, some works [Dang and Yang, 2022; Dang et al., 2024c] employ 3D Convolutional Neural Networks (CNNs) to capture motion information of objects and persons, enabling the construction of long-term feature banks. However, traditional CNNs struggle to model longterm dependencies in visual data. To address this, researchers have developed memory models that enhance the ability to represent complex spatiotemporal information through external or internal memory mechanisms. DynMemNet [Kumar et al., 2015] efficiently comprehends dynamic scenes in videos by introducing a dynamically updated memory mechanism. It excels in action recognition and event prediction tasks, enabling real-time updates to memory content and adaptation to complex visual environments. SelfMem [Cheng et al., 2023] employs a self-supervised learning method to enhance the representational capacity of its memory module through pre-training on large-scale unlabeled video data. It excels in video classification and video question answering tasks, demonstrating the potential of self-supervised memory models to improve visual task performance.

3 Method

3.1 Overview of REVU

We propose an innovative video content understanding model based on retrieval-augmented LLM. As shown in Fig. 2, this model incorporates an extensible external text-object storage that seamlessly integrates static and dynamic multimodal information through an efficient text-visual mapping mechanism. This mechanism enhances the model's alignment capability, making the matching between visual information and textual descriptions more accurate, particularly in complex multimodal scenarios. To further enhance the model's performance in multimodal information fusion, we design two scalable external memory modules: one for static information and another for dynamic information. These memory modules enable the model to retrieve relevant information from external resources, thereby leveraging this data for reasoning and generating descriptions more accurately, particularly when handling complex scenes. Unlike traditional visual-name pair memory models, our external memory does not rely on a single visual-name pair for each object. Instead, it constructs a diverse memory by collecting visual features from multiple objects and actions. This approach significantly enhances the diversity and flexibility of the memory, allowing the model to maintain efficient comparative tension even when processing richer and more complex video scenes.

To achieve this, our method leverages a frozen, pre-trained visual model and adds several trainable layers of LLMs on top, ensuring low computational costs during training. Specifically, we first align visual features extracted from video content with dynamic and static feature embeddings stored in external memory and then retrieve the corresponding object and action names. These retrieved features provide critical context to guide the LLM in generating more accurate descriptions. Additionally, we introduce an attention fusion module that effectively filters out irrelevant retrievals, automatically focusing on the most relevant object and action features to optimize the generation process. After fusion, the integrated visual and object name features form the final prompt, which is input to the LLM for description generation. This design allows the model to efficiently handle long video understanding tasks while significantly lowering training and inference costs without compromising performance.

3.2 External Multimodel Object-ActionMemory

External Memory Source. To construct the Extensible External Text-Visual Memory, we first collect object-name and action-name pairs from multiple open standard datasets (e.g., LVIS, kinetics700) and gather visual information from the internet. Each collected object-name pair and action-name pair represents the diversity of an object or action in the visual space. Each video clip is processed by a trained visual encoder (e.g., ViT) to generate high-dimensional visual embeddings. These visual embeddings are stored as keys in external memory, with the corresponding object or action names stored as values. This design allows the memory to store not only the visual features of objects or actions but also their associated semantic labels (i.e., names). During inference, visual embeddings are matched with object or action features in external memory to retrieve the corresponding labels.

Key-Value Mapping. For each video, we first use a frozen Visual Encoder to process the video frames sequentially, extracting both spatial and temporal features. After encoding each frame, the generated feature vector captures the spatial information of that frame while also embedding temporal changes, which aids in understanding the dynamic processes in the video. These encoded features are stored as keys in external memory, with the corresponding object/action names for each frame stored as values. Specifically, we constructed a Visual Name Memory that stores both the visual features of each object or action in the video and their corresponding semantic labels (object names or action names). This design allows the memory system to accurately retrieve relevant object or action information, providing robust support for subsequent model inference. To enhance retrieval efficiency, we utilize FAISS [Douze et al., 2024] to build an index and perform fast vector searches by measuring the similarity between visual features.

Dynamic Retrieval. We employ fast and slow query methods to independently retrieve Object Multimodal Memory and Action Multimodal Memory, respectively. Given the significant differences in semantic and dynamic features between objects and actions, their retrieval requirements and optimization strategies are also distinctly different.

In the object multimodal memory, we focus on understanding static objects and scenes. Since objects have stable spatial distributions in video frames, their features primarily involve visual information such as shape, color, and texture. To efficiently retrieve relevant object information, we adopt a combination of fast and slow queries. The multimodal information of objects includes static features and potential detail changes. The fast query method quickly retrieves and calculates similarity, while the slow query refines the results by matching object names to textual information and formatting it according to the model's requirements. This approach enables rapid retrieval of object names and their multimodal features from external memory.

In contrast, the Action Multimodal Memory focuses on dynamic changes and action sequences in videos. Actions are typically composed of consecutive frames, whose features include not only visual information but also temporal dynamics. Therefore, action retrieval is more complex than object retrieval, as it requires considering temporal feature changes. Moreover, not all videos contain obvious actions. To reduce unnecessary computation and avoid introducing erroneous information, we use the fast query method for preliminary action retrieval and similarity calculation. We set a default threshold: if the similarity value exceeds the threshold, the video is considered to contain action features, prompting the slow queries to extract relevant action information from Action Multimodal Memory. If the similarity is below the threshold, the video is assumed to lack action features, and the Action Multimodal Memory is skipped.

Fast and slow queries complement each other to enhance both retrieval efficiency and accuracy. Fast queries quickly locate relevant objects or actions, while slow queries provide precise details and semantic alignment. This decoupled design allows better handling of static and dynamic information in videos, ensuring effective utilization of multimodal object and action features for video understanding. Additionally, the query strategy can be dynamically adjusted based on specific requirements, further improving overall performance.

3.3 Retrieval-Augmented Visual Encoding

Visual Encoder. Previous video content understanding methods typically extracted multimodal features through frozen visual encoders, converting them into a unified representation. A Memory Bank stored historical information to support dynamic queries, while Q-former generated task-specific representations by querying relevant data from the Memory Bank, enabling efficient fusion of multimodal information. However, although the memory bank is highly effective in storing and querying historical data, its capacity is limited and cannot extend beyond the knowledge of the current input. As a result, the model's performance may suffer when dealing with rare, ambiguous, or neglected objects in widely used datasets. Therefore, without an external memory base, the performance of existing models remains constrained.

Retrieval-Augmented Q-Former. The visual encoder generates contextual features from the video, which may lack background knowledge or additional commonsense information about objects and their relationships. For instance, while the visual encoder may identify a person in a scene, it might fail to grasp the nuances of their actions or the broader context, such as their role in a social interaction or activity. This limitation can lead to hallucinations or incorrect interpretations, particularly when encountering rare objects or actions not included in the training data. Leveraging external memory, without needing to increase the size of the core model, could flexibly inject diverse and representative knowledge. By doing this, the model could mitigate the hallucination caused by videos with rarely presented objects or actions, increasing the realibilty of the final output.

To address these limitations, we propose integrating external memory into the model without increasing its core size. By utilizing external memory, we inject diverse, representative information about objects and actions, allowing the model to incorporate knowledge not explicitly present in the video content but essential for understanding the broader context. By utilizing external memory, the model can mitigate hallucinations that arise when videos contain rare or unseen objects and actions. For instance, if an uncommon object appears in the video that the core visual encoder has not encountered, the external memory supplements the encoder's interpretation, providing the necessary context. This enhances the model's ability to understand rare or ambiguous scenes, improving the reliability and robustness of the final output, and ensuring correct interpretation of unfamiliar elements.

Specifically, we use two additional Q-formers that independently process retrieved results Raction and Robject. We employ two additional Q-formers that independently process the retrieved results for objects and actions, denoted as Robject and Raction. These Q-formers are essential for integrating external knowledge with the video's visual features. Given the encoded visual feature Vvid extracted from the video frames by the Visual Encoder, we retrieve the corresponding action and object information from external memory, denoted as:

Each Q-former independently processes the retrieved results using stacked cross-attention and self-attention mechanisms to generate two distinct contextual representations object contextual representation Cobject and action contextual representation Caction as follows:

Each Q-former comprises L stacked layers of crossattention and self-attention mechanisms, allowing the model to refine and integrate retrieved knowledge with visual features. The cross-attention mechanism aligns external knowledge (object and action names) with the video's visual content, while self-attention captures long-range dependencies within the retrieved knowledge, ensuring coherent representations.

After processing by the Q-formers, the two contextual representations Cobject and Caction are fused into a unified representation. This final representation captures both dynamic actions and static objects, providing a comprehensive understanding of the video content.

The final representation Cfinal is passed to the LLM for caption generation or other downstream tasks. This ensures that the output is informed by both the visual features and the externally retrieved knowledge, enabling the model to generate more reliable and accurate interpretations, particularly for rare or complex scenes. By incorporating external knowledge in this structured manner, we overcome the limitations of the Visual Encoder, ensuring that the model remains robust and accurate even with rare or unseen objects and actions.

Dynamic Skipping. The introduction of two additional Qforms increases computational complexity. To balance computational load with performance, we designed a dynamic retrieval module that selects query strategies based on video content characteristics, reducing unnecessary computation.

3.4 Text Decoding

During training, we process video frames autoregressively, with the Q-Former outputting a representation containing all historical information at the final time step, which is then fed into the LLM. This approach reduces the number of input text tags, addressing the context length limitation of the LLM and lowering GPU memory requirements. By compressing the dynamic information of the video, the Q-Former provides a concise representation that helps the LLM better understand video semantics. The model is supervised with a cross-entropy loss function, aiming to generate text descriptions for the video. Only the parameters of the Q-Former are updated, while the visual encoder and language model remain frozen. The LLM leverages the self-attention mechanism to process video features and generate accurate text, effectively addressing the context length limitation and enhancing model performance.

4 Experiments

4.1 Tasks and Datasets

We conducted experiments on long video understanding tasks using three widely used datasets: LVU [Wu and Krahenbuhl, 2021], MSRVTT [Xu et al., 2017], and MSVD [Chen and Dolan, 2011], with top-1 classification accuracy as the primary metric. LVU is a large-scale, multimodal video dataset covering diverse domains with complex scenes, objects, and activities, making it suitable for various video analysis tasks. MSVD contains 120,000 sentences describing over 2,000 video clips, collected via Mechanical Turk. MSRVTT includes 10,000 clips from 20 categories, each annotated with 20 captions.

4.2 Implementation Details

Our framework is built upon InstructBLIP [Dai et al., 2024] using methods outlined in [He et al., 2024]. A 3×3 convolutional layer acts as a projector, and the pre-trained ViT-G/14 [Dosovitskiy, 2020] image encoder from EVA-CLIP [Fang et al., 2023] is employed to create spatiotemporal memory. For fair comparison, we use the same backbone architecture as baseline [He et al., 2024]. The Q-Former weights from InstructBLIP are used for query mechanism, while Vicuna7B [Chiang et al., 2023] serves as the large-scale language model. Training was performed on four NVIDIA 4090 GPUs using cosine learning rate decay. We sample 100 frames per video clip from the LVU dataset at 1 FPS.

4.3 Main Results

Long-Term Video Recognition on the LVU Dataset. As shown in Table 1, we compare our method with state-ofthe-art approaches on the LVU [Wu and Krahenbuhl, 2021] dataset across five categories: Relationship, Director, Year, Scene, and Overall Average (Avg). Our method achieves the highest average score of 68.1, outperforming others in the 'Scene' (80.5) and 'Director' (79.1) categories. It also surpasses most competitors in the 'Relationship' category (60.4) and remains competitive in the 'Year' category (52.6). Our method also outperforms Performer [Choromanski et al., 2020] (52.7) and Orthofomer [Patrick et al., 2021] (53.7). While 'MA-LMM' [He et al., 2024] scores slightly higher in 'Year' (74.6), it falls short in 'Scene' and 'Director', resulting in lower overall performance. These results highlight the effectiveness of our method in leveraging spatial and temporal cues for video understanding, delivering balanced performance across all categories. Its strong performance in the 'Scene' and 'Director' categories underscores its ability to handle complex video content, making it well-suited for large-scale multimodal video analysis.

Video Question Answering on the MSVD Dataset. Table 2 highlights the performance of our proposed method on the Video Question Answering task, specifically on the MSVD [Chen and Dolan, 2011] dataset. The results demonstrate the significant advantages of our method over existing state-of-the-art approaches. Our method achieves an accuracy of 65.2%, significantly outperforming another method and showcasing a substantial performance improvement. In comparison, the second-best method, the baseline MA-LMM, achieves an accuracy of 60.6%, indicating a notable improvement of 4.6% by our approach. Furthermore, compared to VideoCoCa, our method achieves an 8.3% improvement, reflecting its superior cross-modal understanding and robust question answering capabilities. The figure also includes results from several prominent approaches introduced in recent years, ranging from JustAsk (2021) to the latest Valley (2023). Our method not only leads to accuracy but also demonstrates stronger robustness and generalization in longterm video question answering tasks.

Video Question Answering on the MSRVTT Dataset. Table 2 illustrates the performance of our proposed method on the video question answering task, specifically evaluated on the MSRVTT dataset, and compares it with various stateof-the-art methods. The results clearly demonstrate the significant accuracy advantage of our approach over existing methods. Our method achieves a top accuracy of 52.1% on the MSRVTT dataset, outperforming all current methods and showcasing strong performance improvements. The secondbest method, Valley, achieves an accuracy of 51.1%, which is 1.0% lower than our approach. Compared to MA-LMM, which achieves 48.5%, our method improves by a notable 3.6%, further highlighting its superior capability in video understanding tasks. These results establish our approach as a new benchmark for video question answering, reflecting its robust and effective design.

Generalization. Table 3 evaluates the generalization capability of our method on the LVU dataset's Relationship task with varying train-validation splits. With a 50%-50% split, our method outperforms the baseline, demonstrating strong generalization. For the 70%-30% split, it achieves Top-1 accuracy of 58.87%, showcasing its ability to effectively utilize training data while maintaining robust performance. The results confirm the adaptability of our method to different data distributions and highlight its generalization effectiveness.

4.4 Visualization

Figure 3 shows the experimental results of our method on six long-term video recognition tasks on the LUV dataset, each representing varying levels of video comprehension. The tasks span recognizing relationships between people and identifying actions, with each experiment using real movie scenes. These scenes were carefully chosen to maintain high semantic relevance and exhibit diversity in visual style, lighting and frame rate. The figure shows input videos for different tasks and the corresponding model outputs. In the first task, the model successfully identifies the relationship between two people (husband and wife), while in the second task, it accurately recognizes actions (e.g., 'chopping meat' and 'cooking'). The results demonstrate that our method excels across various video recognition tasks, showcasing its ability to handle diverse and complex visual inputs.

In Figure 4, we conduct an ablation study on the LVU dataset to verify the impact of external multimodal memory components including Object Memory (OM), Action Memory (AM), and Dynamic Retrieval (DR) on the baseline model's performance in the Relationship and Director tasks. Adding OM improves task accuracy, but combining OM and AM leads to performance loss without Dynamic Retrieval. Integrating all three components achieves the best performance, with accuracies of 61.8% (Relationship) and 79.1%

(Director), highlighting the role of DR in complementing OM and AM for better long-term video understanding.

Figure 5 presents an ablation study evaluating the effect of threshold values and the number of frames (NF) in the action-text memory bank on Top-1 accuracy. Results indicate that lower thresholds consistently improve Top-1 accuracy over higher thresholds at a fixed large NF. For instance, with NF=400, reducing the threshold from 0.9 to 0.7 increases Top-1 accuracy from 61.6% to 61.8%. Increasing NF from 200 to 400 improves performance across all thresholds, highlighting the importance of a larger memory bank for better action-text alignment. These findings underscore the critical role of dynamic retrieval strategies and memory size in optimizing model performance for long video understanding tasks.

5 Conclusion

In this work, we introduce an efficient retrieval-enhanced method for long-term video understanding, addressing challenges of dynamic object evolution and the emergence of new objects in open-world scenarios. By leveraging a lightweight and easily trainable model, our approach integrates static object information from external memory with dynamic contextual cues from videos, enhancing LLMs with world knowledge. Through seamless alignment of retrieved textual data with static and dynamic features, our method achieves robust generalization across domains without requiring additional fine-tuning or retraining.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61976231, Grant 61972435, Grant U20A20185, and Grant U1611461; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515012853, Grant 2022B1515020103, and Grant 2019A1515011869; and in part by the Shenzhen Science and Technology Program under Grant RCYX20200714114641140.

Contribution Statement

Bimei Wang and Huicheng Zheng contributed equally to the correspondence.