Can Retelling Have Adequate Information for Reasoning? An Enhancement Method for Imperfect Video Understanding with Large Language Model

1 Introduction

Nowadays, Large Language Models (LLMs) have achieved tremendous success in Natural Language Processing (NLP) [Touvron et al., 2023]. Multimodal Large Language Models (MLLMs), which can simultaneously address tasks such as object detection and commonsense reasoning, offer greater potential for development and have garnered even more attention from researchers [Ataallah et al., 2024]. Compared to text and images, videos have more complex and heterogeneous modalities in comparison to the common tasks in Natural Language Processing and Computer Vision. A significant amount of work has been conducted to address video understanding, achieving effective results. However, for video understanding and video question answering, especially reasoning based on LLM-generated video descriptions, is an urgent and pressing issue that needs to be addressed.

Existing MLLMs primarily rely on vision-language generation methods, such as CLIP, and leverage the powerful reasoning capabilities of LLMs to understand visual content [Radford et al., 2021]. For video understanding, existing methods primarily involve extracting frames from videos and then using LLMs to generate contents. To understand the increasingly complex and varied videos, some studies have been proposed. The main methods include BLIP2 [Li et al., 2023c], Instruct-BLIP [Dai et al., 2023], VideoLLaMA [Zhang et al., 2023], Video-LLaVA [Lin et al., 2024], Video-ChatGPT [Maaz et al., 2024], and MiniGPT4Video [Ataallah et al., 2024], etc. These methods have demonstrated powerful understanding capabilities in video understanding, enabling to integrate and response to rapidly changing visual dynamics.

However, in real-life scenarios, there are still many cases of incomplete video descriptions. As shown in Figure 1 a), for instance, a child may be unable to provide a complete description of visual content due to a lack of understanding of the scene. This requires adults to infer and respond based on limited incomplete information. Similarly, for vision-language models, due to limitations in model capability and computational resources, it is often impossible to fully learn and provide a complete description of a video. Furthermore, as shown in Figure 1 b), due to limitations in their knowledge boundaries and reasoning capabilities, MLLMs retelling exhibits certain hallucinations and factual reasoning errors. Although current proprietary LLMs, such as ChatGPT, GPT-4*, and Doubao † , can enhance their knowledge retrieval capabilities through web searches, existing open-source LLMs still lag behind them to some extent. Overall, let us think a question. Can retelling have adequate information for reasoning using MLLMs ?

To address the above issues, this paper proposes ERSR, a novel Entity and Relationship based Self-Enhancing Reasoning framework for imperfect video understanding. Specifically, first, an entity and relationships generation pipeline is proposed, which progressively performs rewriting, augmentation, entity recognition, and scene graph prediction and generation on the obtained video description (or retelling). Furthermore, a novel self-enhanced forward and feedback reasoning strategy is developed. Due to the existence of multi-turn questions for the same video, a forward step-by-step chain is designed, which self-enhances and expands the knowledge and video description through different questions. In addition, a feedback chain is proposed to allow LLMs to reassess the correctness of previous answers based on the expanded description. Pruning is also used to reduce computational costs. Finally, to verify the correctness of the generated content, a rethinking verification strategy is designed to recheck, ultimately achieving convergence. In general, the above reasoning method, enhanced by the reasoning capabilities of Qwen2.5-7B, achieved an overall improvement of more than 1.4%.

For these reasons, the main contributions of this paper are as following 3 aspects:

• Entity and Relationships Generation. Based on the powerful capabilities of large language models in entity extraction and relationship reasoning, an entity and relationships generation pipeline is designed to obtain the

* https://chat.openai.com

† https://www.doubao.com

• enhanced entity and relationships of the video descriptions.
• Self-Enhanced Reasoning. A self-enhanced reasoning method based on a forward step-by-step chain and feedback chain is proposed, which enhances the imperfect video descriptions.
• Rethinking and Better Performance. By using rethinking to reduce hallucinations of LLMs. Extensive experimental results show that the proposed method achieves competitive results in video understanding and question answering datasets.

2 RelatedWorks

2.1 Video Understanding and Question Answering

Video understanding has developed over many years [Soomro, 2012]. Current video understanding methods mainly rely on transformer [Vaswani, 2017] and pretraining [Radford et al., 2021]. Several methods, such as cross-modal attention, motion-appearance memory, and others, have been applied in video understanding and question answering [Jiang et al., 2020; Liu et al., 2021].

With the development of LLMs, video understanding and question answering, particularly for complex tasks, have gradually become dominant due to their exceptional inferential capabilities. Amount of benchmarks have been proposed to evaluate these advances [Xiao et al., 2021; Grauman et al., 2022; Mangalam et al., 2023]. To address the issue of complex reasoning in videos, researchers have conducted extensive works [Li et al., 2023a; Chen et al., 2023b]. Some studies combine subtitles, visual information, and other modalities for video understanding [Wang et al., 2022]. Other works focus on using ChatGPT to pose questions to visual-language models [Wang et al., 2024b; Yang et al., 2024]. Furthermore, some works have proposed question-guided visual description video question answering methods [Mogrovejo and Solorio, 2024]. However, most existing methods rely on proprietary LLMs such as GPT-3.5, GPT-4, etc. Although they have achieved SOTA on realworld datasets, the models are complex and overly dependent on the knowledge boundary of the LLM.

2.2 Large Language Model Based Enhancement Reasoning

Large Language Models exhibit their powerful capabilities through prompt enhancement. Existing research indicates that by applying appropriate prompts and step-by-step reasoning, such as Chain-of-Thought [Kojima et al., 2022]. In recent years, many LLM enhancement methods have emerged for reasoning, such as training and calling external reasoning [Creswell and Shanahan, 2022], program interpreters [Schick et al., 2023], RAG [Lewis et al., 2020; Gao et al., 2023], rethinking [He et al., 2022; Ma et al., 2024; He et al., 2024], and knowledge-based question answering (KBQA) [Wang et al., 2024c]. However, these approaches also introduce hallucinations and cumulative errors during the reasoning process, which limits the model's ability to perform multi-step reasoning.

For video reasoning based on LLMs, Video-of-Thought was proposed, which decomposes the original question into multiple sub-solutions [Fei et al., 2024]. Additionally, VideoAgent applies methods such as object detection and object tracking, and designs a database retrieval approach to reduce the hallucination problem in MLLMs [Fan et al., 2024]. However, the above solutions make full use of video but do not effectively address the incomplete visual detection generated by LLMs. Therefore, our method aims to design a zeroshot solution for incomplete video description reasoning and generation.

3 Methodology

This section will introduce the method ERSR in detail, the illustration and the system prompt are shown in Figure 2.

3.1 Preliminaries

Definition 1 (Video Understanding). It refers to the process of extracting and analyzing semantic information from video. The input is video V = {v1, v2, . . . , vT}, and the output is semantic labels, events, or action inferences Y = {y1, y2, . . . , yM}, where yi represents an event or action class at a specific moment or region in the video. It involves various tasks such as object detection and tracking, action recognition, scene understanding, video captioning, video question answering, et al.

Definition 2 (Scene Graph). A scene graph is defined as a graph G = (V, E,A) with objects or entities in the scene V and the set of edges representing the semantic relationships between the objects E. A contains textual descriptions of the relationship of objects (e.g. "on", "take care of").

Task 1 Video Question Answering (VQA) based Retelling is the task of generating answers Y by reasoning over video description V and natural language question Q = {q1, . . . qL}, which can be expressed as:

where N is the number of video descriptions, L is the number of the question of the same video.

3.2 Step 1: Entity and Relationships Generation

Based on the description incompleteness of LLMs, this section designs a video description enhancement method based on entity and relationship generation. The strategy rewrites and enhances the video description by further extracting entities and scene graphs, transforming the video scene relationships from natural language into Euclidean space, thereby improving the reasoning logic of the LLM.

Incomplete Description Rewriting and Augment

The video description mainly involves entities, scenes, and others from the video. These elements may not fully align with the question due to synonyms, similar relationships, or other factors. Therefore, the description and the question need to be aligned first. The description should be rewritten using the LLM, and the few-shot prompt is as follows:

Task: There may be the video description and the question contains synonyms or ambiguous entity relationships. Please align the description and the question, and rewrite the description.

Assistent prompt: For example, "young boy" and "baby" are synonyms. Give your rewritten description:

After this step, all synonyms and entity relationships will be aligned.

Furthermore, since the description provides an incomplete description of the video, it is necessary to pre-augment the description. The prompt format is as follows:

Task: There is incompleteness in the description. Please expand the description and provide 5 complete descriptions that you consider comprehensive.

Compare the relevance of the 5 descriptions above and select the highest relevance to the original description as the candidate. The equation is (1).

where ci, cj are the i-th and the j-th candidate description. xi, xj are attributes of different generated descriptions, k is an iterator, and D here is the description feature dimension.

Entity Generation

The entity is the most fundamental aspect of natural language. Therefore, entity recognition plays a crucial role in language understanding. In this task, the entity is one of the objectives of the question answering process. So entities need to be extracted from both the description and the question. The prompt is as follows:

Task: Based on the question, identify the entities in the scene, match them with the entities in the question and description.

Assistent prompt: Provide the JSON format. You should only include the entities, and the format is as follows: [See Figure 2].

Furthermore, construct the scene graph based on the extracted entities.

Scene Graph Prediction & Generation

This task focuses on visual reasoning representations in natural language. Scene graphs play an important role in vision, as they help understand high-level semantic relationships in images through objects. Furthermore, providing an accurate scene graph can help avoid hallucinations caused by unclear task instructions in LLMs [Fei et al., 2023]. Therefore, the prompt for constructing a scene graph is as follows:

Task: Provide an inference process based on the scene and description, along with a scene relation diagram, and then extend the scene diagram through reasoning.

Assistent prompt: Provide the JSON format.

In this step, the scene graph is constructed, which will be used for the next step of description reasoning.

3.3 Step 2: Self-Enhanced Forward and Feedback Reasoning

Since the same video contains multiple scenes, multiple questions are posed, each corresponding to different scenes. Through question history and the memory of the LLM, it can learn from the already completed reasoning and further understand and expand the incomplete description. Therefore, this section proposes a self-enhanced question answering reasoning method based on forward and feedback reasoning. By leveraging multiple sets of questions and answers for the same video, the subsequent questions are enhanced. Additionally, a feedback reasoning chain is constructed, which updates the understanding of the video description by reasoning feedback from existing facts.

Multi-QA Forward Step-by-Step Chain

Firstly, in the case where a video has multiple different perspectives, this paper designs a content selection scoring mechanism based on multiple options. For each option, output a score si, which represents the probability of the given question and option matching the description. The scoring prompt is as follows:

Task: Based on the option provided, You need to analyze whether to choose that option. You should give the option a score from 1 to 10, where 1 means would not choose it, and 10 means would definitely choose it.

Assistent prompt: Output the scoring in the following JSON format. [json example]

where qt is the t-th question of the same video, dt is the t-th generated description, m is the LLM selected, and Ot,i is the i-th option of qt. Moreover, St,i is the i-th predict score of qt, i is the number of options, S is the matrix of all answers score of Q.

Due to the severe hallucination problem in smaller language models, it is necessary to specify the output format when calculating scores. Therefore, the score output will be in JSON format.

where qt-1, St-1 is the last question and the last answer scores of the video question answering chain of the same video. Similarly, dt-1 is the last generated description. It needs to note that t > 2.

Furthermore, a step-by-step chain reasoning method is designed to progressively answer multiple questions about the same video while saving previous answers. The description is also enhanced using prompt engineering, with the equation 3 and the prompt as follows:

Task: Rewrite the description. Provide a more detailed description based on [question] and [answer].

Assistant prompt: The rewrite format should be: [new description]

Store the option scores calculated for each question in matrix S. It is important to note that there may be cases where different questions have the same score for the options. The strategy is to accept all these options and perform greedy selection and pruning on the selected options.

where ˆ sk is the highest score of the k-th option across all the questions.

where ˆ yi is the predict option, and\ represents the set difference, meaning all paths except for ˆ sk are pruned.

Multi-QA Feedback Chain

After the forward step-by-step reasoning chain, a feedback reasoning chain is constructed. This involves revisiting previous questions and answers to refine the understanding of the video. The model updates its interpretation based on the existing facts.

Furthermore, recalculate the new scores and re-evaluate the previously Withdrawn candidate options, updating matrix A. Additionally, determine whether each option is predicted correct (Accept) or incorrect (Wrong), perform feedback calculation, and further prune the options. This process will yield the preliminary answers for all questions related to the video description.

3.4 Step 3: Rethinking

During the reasoning process, due to the complexity of the task and the length of the entire reasoning chain, the LLM may generate wrong answers. Therefore, it is necessary to validate the answers provided by the LLM. In this section, Rethinking is used to verify the obtained results. The specific process is as follows:

1. Determine whether the answer has factual hallucinations by matching the chosen results with the question and checking whether the answer aligns with common-sense knowledge.
2. Determine whether the answer has fidelity hallucinations by checking if the chosen result contradicts the description.

Task: Complete the answer verification task. The question is: [question], and the answer is: [answer], with alternative options: [options]. Complete the following two tasks [see above]:

During the verification process, if the LLM analyzes the given result and determines that the verification score is below the threshold, it will return to Step 2 for rethinking. After multiple rounds of rethinking and verification, the judgment will converge, leading to a more confident answer. The maximum round number is often set to 3 or 5.

4 Experiments

4.1 Benchmark Datasets

This paper uses 4 widely adopted datasets: IntentQA [Li et al., 2023b], NExT-QA [Xiao et al., 2021], Egoschema [Mangalam et al., 2023], and ActivityNet-QA [Yu et al., 2019]. Among them, NExT-QA and IntentQA involve multi-turn question answering, where videos are divided into different categories for directed multiple-choice questions. Egoschema and ActivityNet-QA each correspond to a single question per video, with Egoschema being a directed multiple-choice question and ActivityNet-QA being an undirected question.

4.2 Baselines

Numerous works have been proposed to understand video and answer questions. This section categorize them into finetuning methods, zero-shot methods, and MLLM methods.

Fine-tuning methods (FT)

There are several approaches such as the Small Language Models (SLMs) such as HQGA [Xiao et al., 2022a], VGT [Xiao et al., 2022b], CoVGT [Xiao et al., 2023], HiTeA [Ye et al., 2023] and MC-ViT-L [Balaˇzevi´c et al., 2024]. For LLMs, there are BLIP-2 [Li et al., 2023c], LLama-VQA [Ko et al., 2023], Vamos [Wang et al., 2024a] and CaVIR [Li et al., 2023b].

Zero-shot methods (ZS)

There are some proprietary LLMs methods like LLoVi [Zhang et al., 2024], MoReVQA [Min et al., 2024],LVNet [Park et al., 2024], IG-VLM [Kim et al., 2023] and VideoAgent [Wang et al., 2024b]. Additionally, opensource LLMs like VFC [Momeni et al., 2023], SeViLA [Yu et al., 2024] and Mistral [Jiang et al., 2023] are also under consideration.

MLLMmethods (MM)

In recent years, MLLMs for video have gained widespread attention. This paper primarily compares SOTA methods such as MiniGPT-v2 [Chen et al., 2023a], MiniGPT4Video [Ataallah et al., 2024], Video-LLaMA [Zhang et al., 2023], Video-LLaVA, LLaVA-NEXT-Video [Lin et al., 2024], LangRepo [Kahatapitiya et al., 2024], and LLoVi [Ge et al., 2024]. It is important to note that the above methods only use a zero-shot prompt approach and do not employ chain-of-thought.

4.3 Experimental Setup

The experiments use AMD Ryzen Threadripper PRO 5995WX 64-Cores CPU and 4 × A6000, and the maximum GPU memory usage for LLM is 16G. Due to the strong hallucinations in LLMs, all experiments are repeated 5 times and take the average. Furthermore, for the video question answering task, accuracy is used for evaluation. Specifically, for ActivityNetQA, the open-ended answering dataset, the score [Wu et al., 2025] is also used.

4.4 Overall Performance

The experimental results are shown in Table 1. Overall, the proposed method achieves competitive performance on all datasets. The four datasets, due to their different features, lead to distinct application scenarios. In the finetuning methods, both NExT-QA and IntentQA achieve close to 70%. However, with the application of large models, the training costs for multimodal models also increase. For the zero-shot results of proprietary large models, it can be observed that at parameter scales of hundreds of billions (e.g., GPT-3.5) and 1.8T (e.g., GPT-4), most methods achieve optimal performance. However, the API call costs for proprietary large models also increase with their accuracy. This paper primarily explores the application scenarios of opensource LLMs. The experimental results show that the proposed ERSR method achieves competitive results, regardless of whether the verification is applied. The method improves performance by 1.4% to 7.8% on the above two datasets.

Furthermore, through experiments on the verify effect, it can be concluded that performing 3-5 rounds of verification, until the results converge, can effectively reduce hallucinations and factual errors in LLMs.

However, for the Egoschema and ActivityNet-QA datasets, the experimental results are not ideal, and there is still room for improvement compared to methods like LangRepo. The main reason is that for incomplete video descriptions, the content is limited, and the model is unable to extract useful information from such sparse data.

4.5 Ablation Study

To verify the effectiveness and robustness of the proposed method, two ablation experiments were designed in this paper: Different Strategies and Different Backbones.

Different Strategies

Since the verify method has been tested and applied in Section 4.4, as shown in Table 2, this section only examines the effects of Step 1 and Step 2.

• w/o Step 1. Without entity and scene graph extraction, directly using a simple prompt combined with self-entity augmentation.
• w/o Step 2 Using Chain-of-Thought to extract entities and scene graphs from incomplete descriptions, without considering the enhancement of the same video question answering.

Experiments show that both Step 1 and Step 2 enhance the VQA task on both datasets. When Step 1 is removed, performance drops by 2.6% and 0.2% on the NExT-QA and IntentQA datasets, respectively. The main reason is that Step 1 extracts key descriptive information from natural language, reducing LLM's hallucination. As for Step 2, the performance decreases by 3.3% and 1.4%, respectively. This strategy can perform self-enhancement on the previous question history, effectively feeding back valuable information.

Different Backbones

To validate the relationship between the proposed method and the model backbone, experiments were conducted on LLaMA3.1-8B and Qwen2.5-7B. The experimental results are shown in Figure 3.

Overall, both models achieve similar performance across the three datasets. However, Qwen2.5-7B performs better on NExT-QA and IntentQA, while LLaMA3.1-8B shows better performance on Egoschema.

Therefore, it can be concluded that the performance of LLM-based question answering is related to the model's reasoning ability as well as the distribution of the dataset.

4.6 Case Study

Figure 4 shows the application results of the proposed method on a video from the IntentQA dataset. Through the entire pipeline, it can be observed that the method logically infers incomplete descriptions (retelling). By extracting entities, scene graphs, and considering the relationships between different questions, it is ultimately able to complete the question answering task.

5 Conclusion and FutureWork

This paper proposes a novel Entity and Relationship-based Self-Enhanced Reasoning method for imperfect video understanding. Specifically, three modules are designed: a) Entity and Relationships Generation, b) Self-Enhanced Forward and Feedback Reasoning, and c) Rethinking and Verification. Through these three modules, the method enhances the ability of open-source Large Language Models to process incomplete video descriptions. Experiments on four datasets demonstrate the effectiveness of the proposed method.

However, it should be noted that due to the limitations in the reasoning abilities of some open-source LLMs, hallucination issues may arise during generation, leading to incorrect reasoning results. Therefore, the next step is to reduce hallucinations in LLMs, which plays an important role in reasoning with incomplete information.

IntentQA Video ID 2448077748

Contribution Statement

Dr. Xianghua Li∗ is the corresponding author of this paper. Mingxin Li † and Wenhao Wang † contribute equally.

Acknowledgements

This research was supported by the National Natural Science Foundation of China (Nos. 62271411, U22A2098, 62471403, 62261136549), the Technological Innovation Team of Shaanxi Province (No. 2025RS-CXTD-009), the International Cooperation Project of Shaanxi Province (No. 2025GH-YBXM-017), the Fundamental Research Funds for the Central Universities (Nos. G2024WD0151, D5000240309).