Egocentric Object-Interaction Anticipation with Retentive and Predictive Learning

1 Introduction

Recent advances in wearable camera technologies facilitate the collection of large-scale egocentric video datasets [Damen et al., 2018; Huang et al., 2024; Grauman et al., 2022], thereby significantly boosting the research in egocentric video analysis. While the main focus of research is on recognition tasks such as action recognition [Wang et al., 2021; Patrick et al., 2021; N´u˜nez-Marcos et al., 2022] and hand grasp analysis [Cai et al., 2016], the anticipation of future human-object interaction is receiving increasing attention [Grauman et al., 2022; Thakur et al., 2024]. This increased focus is largely driven by its potential applications, particularly in VR/AR [Wang et al., 2023; Huang et al., 2018], where, for example, AR assistants can proactively suggest upcoming actions, predict upcoming movements in telesurgery to prepare robotic systems.

The task of anticipating egocentric object interaction aims to predict the future state of human-object interaction without directly observing future videos. This includes forecasting the object's position and category, the form of interaction, and the timing of the interaction. A straightforward solution is to extend the current spatio-temporal localization problem [K¨op¨ukl¨u et al., 2019; Murray et al., 2012] to fit this anticipation task. These methods [Grauman et al., 2022; Ragusa et al., 2023] focus on analyzing the current input to enhance future interaction anticipations. However, these methods often underperform due to three key limitations.

Firstly, there is a misalignment in egocentric encoding. Existing methods rely on pre-training on general video datasets with mixed viewpoints, such as Kinetics [Kay et al., 2017]. This results in encoders that lack domain-specific alignment for egocentric perspectives. In addition, these encoders are typically trained with descriptive labels that emphasize "what is happening", which restricts their ability to encode the predictive context in the video representations.

Secondly, supervision signals are insufficient. In realworld scenarios, future events are probabilistic rather than deterministic. However, datasets often employ one-hot hard labels, assuming a single outcome with absolute certainty. This undermines the effectiveness of supervision and introduces bias, limiting the models' predictive ability in capturing the inherent uncertainty and variability of future events.

Thirdly, there is insufficient utilization of historic information. Existing methods [Ragusa et al., 2023; Mur-Labadia et al., 2024] typically focus solely on short video clips without leveraging the retention of long-term historical data. This oversight can lead to the omission of crucial context that could significantly enhance anticipation accuracy.

To address these limitations, in this work, we propose a Retentive and Predictive Learning framework, named EgoAnticipator, specifically designed for egocentric objectinteraction anticipation. Our framework aims to mitigate the above three issues.. To formulate pertinent memory encoding that is effective in the anticipation context, we first conduct a retentive and predictive pre-training. This step encourages the model to retain important cues of human-object interaction and transfer the cues into a format conducive to anticipation. Following this, we implement a mirror distillation strategy, which utilizes a teacher model with superior future insight, to assist in both the formulation of memory and the execution of the anticipation task. This teacher model is trained on future frames that are mirror-flipped relative to the current timestamp, aiding in memory formulation. Moreover, it introduces a level of predictive uncertainty, which is critical for modeling anticipation tasks. Thirdly, to effectively retrieve pertinent object interaction information from the memory, we craft a long-term memory prompting method. We formulate a prompting feature from long-term history using the object attributes generated by the student or teacher model. Since object attributes are critical in the object interaction anticipation task, long-term memory prompting can effectively enhance the model's anticipatory capability.

We evaluate the performance of our method on the Ego4DSTA benchmark [Grauman et al., 2022]. EgoAnticipator effectively addresses the challenges of domain-specific encoding, probabilistic supervision, and long-term information utilization, resulting in significantly enhanced prediction accuracy. Through comprehensive evaluations, our framework consistently outperforms existing state-of-the-art methods, demonstrating its effectiveness and robustness in anticipating human-object interactions from egocentric perspectives.

2 RelatedWork

2.1 Action and Object-Interaction Anticipation

Prior works [Furnari and Farinella, 2020; Girdhar and Grauman, 2021; Xu et al., 2021; Sener et al., 2020] in action anticipation mainly focus on classifying future actions within a preset time frame. Approaches [Zhao and Kr¨ahenb¨uhl, 2022; Furnari and Farinella, 2020; Xu et al., 2021; Sener et al., 2020] delve into temporal modeling strategies for forecasting forthcoming events at predetermined timestamps. Diverging from the typical temporal action anticipation tasks with fixed intervals, Ego4D introduces a novel short-term objectinteraction anticipation task, requiring predictions about interactions with the next active objects [Pirsiavash and Ramanan, 2012; Furnari et al., 2017] and time to contact them. This task has recently been further explored in studies such as [Ragusa et al., 2023; Thakur et al., 2024; Chen et al., 2022; Mur-Labadia et al., 2024]. In this work, we take a step forward by broadening the scope of information utilized in object interaction anticipation tasks. By incorporating our novel retentive and predictive learning framework, we can effectively establish historical memory and retrieve relevant object interaction from this memory for more accurate and nuanced anticipation about future interactions.

2.2 Knowledge Distillation (KD)

[Hinton et al., 2015] has been established as an effective approach to transfer knowledge from a complex, high-capacity teacher model to a smaller, simpler student model. Its applications span diverse tasks, including image classification [Huang et al., 2022], and dense prediction [Liu et al., 2019]. In the typical implementation of KD [Chen et al., 2017; Hinton et al., 2015], the student model is trained to mimic the soft targets generated by the teacher model. Other methods [Chen et al., 2017; Zhang and Ma, 2020] have focused on minimizing discrepancies between intermediate layer representations of the student and teacher models. KD's versatility extends into the various video domains [Li et al., 2021; Zhao et al., 2020; Mullapudi et al., 2019] or action anticipation tasks [Tran et al., 2021; Fernando and Herath, 2021; Furnari and Farinella, 2023] as well. Our key insight about STA is that the most valuable clues reside in future events. Our proposed mirror distillation utilizes the object frame as a demarcation point to build future teachers and current students. This approach significantly enhances the student's capability of learning anticipation from memory by providing better uncertainty estimates.

2.3 Long-termMemory Learning

The efficacy of context learning in language memory is well-established, as demonstrated by the success of large language models [Brown et al., 2020; Min et al., 2022; Dong et al., 2023]. Similarly, video is also a form of sequential data that can benefit from extended contextual memory, as shown in recent studies [Xu et al., 2021; Wu et al., 2022; Gao and Wang, 2023]. For example, MeMViT [Wu et al., 2022] enhances MViT [Fan et al., 2021] by incorporating long-term multi-scale memory prompts. In our work, we leverage the future teacher in our framework to retroactively update interaction and contact timings of past objects, retaining a refined object-interaction memory that provides more reliable contextual cues for future anticipation.

3 Preliminaries

To illustrate our method clearly, we first define the STA problem and describe the baseline model.

3.1 ProblemDefinition

We define the STA task as proposed in [Grauman et al., 2022]. Given an input video V and a timestamp t, the input to the model is the video from its beginning up to time t (denoted as V1:t). The model is required to anticipate when and where, and what kind of object interaction will happen. The predictions at time t can be represented as Ψ = {φi = (b, n, v, δ, p)}Np i=1 , where b denotes the spatial location of object box, n is the object's category, while v, δ, and p denote the anticipated action, the time to contact (TTC) and the confidence score used to rank predictions for evaluation, respectively.

3.2 Baseline

We use the two-stage baseline from Ego4D [Grauman et al., 2022], which we build upon in the next section. The stages are object detection and anticipation.

Stage1: Object Detection. In the first stage, an object detector such as Faster R-CNN [Girshick, 2015] generates multiple potential next-active objects per frame, resulting in predictions represented as ˜Ψ = {φi = (b, ˜n, ˜ p)}Np i=1 , where b, ˜n and ˜ p are the location, noun category and score of object predicted by object detector. These serve as inputs for the second stage.

Stage2: Short-term Anticipation. In the second stage, a video encoder processes low-resolution clips Vt-l:t as shorttermmemory, covering l history frames and ending at timestamp t. The encoder extracts dynamic features, capturing the evolving actions and interactions. The video features f3d and predictions ˜Ψ are fed into an anticipation network, which pools the spatial-temporal features to a 2D feature map f2d, then uses RoIAlign to sample region features f b 2d for each bounding box b. These features fb aid in forecasting the verb and time to contact. Finally, fb is passed to classifiers to predict the object category n, verb v, and time to contact δ. The loss function combines cross-entropy loss LCE on n and v with smooth L1 loss LL1 on δ, with λ1 = 10:

4 Retentive and Predictive Learning

We introduce EgoAnticipator, a novel retentive and predictive learning framework for short-term object interaction anticipation in egocentric videos. As shown in Figure 1, it enhances the two-stage baseline from Section 3 with three key components: retentive and predictive pre-training, mirror distillation, and long-term memory prompting.

4.1 Retentive and Predictive Pre-training

A direct option to encode short-term memory with short video input Vt-l:t is to use features extracted from a pre-trained video model. However, such models often lack domainspecific alignment for egocentric perspectives and are optimized for descriptive tasks rather than predictive ones. Thus, we craft Retentive and Predictive Pre-training (RPP), a simple yet effective process to optimize the model's ability to retain and anticipate egocentric object interactions, as shown in Figure 2.

Retentive Pretraining. The first part of RPP focuses on training the video encoder ϕr to retain short-term egocentric memory, emphasizing object-centric interactions. We utilize the narration annotations from Ego4D as our training data. Following EgoVLP [Lin et al., 2022], we extract verbs and nouns from each timestamp narration and extend their context boundaries. To ensure the relevance of our data to egocentric interactions, we selectively filter out entries marked with "#o" and "#O", as they relate to exocentric information. The curated dataset, DEgoIR , encompasses 3.2M video clips featuring 115 verbs, 554 nouns, and forms the core dataset for training ϕr. Training ϕr on DEgoIR is approached as a multi-task, multi-label classification task.

Predictive Pretraining. The second stage enhances the encoder's predictive capabilities by training ϕp on DEgoIP , a dataset derived from DEgoIR , using a sliding window of Tw seconds on each video. Clips are selected if a timestamped annotation occurs within the next Tf seconds, resulting in 6.5M clips annotated with future events. In DEgoIP , for each selected video window, future timestamp annotations are divided into Nf grids, each covering a fraction of Tf Nf seconds into the future. We employ Nf future queries and a transformer decoder block to decode future action sequences from each video window. The classification heads used in this stage are identical to those in retentive pre-training, ensuring consistency.

4.2 Mirror Distillation

To enhance the final anticipation model, we propose Mirror Distillation, which leverages a teacher model ϕT to provide (1) a reliable estimate of uncertainty for training ϕS , and (2) a true representation of short-term future features for enhancing ϕS via knowledge distillation. An overview of this step is illustrated in Figure 3.

Future Teacher. One primary limitation of the baseline model (ϕS without our enhancements) is its reliance solely on short-term memory Vt-l:t, limiting its ability to anticipate future interactions. To address this, we introduce a future teacher model ϕT with direct access to future data for guidance. The future teacher ϕT utilizes a mirror-flipping technique applied to video frame sampling around timestamp t, allowing it to access short-term future information Vt:t+l, whereas the student model ϕS only accesses Vt-l:t. As illustrated in Figure 3, the teacher employs a retentive encoder ϕr, while the student uses a predictive encoder ϕp. We guide the learning of ϕS in two ways: (1) task distillation to provide reliable uncertainty estimation and (2) residual feature distillation to align phiS 's features more closely with future features.

Task Distillation. Task distillation aligns ϕT and ϕS by leveraging their shared context, i.e., the center frame at t. Although they process different video segments, they both generate features f T b and f S b for the same objects in the center frame. We minimize the KL divergence between their predicted verb logits yT v and yS v using the following loss:

where Np is the number of objects, τ controls the softening, and σ is the Softmax function. Similarly, we define LnTD for nouns. We also distill knowledge related to time-to-contact (δ) prediction. Given δT and δS predicted by the teacher ϕT and the student ϕS , we minimize the smooth L1 loss LδTD between them by employ a separate MLP in ϕS for the distilled δ prediction. The teacher ϕT often generates higher-quality confidence scores p for each object, so we also distill knowledge from this score. We add an MLP to enable ϕS to learn the confidence score as fused by ϕT . The overall distillation loss for the 4 tasks can be represented as:

where λ2 keeps consistent with λ1.

Feature Residual Distillation. The task distillation transfers knowledge of individual objects from future to past frames, enhancing the student model's instance-level anticipation. To improve anticipation at a holistic level, we introduce a feature-level distillation.

Since ϕS and ϕT differ in their input data, directly enforcing the student's representation to be similar to the teacher's can result in the student model overly focusing on the future. This can cause the student model to overlook critical cues present in the current frames. To circumvent this issue, Feature Residual Distillation employs the residual connection for fusing predicted future features with current ones. Given the 2D feature map f S 2d and f T 2d at timestamp t, generated by ϕS and ϕT respectively, we utilize a lightweight feature decoder ψ within ϕS to predict future feature f ′2d = ψ(f S 2d ). We then minimize the smooth L1 loss between f ′2d and f T 2d , denoted as

Subsequently, we update f S 2d by fusing both features: f S 2d = f ′2d + f S 2d . The newly updated features are then utilized in RoIAlign for the subsequent network.

4.3 Long-termMemory Prompting

While mirror distillation enhances near-term context encoding, it neglects long-term historical information. To this end, we propose Long-term Memory Prompting (depicted in Figure 4), to retrieve a broader range of object interaction information extended further back in time.

Long-term Object Interaction Representation. In the observed video V1:t, we extract key object attributes for interaction anticipation: spatial location b, noun n, verb v, and contact time δ. The detected object-interaction memories are represented as Ψm1:t = {φmi = (b, n, v, p, d)}Np i=1 , where b is obtained by an object detector, and n, v, p are inferred by the memory encoder ϕM . Here, d denotes the temporal distance from the latest timestamp t. Each memory φmi = (b, n, v, δ, d) is described as "At time t - d: after δ seconds, action v occurs on n at position b." We use separate MLPs to embed b, δ, and d, and learnable embeddings for n and v. These embeddings are concatenated for each object φmi to form the representation f m i , which is then input to the encoder of ϕS . There are two options for ϕM to create the long-term memory representation under different perspectives: 1) ϕM = ϕS : Use ϕS to represent the information before the historical target frame; 2) ϕM = ϕT : Use ϕT to represent the information after the historical target frame, as shown in Figure 5. The study of both options is further explained later in the next sections.

Long-term Memory Prompting. We incorporate a specialized lightweight network, denoted as ϕm and illustrated in Figure 4, to integrate the history of object interactions. The lightweight network consists of a single attention layer followed by an MLP, which encodes historical tuples into prompts for fusion with current-frame features. Its primary function is to encode the historical object interaction embeddings f m i into a long-term memory prompt, denoted by f m prompt = ϕm(f m i ). This prompt assists in the final anticipation by fusing f m prompt with each object RoI feature fb on the current frame t (see Sec. 3) through channel-wise concatenation followed by a fully-connected layer. The resulting fused object feature is then used to output the final anticipation result. Notably, during training with ϕM = ϕT , since ϕT requires the use of the l-frame context after f-1 frame, we only utilize objects that satisfy d > l as accessible memory to prevent information leakage.

4.4 Multi-stage Training

The training schedule for EgoAnticipator consists of four steps: 1) Pre-train the video encoder ϕ on DEgoIR to obtain

ϕr. 2) Pre-train ϕr on DEgoIP to acquire ϕp. 3) Use ϕp to train the teacher model ϕT , which observes an l-frame future, on DSTA . 4) Train the student model ϕS with long-term memory prompts and perform mirror distillation with the teacher on DSTA . The final loss for the STA student model combines three components:

Additional details on the model architecture and the training of ϕr, ϕp, and ϕT are provided in the supplemental material.

4.5 Online Inference

The inference phase utilizes the trained student and teacher models. As illustrated in Figure 5, the orange and green regions represent the context ranges l fed into the model. We Table 2: Results of our model and other baseline methods on STAv2. † denotes ϕM = ϕS .

employ the student model ϕS to encode the short-term visual memory (green region) online and to aggregate the longterm interaction history queue (Q = {. . . , f-3, f-2, f-1}) extracted by ϕM . As the visual input stream accumulates, the memory model ϕM continuously forms new interaction memories from the past (orange region). During inference with ϕM = ϕS , we directly append the already encoded short-term memory to the queue without recalculating it. However, when ϕM = ϕT , we must recalculate the new historical memory representation before inserting it into the queue. Additionally, we must ensure that the latest interaction memory f-1 (which may leak future information) is not involved in anticipating the current timestamp. To achieve this, we ensure that the time distance d between f-1 and t is greater than l by only appending f-1 to Q if d > l. This issue does not arise when ϕM = ϕS .

5 Experiments

5.1 Datasets and EvaluationMetric

We evaluate our framework on the STAv1 and STAv2 benchmarks from the Ego4D dataset. These benchmarks comprise 120 hours of annotated clips, including 27,801/98,276 training, 17,217/47,395 validation, and 19,780/19,780 test samples, spanning 87/128 noun and 74/81 verb classes. Evaluation employs Top-K AP/mAP, focusing on Top-5 metrics: Noun (b+n), Noun + Verb (b+n+v), Noun + TTC (b+n+δ), and Noun + Verb + TTC (b + n + v + δ). This measures the model's ability to predict next-active object interactions at varying granularities.

5.2 Implementation Details

For the first-stage object detection, we use Faster R-CNN pretrained on Ego4D. For the second stage, ViT-B serves as the backbone for video feature extraction. During training, only prediction boxes with IoU > 0.5 with the ground truth are utilized. Loss coefficients are balanced as λ1 = λ2 = 10 and τ = 3. The teacher model employs the trained retentive encoder, while the student model uses the predictive encoder. We sample 8 frames with a stride of 8, resulting in l = 64. Implementation details of retentive and predictive pre-training are provided in the supplementary materials.

5.3 Comparison with the State of the Art

We evaluate EgoAnticipator against previous state-of-the-art models using mean Average Precision (mAP) metrics. In the STAv1 benchmark (Table 1), EgoAnticipator outperforms other two-stage methods [Chen et al., 2022; Grauman et al., 2022]. Notably, with a ViT-B backbone, EgoAnticipator exceeds InternVideo [Chen et al., 2022], which utilizes a ViT-L pretrained on Ego4D, by +2.0 mAP. For the STAv2 dataset (Table 2), EgoAnticipator was evaluated by generating detection results in the first stage. Replicating the baseline with our detections and applying EgoAnticipator led to a significant mAP improvement of +2.7 over two-stage methods, nearing the performance of one-stage approaches. Furthermore, integrating our retentive and predictive encoders with the onestage method StillFast [Tan et al., 2023] further enhances performance. These results underscore the effectiveness of our retentive and predictive pre-training strategy. We also compare the performance of EgoAnticipator with ϕM = ϕS and ϕM = ϕT . The findings indicate that the former achieves strong performance, while the latter can achieve further improvements by increasing computation during testing.

5.4 Ablation Studies

A detailed analysis of the EgoAnticipator was conducted on Ego4D-STAv1 using mAP as the main metric. A total of 4 ablation studies were conducted to assess the contributions of various components. More detailed ablation studies can be found in the supplementary material.

1) Retentive and predictive pre-training data. To evaluate RPP's effectiveness, we train multiple STA baseline models with video encoders pre-trained on different datasets, as detailed in Table 3. The results show that encoders trained on DEgoIR outperform those pre-trained on K400 by +23.1%. Training on DEgoIP yields an additional +4.6% improvement. Moreover, encoders trained on the complete Ego4D dataset, which includes approximately 0.6M exocentric action narrations, do not enhance the STA task, suggesting that exocentric information may not be beneficial.

1. Distillation supervision. In Table 4, we carry out diverse experiments to determine the supervision of distillation. Beyond KL divergence supervision for the classification of n and v, we explored time to contact δ, ranking score p, and feature supervision. The results show that as supervision increases, the student's performance continuously improves +7.0%, +4.7%, +0.3%, and +3.4%. These findings highlight the importance of comprehensive distillation supervision in bridging the gap between past and future representations. By learning from the teacher model's superior future insights, the student model aligns its feature distribution more closely with the teacher's, effectively reducing the performance disparity.
2. Impact of long-termmemory. Table 5 shows how different memory types affect model performance. "short-term" is the baseline without long-termmemory, "long-i" is our proposed long-termmemory prompting, and "long-v" is the temporal memory method from LSTR [Xu et al., 2021]. Temporal memory ("long-v") provides a modest improvement of +19.7% by capturing vague visual information. In contrast, long-term memory prompting with interactive memory embedding ("long-i") captures object-centric interactions, including verbs and contact times, resulting in a larger enhancement of +30.8%.

4) Overall ablation. As delineated in Table 6, we offer a comprehensive ablation study for each learning approach proposed within our research. The data, organized from top to bottom, reveals a sequential improvement in model performance. The contribution of Object Embedding (OE) is primarily in terms of mAP. Additionally, we observe that retentive and predictive pre-training (RPP), mirror distillation (MD), and long-term memory prompting (LMP) are instrumental in retaining memory and forecasting future events, thus significantly bolstering the STA task in terms of mAP.

10

0

wood indument vegetable container dough plant bag wire faucet brush wheel food screwdriver paper bottle rod wrench book metal phone cement plate car tablet jug brick knife mold door drawer sponge pan tape cup wall pot spoon ball table mower bowl drill engine shoe bicycle cabinet bucket hammer towel sieve artwork fridge mat playing scissors floor stone glass trash rope computer dumbbell mop broom

Figure 6: Per-noun-category APb+n on STAv1: an improved baseline model with K400 pretrainning (15.52 mAPb+n) vs. its EgoAnticipator

counterpart (19.12 mAPb+n). Categories are sorted by the number of examples.

35

30

25

20

15

10

5

0

Baseline

EgoAnticipator

0.0-0.5 0.5-1.0 1.0-1.5 1.5-2.0 2.0-2.5 2.5-3.0 3.0-3.5 3.5-4.0 4.0-4.5 4.5-5.0 5.0-5.5 5.5-6.0 6.0-6.5 6.5-7.0 7.0-7.5 7.5-8.0

Figure 7: Per-TTC-range APb+δ on STAv1: an improved baseline

model with K400 pretrainning (9.60 APb+δ) vs. its EgoAnticipa-

tor counterpart (15.56 APb+δ). We split the TTC into 16 bins, each

spanning 0.5 seconds. The segmentation is based on the TTC statis-

tics in the validation set.

5.5 Results Analysis

We present quantitative and qualitative results demonstrating

EgoAnticipator's performance on STAv1.

Quantitative results. As shown in Figure 6, EgoAnticipator outperforms the baseline in 56 of 85 categories, notably with small objects like "bottle", "dumbbell", and "phone". TTC anticipation analysis (Figure 7) reveals significant improvements in short-term (0-0.5) and middle-term (1.5-2.5) predictions.

Qualitative results. Figure 8 displays two successful predictions and two failures. The first failure likely stems from sparse annotations, while the second indicates the model's difficulty distinguishing actions with similar motions but different semantics.

Efficiency analysis. Building on the Ego4D baseline, our framework utilizes ViT as the video encoder, offering a simpler and more efficient structure compared to SlowFast, especially with Flash Attention† . Table 7 presents relevant parameters and inference speeds on RTX 4090 GPU. Additionally, our decoder is streamlined with MLPs and a single attention layer, totaling only 13.5M parameters. When ϕM = ϕS , our model can achieve the best efficiency (121 FPS) and leading performance (4.29 mAP on STAv1). When ϕM = ϕS , our model can achieve the best performance (4.43 mAP on STAv1) yet relatively high FPS (59). Overall, EgoAnticipator is an efficient framework.

6 Conclusion

In this paper, we introduce EgoAnticipator, a novel Retentive and Predictive Learning framework for egocentric objectinteraction anticipation. EgoAnticipator improves anticipation capabilities by addressing misalignment in egocentric encoding, insufficient supervision signals, and suboptimal use of historical data. It incorporates long-term object-centric interactions and leverages both current video data and extended historical context for more accurate predictions. Experiments show that the retentive and predictive encoding mechanism, with mirror knowledge distillation, effectively shapes memory and introduces predictive uncertainty. Additionally, longterm prompting efficiently integrates historical information into the anticipatory process. EgoAnticipator unifies memory and predictive learning, achieving significant performance improvements for egocentric interaction anticipation.

AP (%)

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (GrantNo. 62372223, U24A20330), in part by Nanjing University-China Mobile Communications Group Co., Ltd. Joint Institute (No. NR2310J7M), and the Fundamental Research Funds for the Central Universities.

Contribution Statement

Tong Lu served as the corresponding author for communication and correspondence.