A3fford-HOI: Anatomy-Aligned Affordance Disentanglement for Fine-grained and Generalizable Hand Object Interaction

1 Introduction

Hand Object Interaction (HOI) generation aims to synthesize realistic and interactive hand-object motion sequences.

In practice, HOI has broad applications in various fields, including Virtual Reality (VR) [Zhang and et al., 2024; Fang et al., 2023], Embodied AI [Wan et al., 2023], and robotics [Wang and et al., 2024]. Conventionally, HOI generation demands not merely visual realism, but also a grounding in physically plausible contact and interaction dynamics.

Recently, instruction-driven methods [Cha and et al., 2024; Huang et al., 2025; Zhang et al., 2025] have provided feasible paradigms for HOI generation by mapping natural language to 3D motion spaces. However, they remain circumscribed by monotonous interaction patterns and limited object categories. Researchers [Christen et al., 2024; Huang et al., 2025] primarily focus on kinematic motion synthesis, ignoring the explicit contact constraints, inevitably resulting in unnatural artifacts and physical implausibility. Only a few approaches [Cha and et al., 2024; Zhang et al., 2025] incorporate coarse contact information to guide generation. Specifically, they leverage interaction cues as guidance by calculating the contact probabilities between hand and object surfaces. However, they overlook the distinct physical roles of hand anatomical structures. Crucially, different hand parts - such as fingertips for precision and palms for stability - must be aligned with specific disentangled affordance regions on the object to guarantee functionally valid manipulation. As illustrated in Figure 2, different colors on the hand denote the contact regions of distinct anatomical structures, visually revealing the specific functional contribution to the interaction. Notably, due to the diversity of manipulation modes, each anatomical part may exhibit high contact probabilities across multiple distinct regions of the object surface. While pursuing fine-grained accuracy, generalization is also a more critical challenge. When confronted with rare object shapes and complex interaction patterns, existing methods [Cha and et al., 2024; Christen et al., 2024; Huang et al., 2025] exhibit significant performance degradation, due to the lack of external physical knowledge and geometric reasoning capabilities. Moreover, current datasets are limited in object diversity and manipulation patterns, which is a main reason for limiting the model generalization.

Thus, the HOI generation task still encounters two main challenges: 1) Building an extensive HOI dataset, including diverse objects and rich motion patterns. 2) Modeling finegrained and generalizable hand-object interaction motion following physically plausible contact and dynamics.

To address the data scarcity, we present HOI-X, an integrated comprehensive HOI dataset. By consolidating several public HOI datasets [Kwon et al., 2021; Taheri et al., 2020; Zhan and et al., 2024], our HOI-X contains various object categories and rich motion patterns. The rigorous pre-processing is then performed to unify data formats across different data sources. Significantly, we conduct part-level contact annotations (i.e., fine-grained quantification of object affordance) by calculating contact probabilities between specific hand anatomical structures and object surfaces, as shown in Figure 1. Along with HOI-X, we propose A3 fford-HOI, a novel two-stage framework that enables fine-grained and generalizable HOI motion generation. We formulate the generation process as a Disentangle-2-Interact paradigm. To decouple the complexities of interaction, in the Disentangle stage, we devise an Anatomy-Aligned Affordance Predictor. Specifically, we prompt 3D MLLMs with textual instructions and object geometries to extract latent physical commonsense-external knowledge that is often scarce in limited HOI datasets. Here, the predictor effectively transforms high-level semantic knowledge into fine-grained, partspecific functional cues. In this manner, the textual instructions and object geometries are adaptively transformed into spatially explicit and disentangled affordance maps. Subsequently, these disentangled affordance priors are seamlessly integrated into the Interact stage to guide the interaction synthesis. In particular, we present the Affordance-Driven Interactor, where textual instructions and object geometric features are fused to provide the global condition for coarse motion generation. Once we obtain initial results, we employ the MLLM-enhanced affordance map as an interactive guidance to refine local hand-object interactions, ensuring high-fidelity contact plausibility. The modulated features are finally fed into the diffusion-based transformer backbone to produce harmonic and physically plausible HOI sequences.

Overall, our contributions are summarized as follows:

• We present HOI-X, a comprehensive hand object interaction dataset with diverse objects, rich motion patterns, and part-level contact annotations for fine-grained interaction.
• We propose A3 fford-HOI, a novel two-stage framework

that enables fine-grained and generalizable hand object interaction generation.

• We design an Anatomy-Aligned Affordance Predictor that injects external physical priors provided by 3D MLLMs for disentangled affordance prediction. Furthermore, we present an Affordance-Driven Interactor, integrating MLLM-enhanced affordance priors to ensure fine-grained and generalizable interaction synthesis.
• Extensive experiments demonstrate that A3 fford-HOI achieves SOTA performances, producing highly realistic and functionally accurate hand-object interaction on finegrained and generalizable scenarios.

2 RelatedWork

Hand Object Interaction Generation: Research in 3D Hand-Object Interaction (HOI) can be broadly categorized into two primary streams: reconstruction and generation. While reconstruction methods [Tzionas and Gall, 2015; Cao et al., 2021; Zhang et al., 2019; Hu et al., 2022; Liu et al., 2025] typically focus on recovering 3D hand-object dynamics from visual observations such as RGB videos or depth maps, the core objective of HOI generation is to synthesize realistic interaction sequences from multi-modal inputs. Early generation approaches [Ghosh et al., 2023; Brahmbhatt et al., 2019; Corona and et al., 2020] predominantly concentrated on synthesizing static grasping postures or simple manipulation primitives. Propelled by the advancements in Diffusion Models and Large Language Models (LLMs), recent frameworks like DiffH2O [Christen et al., 2024] and HOIGPT [Huang et al., 2025] have harnessed textual guidance to facilitate diverse interaction synthesis. Building on this, Text2HOI [Cha and et al., 2024] and OpenHOI [Zhang et al., 2025] explicitly incorporate interaction cues to further refine generation quality. However, existing methodologies largely overlook the distinct functional roles of different hand anatomical structures during interaction. In addition, existing datasets [Fan et al., 2023; Kwon et al., 2021; Taheri et al., 2020; Zhan and et al., 2024] are often limited by repetitive interaction patterns and a scarce variety of object shapes. To address these limitations, we introduce HOIX, a comprehensive dataset enriched with diverse objects and motions, and propose the A3 fford-HOI framework, aiming to achieve fine-grained and generalizable HOI generation.

Affordance Learning: The genesis of affordance learning lies in the image domain, with pioneering works [Do et al., 2018; Zhao et al., 2020] primarily targeting the detection of functional objects. Subsequent advancements [Lu et al., 2022; Mi et al., 2020] integrate linguistic context to refine this process. Nevertheless, these approaches remain constrained to coarse, object-level predictions. To overcome this limitation, contemporary research [Fang et al., 2018; Luo et al., 2022; Nagarajan et al., 2019] shifts towards specific affordance parts, establishing a new norm for precision in the field. With the rise of Embodied AI, affordance learning has rapidly expanded into the 3D domain. 3D AffordanceNet [Deng et al., 2021] constructs the first dataset for affordance detection in 3D point clouds. Building on this, works like LASO [Li et al., 2024] and AffordanceLLM [Chu et al., 2025] establish a bridge between the powerful natural language capabilities of LLMs and 3D object affordances, significantly enhancing static 3D understanding. Further advancements, such as SeqAfford [Wang et al., 2024] and OpenAfford [Wu et al., 2025], refine and expand this capability, realizing compositional affordance reasoning. However, these approaches remain confined to fundamental perception tasks, leaving the application of affordance insights to hand-object interaction generation unexplored. In contrast, our work introduces a "Disentangle-2-Interact" paradigm. Through this framework, we seamlessly integrate disentangled affordance into the HOI task, achieving superior finegrained and generalizable motion synthesis.

3 HOI-XDataset

To alleviate the limitations of object diversity and motion patterns in current datasets, we introduce HOI-X, a comprehensive hand object interaction dataset with Anatomy-Aligned Affordance (A3 fford) annotations, which consists of 5.1K high-quality motion sequences, with a total of 6.2M frames and 134 object templates.

We devise a rigorous data processing pipeline to integrate and canonicalize interaction data from several sources: GRAB [Taheri et al., 2020], H2O [Kwon et al., 2021], and OakInk2 [Zhan and et al., 2024]. The pipeline consists of two key phases. 1) Unified Spatial Projection. To eliminate spatial discrepancies caused by different capture systems, all hand-object interaction sequences are projected into a unified world coordinate system and normalized to a standard scale.

2) Object Representation Standardization. To facilitate efficient geometric learning, we perform uniform sampling on the diverse object meshes, converting them into standardized point cloud models with a fixed resolution of 1,024 points. The specific data statistics and comparisons are shown in Table 1.

4 ProposedMethod

4.1 ProblemFormulation

Given the multi-modal inputs of text descriptions text and object point clouds Po, the goal of our framework is to generate realistic and interactive hand-object motion sequences M. In practice, we leverage the parameterized hand model MANO [Romero and et al., 2022] to represent hand poses. We tackle this challenging problem with the designed framework A3 fford-HOI. The overall objective can be expressed as:

4.2 Disentangle-2-Interact Paradigm

Drawing from classical grasp taxonomy theory [Cutkosky and others, 1989], the anatomical structures of the hand assume distinct functional roles during interaction, governed by dual constraints of task requirements and object geometry. Inspired by these theoretical insights, we propose AnatomyAligned Affordance (A3 fford) to explicitly model disentangled part-specific contact potentials for fine-grained interaction synthesis. To implement this guidance, we introduce a Disentangle-2-Interact paradigm. As shown in Figure 3, we utilize the anatomy disentangled affordance, augmenting with external knowledge from MLLMs to guide the fine-grained and generalizable interactive motion generation.

Disentangle Stage: A3 fford Predictor To harness the advanced reasoning capabilities of large models for part-level contact prior prediction, we build an A3 fford Predictor upon the 3D MLLM to enhance fine-grained and external physical knowledge. The backbone integrates a 3D Vision Encoder (ReCon++ [Qi and et al., 2023]) for geometric object embedding, a pre-trained text encoder (CLIP [Sanghi and et al., 2022]) for text embedding, and a generic Large Language Model (LLaMA [Touvron and et al., 2023]) for affordance prediction. The multi-modal input features include representations of object structure (point features Fpoint) , text prompt (prompt features Fprompt), and motion instruction (instruction features Fins), which are then transformed into hybrid latent tokens H via LLM. Thus, the LLM inference process can be formulated as:

where H is the hybrid latent tokens and can also represent the textual tokens Htext, constrained by the annotated answer text in the following LLM tuning.

Meanwhile, to bridge the gap between textual semantics and geometric interactions in LLM, we expand the LLM

vocabulary with two specialized tokens: <Left/Right Hand Affordance> and <Object Affordance>, as shown in Figure 3 (left), serving as learnable instructors for affordance disentanglement. Thus, the hand mask tokens Hhand and object affordance tokens Hobject are separated from the hybrid latent tokens Htext/H. Finally, Htext, Hhand, Hobject are then fed into the specific Text Decoder, Hand Mask Decoder, and Object Affordance Decoder for subsequent features extraction, respectively.

The textual tokens are decoded into the answer text by Text Decoder (TD), which serves as a semantic constraint to ensure the synthesized interaction affordance aligns with the linguistic instructions.

where Tanswer is the answer text. To acquire the disentangled contact priors, the Hand Mask Decoder (HMD) projects hand mask tokens into active and anatomical hand contact affordance. Simultaneously, the Object Affordance Decoder (OAD) projects object affordance tokens into the geometric object affordance.

where Ahand is the predicted hand affordance, Aobject is the predicted object affordance. We employ the LoRA techniques [Hu and et al., 2022] to efficiently fine-tune the overall large model constrained by the annotated answer text. The specific decoders are trained with the supervision of annotated affordances.

Therefore, the A3 fford Predictor effectively transforms high-level interaction intent into spatially explicit, finegrained and disentangled affordance maps, ensuring strict alignment between anatomical hand structures and their corresponding functional regions on the object. Meanwhile, the introduction of large model provides external knowledge cues for the next Interact stage, improving both the fine-grained fidelity of the interaction and the generalization ability across diverse object categories.

Interact Stage: Affordance Driven Interactor In the Interact stage, we employ the anatomy-aligned affordance from A3 fford Predictor to facilitate high-fidelity and physically plausible hand-object interaction synthesis. The generation process is fundamentally guided by two-level conditions: holistic semantic constraints and the affordance features acquired from Affordance Encoder. The holistic constraints are composed of motion instruction feature, textural prompt feature and object structure feature. These global cues are integrated into our backbone via multi-head attention, which can generate rough motion trajectory and hand pose. Though the above holistic conditions determine the overall motion trend, they can not guarantee precise finger-level contacts without the contact priors. Hence, we introduce the Affordance Control mechanism that employs the aforementioned fine-grained affordance priors to explicitly guide the interaction details.

Affordance Control . As illustrated in Figure 3 (bottom right corner), we utilize the affordance features as the query Q to match the motion embeddings as key Kand value V , through the cross-attention mechanism. Then we obtain the affordance-guided motion features via the AdaLN [Huang and et al., 2017] layers. The modulated motion features are derived as:

where Fmotion donates the interaction motion features and Faff indicates the affordance features. Ultimately, the Motion Decoder translates these features into the final finegrained and physically plausible HOI sequences.

4.3 Objective Function

In the Disentangle stage, to effectively fit the complex distribution of anatomy-aligned affordances, we employ the Channel-Wise Tversky Loss (LTversky) [Salehi and et al., 2017]. Formally, we calculate the weighted overlap between the predicted probability map P and the ground truth G as

Similar to [Jiang and et al., 2021], we optimize the Interact stage using the composition of simple loss Lsimple, contact loss Lcontact, and penetration loss Lpenet to generate the high-fidelity interaction sequences that are both geometrically precise and physically plausible. The overall objective function is expressed as:

where λc, λp are the trade-off weight coefficient.

5 Experiments

5.1 Experimental Settings

Implementation Details: In our experiments, we set the generated motion length to 200 with FPS = 30. We employ a frozen CLIP-ViT-B-32 model as the text encoder. To train the disentangle stage, the batchsize is set to 128, and the epoch number is 100. The interact stage is trained for 600 epochs with a batchsize as 64. The learning rate of the disentangle stage is 1e-3, and weight decay is 2e-4. The interact stage is trained with 5e-5 learning rate within 2e-6 weight decay. All experiments are deployed on an NVIDIA A100 GPU for 24 hours. The trade-off weight coefficients λc, λp are set to 20 and 10, respectively.

Evaluation Metrics: To comprehensively evaluate the quality, diversity, and realism of our generated interaction motions, we utilize a set of quantitative metrics inspired by [Ghosh et al., 2023]. We roughly divide the 6 evaluation indicators into three categories:

• Motion Accuracy: Average Variance Error (AVE) measures the variance error between the joint positions. Fr´echet Inception Distance (FID) quantifies the distributional discrepancy between the synthesized results and the GT.
• Diversity: Diversity and multi-modality (MModality) measure the variability across different prompts.
• Interaction Quality: Contact percentage(C%) calculates the percentage of frames detected with contact. Penetra-

tion quantifies physical implausibility by measuring the extent to which the hand mesh intersects with the object mesh.

5.2 Quantitative Evaluation

Comparing to SOTA Methods: We compare A3 fford-HOI with four state-of-the-art approaches, MDM [Tevet and et al., 2022], HIMO [Lv and et al., 2024], Text2HOI [Cha and et al., 2024], and OpenHOI [Zhang et al., 2025]. These methods are retrained and tested on the HOI-X dataset with the open-source code released by the authors. In particular, to evaluate the generalization capability of our model, we divide the test data into two categories: common objects and rare objects. The details of the dataset splitting are provided in the Supplementary Materials. The results in Table 2 indicate that our model outperforms the competitive methods on all the metrics in common objects, with the most notable lead observed in Interaction Quality. By leveraging anatomy-aligned affordance, our model employs the Affordance Control to seamlessly align interactive motions with fine-grained contact information, successfully enhancing the synthesis quality of HOI motions. Besides, in terms of scarce-category objects as depicted in Table 3, other methods suffer a notable decline in FID, C%, and Penetration scores. In contrast, our approach maintains strong performance in both motion accuracy and fine-grained interaction quality. This is attributed to our disentangle-to-interact paradigm, which enhances generalization by predicting object affordances that are fused with external physical priors.

Ablation Study: To evaluate the Disentangle-2-Iteract paradigm, we conduct an ablation study with different key components. Considering the fair comparison, our ablation study is conducted on the whole testing set, directly. For verifying the effects of the Disentangle stage, we conduct ablation studies on A3 fford Predictor. As shown in Table 4, removing the A3 fford Predictor leads to a severe drop in both contact ratio (C%) and penetration, which reveals that without external physical priors enhanced affordance (i.e.LLM tuning), the model can not generate reasonable hand-object contact. When we employ simple affordance without anatomy-aligned distinction, interaction quality decrease significantly, indicating the anatomy-aligned affordance plays a crucial role in fine-grained control. To demonstrate the effectiveness of the Interact stage, we further conduct an ablation by removing the Affordance Control module. In this impact, the interaction motions display misalignment with the affordance guidance, leading to the inaccurate contact pattern.

5.3 Qualitative Evaluation

To demonstrate the effectiveness of our method, we present the visualized comparison between A3 fford-HOI and other works with the same motion patterns and object templates, as shown in Figure 4. A3 fford-HOI synthesizes superior, natural, and physically plausible HOI sequences. Other approaches are limited in their ability to capture sufficiently detailed contact information, which leads to unnatural interaction motions and severe penetration. Especially, in scenarios with scarce objects, our approach demonstrates stronger generalization capabilities. The key capability lies in generating fine-grained, physically plausible interactions without being confined to specific object shapes.

Additionally, we display various disentangled affordances across diverse objects and motion patterns corresponding to different actions, as shown in Figure 5. The LLM-enhanced external knowledge embedded within the affordance provides a strong prior and guidance for our method to generate finegrained and generalizable interactive motions. Moreover, we conduct a user study to assess the quality of the generated results from the perspective of the subjective metrics. The details are shown in the Supplementary Materials.

6 Conclusion

In this paper, we propose A3 fford-HOI, a fine-grained and generalizable Hand Object Interaction generation framework with Anatomy-Aligned Affordance Disentanglement. Meanwhile, we introduce HOI-X, a comprehensive HOI dataset which enriches object diversity, motion patterns, and finegrained contact information. Specifically, we propose a Disentangle-2-Interact Paradigm, including an A3 fford Predictor to acquire disentangled affordance and the Affordance Control in the interact stage to guide HOI synthesis. Finally, our method achieves SOTA performances with physically plausible, fine-grained and generalizable HOI sequences. In the future, we will focus on interaction tasks in more complex scenarios, such as simultaneously generation for hand and multi-object interaction.

Acknowledgements

This work is supported by NSFC No. 62306309, 62276031 and NKRDPC No. 2023YFF0904700.

Contribution Statement

Xurui Hu conducted the algorithm design, experiments, and paper writing. Xingqun Qi served as the project leader. Muyi Sun served as the corresponding author and provided research guidance. All authors contributed to the discussion and approved the final manuscript.