Learning to Harvest: VR-Guided Expert Behaviour Capture for Decision Modelling in Agricultural Robots

Learning to Harvest: VR-Guided Expert Behaviour Capture for Decision Modelling in Agricultural Robots

Yining Lang, Zhaoxin Li, Xiujuan Chai

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Main Track. Pages 3630-3638. https://doi.org/10.24963/ijcai.2026/404

While deep learning has significantly enhanced robotic perception in agriculture, autonomous decision-making in dense and occluded environments remains a persistent challenge. This paper proposes a VR-based expert motion capture framework to bridge this gap by integrating high-fidelity virtual environments with human expertise. By reconstructing a realistic tomato greenhouse in Unity and utilizing 27-point motion capture suits, we captured multi-dimensional expert demonstration data encompassing trajectories, postures, and velocities. In the VR environment, ``hard cases" featuring frequent collisions can be selectively generated at a higher frequency, allowing experts to provide concentrated demonstrations of specialized postures that markedly accelerate algorithmic learning efficiency. Crucially, the virtual framework enables multiple experts to repetitively harvest identical targets to match optimal trajectories—a process precluded in reality by the destructive nature of harvesting. Moreover, it bypasses biological growth and seasonal constraints, facilitating continuous, high-frequency experimentation independent of crop maturation. These expert data were subsequently integrated into a multi-task reinforcement learning model as experience replay and reward templates. Experimental results demonstrate that our expert-driven algorithm significantly outperforms baseline methods in harvesting efficiency, collision avoidance, and stability, establishing VR-guided motion capture as a robust and scalable paradigm for advancing agricultural robotics.
Keywords:
Humans and AI: Applications
Humans and AI: Human-AI collaboration
Robotics: Applications