Learning to Harvest: VR-Guided Expert Behaviour Capture for Decision Modelling in Agricultural Robots

1 Introduction

The accelerated evolution of computer-related technologies has driven digital transformation across many domains, and agriculture has recently experienced explosive growth in intelligent applications [Singh et al., 2022]. From large-scale farmland remote sensing [Weiss et al., 2020] and greenhouse monitoring [Soheli et al., 2022], to localized tasks such as crop phenotyping [Domingues et al., 2022] and harvesting decision-making [Li et al., 2024b], and even to microscopic levels such as genetic breeding [Farooq et al., 2024], artificial for the future of agriculture. Virtual reality (VR) technologies have also played a significant role in smart agriculture [De Oliveira and Corrˆea, 2020]. Owing to their unique capability for immersive and interactive simulation, VR-based systems have been successfully applied in crop cultivation education [Kim et al., 2018] and agricultural machinery operation training [Cutini et al., 2023].

In the expanding field of smart agriculture, harvesting robots have emerged as a primary application due to their mature infrastructure. While deep learning has significantly advanced crop perception-with YOLO-based models [Junos et al., 2021; Aljaafreh et al., 2023] and keypoint estimation [Li et al., 2024a] identifying optimal picking positions-autonomous decision-making in occluded, highdensity environments remains a critical bottleneck. In particular, algorithm-driven robotic manipulators remain markedly inferior to human experts in terms of posture control and decision-making. This raises a critical question: how can harvesting robots make decisions that more closely resemble those of experienced human pickers? Inspired by this challenge, we explore the integration of motion capture technologies within VR environments to bridge this gap. By constructing high-fidelity virtual spaces, our framework enables the extensive simulation of "hard cases" at scale, providing experts with ample opportunities to perform concentrated demonstrations that markedly accelerate algorithmic learning efficiency. Crucially, the virtual environment allows multiple experts to repetitively harvest identical targets to match optimal trajectories-a process precluded in reality by the destructive nature of harvesting. Moreover, this approach bypasses biological growth and seasonal constraints, permitting continuous experimentation independent of crop maturation. These multi-dimensional data are ultimately leveraged to drive robust robotic decision-making algorithms.

In this work, we focus on tomato harvesting, one of the most widely adopted tasks in agricultural robotics. We first scanned a real tomato greenhouse using a robot-mounted, automatically controlled sliding RGB-D camera. The captured data were reconstructed in Unity [Unity Technologies, 2025] through a series of visual perception and 3D modeling algorithms to approximate the spatial distribution and ripeness of plants and fruits. Next, multiple human experts, equipped with 27-point motion capture suits and VR headsets, performed extensive harvesting operations in the virtual greenhouse. These demonstrations covered a wide range of harvesting difficulties and encoded essential information such as picking order, posture, and speed. Finally, we employed this expert dataset to train a multi-task reinforcement learning model, enabling robots to acquire composite decisionmaking capabilities for tomato harvesting. We validated our approach in both simulation and real robotic environments. Experimental results show that the expert-data-driven decision algorithm consistently outperforms existing methods in terms of harvesting efficiency and collision rates, while also achieving superior success rates and stability.

In summary, the contributions of this paper are threefold:

• We introduce a novel task of VR-based expert motion capture to drive robotic harvesting decision-making.
• We design a system-level integration of algorithms and hardware through a multi-task reinforcement learning framework, enabling robots to accomplish harvesting decision.
• We demonstrate that expert-data-driven decision algorithms outperform existing approaches in terms of harvesting efficiency, stability, and collision reduction.

2 RelatedWork

2.1 Tomato Detection and Harvesting Algorithm

Early tomato harvesting robots primarily relied on traditional computer vision and graphics-based techniques, which performed coarse-grained recognition of tomatoes based on features such as color and simple geometric descriptors [Yin et al., 2009; Kondo et al., 1996]. Although these methods enabled basic fruit detection, their limited precision highlighted the constraints of relying solely on shallow visual cues. Later, machine learning algorithms were introduced for tasks such as ripeness estimation [Qiang et al., 2014], offering incremental improvements over these coarse-grained approaches. With the advent of deep learning, perception in harvesting has made a qualitative leap [Liu et al., 2020; Zheng et al., 2022]. For instance, Jun et al. [2021] integrated 3D perception and reconstruction methods to alleviate occlusion issues during fruit detection.

Despite such advances, most current harvesting robots rely on standardized agronomy, such as defoliation, to maintain operability. In realistic, dense planting scenarios, leaf occlusion complicates the determination of viable harvesting postures. A naive approach of aligning the manipulator directly with the fruit along a straight trajectory (i.e., the normal direction of the plant wall) risks plant damage and leads to repeated failures, severely reducing efficiency. Consequently, while fruit perception is foundational, strategic manipulator posture decision is crucial for enhancing performance.

2.2 Robot Control and Collaboration Algorithm

Early research on harvesting robots primarily relied on analytical and optimization-based approaches. Li [Li et al., 2008] studied inverse kinematics for computing joint configurations from end-effector positions. Liang et al. [2010] proposed an optimization-based motion planner for a sevenDOF manipulator, achieving smooth trajectories and precise end-effector positioning. Boryga et al. [2015] introduced the PR-APT method for efficient trajectory planning in only a few computational steps. However, these analytic approaches [Yin JianJun et al., 2012] often lacked robustness in nostandard condition.

With the rise of reinforcement learning, it has become increasingly applied to planning and control in harvesting robots. Li et al. [2024c] applied RL algorithms to optimize parking positions and task allocation, achieving promising results in both areas, while posture control received comparatively less attention. More recently, Li et al. [2024b] proposed a deep RL-based peduncle grasping pose model, training an improved HER-SAC algorithm with digital tomato plant models reconstructed from greenhouse images and geometric measurements. This enabled dynamic, collisionfree grasping motions in simulation or standard conditions. However, in more complex in-the-wild scenarios, identifying leaves and plants becomes significantly harder, making collision-free harvesting challenging or reducing completion rates due to skipped difficult targets. Additionally, relying solely on simulated data without large-scale expert demonstrations increases training difficulty and may limit performance compared to human-level harvesting. Furthermore, critical factors like speed modulation in human postures are not effectively captured. Motion capture, recognized as one of the most effective ways to accumulate expert demonstration data, has already been widely adopted in diverse application domains [Menolotto et al., 2020; Salisu et al., 2023]. Building on this foundation, our work aims to bring the integration of human-robot interaction and virtual reality simulation into digital agriculture, leveraging expert demonstration data to enhance decision-making in tomato harvesting tasks.

3 Method

3.1 ProblemFormulation

This work develops two independent reinforcement learning (RL) models for a tomato harvesting robot: one for path planning and another for posture decision-making. The input includes the base pose b, spatial positions xi ∈ R3 , maturity levels mi ∈ {0, 1}, and occlusion estimates oi for all tomatoes i ∈ T . The output consists of a harvesting sequence π = (g1, g2, . . . , gK), posture actions ut, and adaptive speed control parameters.

The problem is formulated as two independent reinforcement learning tasks, optimized sequentially. The overall objective is to first optimize path planning and then optimize posture decision-making based on the output of path planning:

The first term corresponds to the Path RL task, where the agent optimizes the harvesting sequence π based on the robot's state spath and action apath, aiming to minimize travel distance and avoid occlusion. The second term corresponds to the Posture RL task, where the agent optimizes posture actions ut based on the state sposture, ensuring efficient and precise harvesting.

In addition to interaction data from a virtual greenhouse environment, the system is augmented with expert demonstration data collected from professional human pickers wearing motion capture suits in reconstructed greenhouse scenarios. These expert trajectories are stored in the replay memory to accelerate training, while adaptive speed control, based on expert data, is directly encoded into the manipulator's control program to ensure safe and efficient harvesting in real-world operations.

3.2 VRGreenhouse Scanning and Reconstruction

Through a sliding-rail RGB-D camera system (Intel RealSense D435) mounted on the tomato harvesting robot, we scanned a real greenhouse with dimensions of 30 meters in length, 30 meters in width, and 2 meters of inter-row spacing. This real-scene scanning serves two purposes: it provides a realistic reconstructed environment for reinforcement learning training, while also enabling expert demonstration through VR-based motion capture to supply valuable replay experiences for the learning process.

The data collection begins with RGB and depth data captured in a real greenhouse. The RGB data plays a crucial role in identifying key characteristics, such as the ripeness of tomatoes, the density of leaves, and the locations of tomatoes, leaf clusters, and stems in a flat view. On the other hand, depth images offer vital insights into determining the 3D positioning and depth of these elements. The segmentation and identification of tomato fruits are performed through the Mask R-CNN [He et al., 2017] algorithm applied to the RGB images. By integrating the segmentation outcomes with the associated depth information, we can precisely calculate the 3D spatial positions of the tomatoes in the environment. Once the 3D models of the tomatoes and plants are reconstructed, they are integrated into the Unity engine to create a virtual reality greenhouse.

3.3 Expert Demonstration viaMotion Capture

Hardware Setup. The VR environment was rendered on a workstation equipped with an NVIDIA RTX 4090 GPU, an Intel i9 processor, and 64 GB of DDR5 memory. For immersive interaction, we used an HTC VIVE PRO 2 headset with a wireless receiver. Expert motions were captured using a 27point inertial motion capture system produced by VIRDYN, which also includes data gloves capable of tracking 10 finger joints. The robotic platform was based on the ELITE EC60, a six-DOF manipulator, outfitted with a custom-designed endeffector specifically tailored for tomato harvesting.

Expert Data Collection. Five expert pickers (height 175-180 cm) were recruited to ensure uniform anthropometric reach and motion consistency. The experiments were conducted in a 10×10 m obstacle-free indoor environment, with participants equipped with an inertial motion capture suit and a VR headset. Following a 3-minute unrecorded warmup phase for environmental adaptation, each participant performed harvesting tasks across 50 virtual scenes categorized into five difficulty levels based on planting density and fruit distribution. Each scene contained 5-20 standard tomatoes. Participants were granted a 10-second observation window to formulate a strategic harvesting trajectory before executing sequential picking. To align with the mechanical configuration of mainstream harvesting robots-which utilize collection tubes for transport-participants were instructed to release each fruit immediately upon detachment. This protocol ensures that the captured demonstrations reflect the operational logic and kinematic constraints of autonomous robotic systems.

Motion Capture Data Processing. The motion-capture (MoCap) output comprises three parts: (i) tomato-picking trajectories (for target sequencing), (ii) postures of the picking arm segment, and (iii) per-posture velocity profiles. Trajectories align naturally with the RL transition tuples and are directly placed into the replay buffer.

Postures require a kinematic conversion because a human arm and a 6-DOF manipulator differ structurally. We define the expert's effective posture segment as the forearm-hand chain (from elbow to fingertip). This segment is mapped to the robot's tool chain (from joint 6 along the cylindrical actuator to the rotary gripper), which matches in structure and effective length by design. Consequently, for a given tomato position, we set the human elbow position as the robot joint-6

origin, and solve a 6-DOF inverse kinematics (achieved by MATLAB Robotics Toolbox [2025]) to recover a full joint configuration consistent with the expert's forearm.

By this way, expert demonstrations captured via VR-based motion capture, as shown in Figure 3, are converted into jointspace trajectories using inverse kinematics and then injected into the replay buffers to accelerate convergence. These demonstrations provide initial guidance for the reinforcement learning process, allowing the policy to start with a better understanding of the task and environment, thus reducing training time and improving efficiency.

Adaptive Speed Control. Analysis of expert MoCap data reveals a distinct two-phase velocity structure: a targettransfer phase (> 30 cm from target) characterized by highspeed movement (∼ 125% baseline) for efficiency, and a target-approach phase (≤ 30 cm) where speed is halved (∼ 50% baseline) to ensure collision avoidance. This 30 cm threshold serves as a heuristic boundary separating coarse transfer from fine manipulation. To prioritize operational safety and stability, this velocity modulation is directly hardcoded into the manipulator's control firmware rather than learned via reinforcement learning.

3.4 Reinforcement Learning Framework

We formulate the harvesting decision-making problem as a two-level reinforcement learning (RL) framework, consisting of (i) Path RL for global sequence planning and (ii) Posture RL for local grasping. The Path module follows the standard DDPG [Lillicrap et al., 2015] Actor-Critic framework, where the policy predicts continuous harvesting priorities for candidate tomatoes. The Posture module extends the same backbone into a single-step continuous posture optimization formulation tailored for local posture selection.

Path RL: Harvesting Sequence Planning. At each parking location, the robot perceives all tomatoes T and selects a feasible harvesting sequence that minimizes travel while avoiding severe occlusion. The harvestable set is

where mi ∈ {0, 1} indicates maturity (1 = harvestable, 0 = unripe), ˜ oi ∈ [0, 1] is the normalized occlusion score (derived from the confidence score of the tomato detection model, with lower confidence reflecting heavier occlusion), and othr is the threshold for severe occlusion. The state at time t is:

where b is the robot base pose, (xee t ,Ree t ) are the end-effector position and orientation, xi is the 3D position of tomato i, pickedi ∈ {0, 1} indicates whether tomato i has been harvested, and

is the normalized Euclidean distance to the target, with Dmax denoting the maximum reachable distance of the workspace.

At each decision step, the actor assigns continuous harvesting priorities to feasible candidate tomatoes, where higher values correspond to stronger harvesting preference. The episode terminates when T ⋆ \ picked = ∅ (i.e., no harvestable and feasible targets remain, including those excluded due to severe occlusion). The per-step reward is

where gt is the selected tomato at step t, α1, α2 > 0 are weighting factors for occlusion and distance penalties (set to 1 by default). A positive terminal bonus Rterm is granted only if all feasible targets are successfully harvested. The overall objective is

where γ ∈ (0, 1] is the discount factor. The policy parameters θpath are optimized using an Actor-Critic framework.

Posture RL: Harvesting Posture Planning. Given a target tomato at position xg ∈ R3 and a fixed tool length Le > 0, the contact-side of the end-effector is assumed to coincide with xg. The policy selects an approach direction u ∈ S2+ on the reachable hemisphere (the plant backside being infeasible). At time t, the state is defined as

where (xee t ,Ree t ) is the current end-effector pose, qt ∈ R6 are joint angles, xg is the target position, and ωg is an occlusion descriptor extracted from the segmentation results of the target tomato. The candidate endpoint is determined by

The end-effector orientation is aligned with the chosen u.

From VR-MoCap data, we construct a template bank of K = 8 expert groups, each corresponding to a distinct occlusion direction and containing the ten fastest entry trajectories demonstrated under that pattern. For a given tomato, the segmentation mask is analyzed: if multiple occlusion regions exist, the region with the highest proportion of green pixels (indicating the most severe leaf coverage) is chosen. The template group on the opposite side of this dominant occlusion is then selected, and its ten expert directions are averaged to form the expert-informed target direction ntmpl.

The one-shot posture reward is

where ∆q(u) is the IK displacement from the current configuration. This encourages alignment with expert templates, penalizes unnecessary joint displacement, and discourages collisions. The weighting factors are set by default to βalign = 2, βjoint = 1, and βcol = 1. Since posture decisions are one-shot per target, the optimization goal is simply

where πθ is the posture policy. The policy thus maximizes expected efficiency and safety at each harvesting step.

Training Procedure. Both Path RL and Posture RL are trained independently in the reconstructed VR greenhouse, with four hidden layers whose neuron counts start at 512 and progressively decrease. The training employs the Adam optimizer with a learning rate of 10-3 and a batch size of 32. During training, the Actor networks generate continuous actions (harvesting priorities or entry directions), while the Critic networks estimate their state-action values. Target networks are softly updated to stabilize training, and mini-batch sampling from the replay buffers ensures stability. After convergence, the two modules are integrated into a unified framework that produces both the harvesting sequence and the collision-free posture decision for each parking location.

4 Experiments

4.1 Simulation Experiments

Experiment Setup. Four performance metrics were evaluated: average harvesting time per fruit, average end-effector travel distance, collision occurrence rate, and harvesting success rate. Three representative methods were benchmarked: (i) a baseline sequential harvesting strategy, which processes tomatoes from left to right and applies a conventional posture controller [Kondo et al., 1996; Qiang et al., 2014] by first positioning the end-effector directly in front of the target (i.e., along the normal direction of the plant wall) and then executing a straight approach; (ii) a geometry-based method, which combines the commonly adopted A* path planner [Hart et al., 1968] for sequencing with the geometry-based posture algorithm widely used in prior tomato harvesting studies [Zeeshan and Aized, 2023; Boryga et al., 2015]; and (iii) our proposed expert-driven reinforcement learning (RL) framework.

Two levels of task complexity were designed: simple and complex, which differ primarily in the degree of occlusion, with complex scenarios involving more severe leaf and branch interference. In addition, both fully ripe and mixed ripeness conditions were tested. For every scenario, the total number of tomatoes was fixed at 100 (excluding unripe ones from the success rate calculation). To account for safety and robustness, each collision between the manipulator and the plant was recorded. If three consecutive collisions occurred when attempting to harvest the same tomato, the attempt was marked as a failure, and the manipulator was reset to its initial pose before proceeding to the next target.

Results and Analysis. Figure 5 summarizes the performance across four evaluation metrics, where our expertdriven RL framework consistently outperformed both baseline and geometry-based methods, with its advantages becoming more pronounced as task difficulty increased. In terms of efficiency, RL achieved the lowest average harvesting time, ranging from 7.1,s (simple-fully ripe) to 8.2,s (complex-mixed); despite increased occlusion, it maintained a ∼20% improvement over the geometry-based method and a > 30% gain over the baseline. This robustness stems from the joint optimization of harvesting sequences and postures, which effectively eliminates redundant motions and failed attempts. Kinematically, RL reduced end-effector travel distance to 28-35,cm, whereas geometry-based and baseline methods required 31-38,cm and 35-42,cm, respectively, as the learned policy exploited direct entry directions to bypass sub-optimal detours in complex scenes. The benefit of expert-informed control was most evident in safety metrics: RL maintained a low collision rate (13-21%) even in dense environments, compared to 17-26% for the geometrybased method and up to 31% for the baseline. Unlike traditional heuristics that often force collisions with occluded targets, our planner strategically skipped low-reward fruits to preserve plant integrity. Consequently, RL delivered a stable success rate of 93.8-98%, outperforming the geometrybased and baseline methods by nearly 4 and over 12 percentage points in the complex-mixed scenario.

4.2 Experiments on a Real Robotic System

Experiment Setup. In this section, we evaluate the expertdriven RL decision-making framework when deployed on a real robotic manipulator. The objective is to validate whether the harvesting trajectories and posture decisions learned in simulation, guided by expert demonstrations, can effectively transfer to a physical platform. The trained policy was integrated into the ELITE EC60 six-DOF manipulator equipped with our custom-designed pipeline-style end-effector.

Unlike the simulation environment, comparing decisionmaking algorithms on real hardware is inherently more challenging, as robot embodiments, end-effector structures, and experimental environments differ significantly across studies. To ensure fairness and comparability, we adopt two widely recognized evaluation metrics in the agricultural robotics literature: (i) average harvesting time per fruit, which directly reflects operational efficiency, and (ii) harvesting success rate, which captures robustness and practical viability. These two metrics are also the most relevant for large-scale deployment in smart agriculture applications. In our experiments, each harvesting view contained 5-20 tomatoes, with a total of 100 mature fruits tested across trials. For the baseline comparisons, the reported results of alternative methods are directly taken from their papers, since their hardware configurations and experimental conditions could not be fully replicated.

Results and Analysis. Table 1 benchmarks our expertdriven RL framework against representative baselines. Our approach achieved the shortest average harvesting time per fruit (7.2,s) with a high success rate (91.0%). Compared to non-expert-driven methods, our framework demonstrated substantial temporal gains: reducing harvesting time by 53.5% versus Novel-EE (15.5,s), 50.7% versus PoseRec (14.6,s), 52.0% versus PID (15.0,s), and 70.0% versus HIS (24.0,s). While Light-FS and GA yielded relatively high success rates (92.9% and 88.0%, respectively), their cycle times-17.4,s and 20.0,s-were over twice as slow as our method. Conversely, the second-fastest method, 3MSP2 (8.6,s), exhibited a significantly lower success rate (70.9%), highlighting a speed-reliability trade-off that our method effectively circumvents. To evaluate overall productivity in large-scale scenarios, we analyzed the throughput metric-successful picks per minute. Our framework achieved 7.58 fruits/min, significantly outperforming PID (3.44), 3MSP2 (4.95), GA (2.64), Novel-EE (3.12), PoseRec (2.96), HIS (2.10), and Light-FS (3.20). Despite inherent hardware and protocol variations across studies, our approach consistently demonstrated superior throughput. This performance is attributed to a dual-synergy: (i) the unified expertinformed policy that optimizes path planning, posture selection, and variable-speed motion, and (ii) an integrated hardware design featuring a scalable soft-tube collection system. By extending the tube beneath the rotating claw immediately upon detachment, the system ensures the fruit rolls smoothly into a cushioned storage box.

4.3 Ablation Study

To quantify the individual contributions of the framework's components, we conducted ablation experiments under the most challenging complex-mixed ripe scenario, comparing the full expert-driven RL framework against five variants:

path-only (expert), posture-only (expert), full RL without expert data, full RL without adaptive speed control, and a sequential baseline. As summarized in Table 2, the full framework achieved the optimal balance across all metrics, yielding an average harvesting time of 8.2,s-a significant improvement over the path-only (9.8,s), posture-only (10.2,s), and no-expert (9.5,s) variants. While the no-speed-control version maintained a comparable travel distance (35,cm) and success rate (93.8%), its harvesting time increased to 8.7,s, underscoring the efficiency gains from adaptive velocity modulation. Furthermore, the integration of expert demonstrations proved critical for operational safety; the full model maintained the lowest collision rate (21%) and the highest success rate (93.8%), notably outperforming the noexpert model (24% collision, 91.3% success) and the baseline (31% collision, 81.3% success). By incorporating rich experience data and fitting the fastest expert postures to provide high-quality priors, the agent effectively emulated professional strategies to resolve complex grasping tasks with fewer attempts.

5 Conclusion

Despite significant advancements in robotic perception, autonomous decision-making remains a bottleneck for harvesting in dense, occluded environments. This paper addresses the limitations of conventional heuristics and explorationlimited reinforcement learning by proposing a VR-based expert motion capture framework. By reconstructing real-world tomato greenhouses in Unity and mapping professional harvesting demonstrations to manipulator kinematics via inverse kinematics, we successfully integrated human expertise into a multi-task RL model for joint path planning and posture control. Comprehensive evaluations demonstrate that this expertdriven approach effectively bridges the gap between machine logic and human skill, achieving stable, efficient, and lowdamage harvesting across diverse scenarios. While currently validated on tomatoes, the proposed paradigm offers a scalable solution for diverse crops.

Acknowledgments

This work was supported by the Basic Research Center, Innovation Program of Chinese Academy of Agricultural Sciences (CAAS-BRC-SAE-2025-01), the Jiangsu Agriculture Science and Technology Innovation Fund (CX(24)1021), and the Central Public-interest Scientific Institution Basal Research Fund(JBYW-AII-2026-21).