Precisely imitating human motions in real-time poses a challenge for the robots due to difference in their physical structures. This paper proposes a human–computer interaction method for remotely manipulating life-size humanoid robots with a new metrics for evaluating motion similarity. First, we establish a motion capture system to acquire the operator’s motion data and retarget it to the standard bone model. Secondly, we develop a fast mapping algorithm, by mapping the BVH (BioVision Hierarchy) data collected by the motion capture system to each joint motion angle of the robot to realize the imitated motion control of the humanoid robot. Thirdly, a DTW (Dynamic Time Warping)-based trajectory evaluation method is proposed to quantitatively evaluate the difference between robot trajectory and human motion, and meanwhile, visualization terminals render it more convenient to make comparisons between two different but simultaneous motion systems. We design a complex gesture simulation experiment to verify the feasibility and real-time performance of the control method. The proposed human-in-the-loop imitation control method addresses a prominent non-isostructural retargeting problem between human and robot, enhances robot interaction capability in a more natural way, and improves robot adaptability to uncertain and dynamic environments.

A large amount of data acquisition and annotation work is required to train a supervised machine learning model for open scenes. However, traditional manual approaches are inefficient. Here, a method is proposed for on-site image acquisition and semiautomatic annotation based on eye-tracking. This method uses the recognition capabilities and computational advantages of humans and machines to improve annotation efficiency, overcoming the bottleneck of AI-based approaches to the natural scenery understanding of field robots. The proposed method contains three advancements. First, we designed a head-mounted display with a built-in pose measurement module to achieve first-person teleoperation data acquisition, where a pseudoframe interpolation algorithm is designed to overcome the latency problem in immersive remote data transmission and to achieve efficient field data acquisition. Second, we propose an adaptive superpixel segmentation algorithm to reduce human–machine interactions based on eye-tracking. Third, since traditionally, the annotation process cannot provide feedback to the acquisition process and results in a low conversion rate. We proposed a new conversion rate index denoting the rate of transforming collected data into valid data to quantify the acquisition quality in real time. While achieving an annotation quality of 0.964 per the Dice index, which is approximately equal to that of the manual method, the proposed method improves the annotation efficiency by more than 3 times. Finally, the agricultural field experiments containing a real-life scene of robotic tomato-picking verified that the proposed method based on human–computer interaction can make full use of human perception and recognition intelligence.

Field phenotyping is a crucial process in crop breeding, and traditional manual phenotyping is labor-intensive and time-consuming. Therefore, many automatic high-throughput phenotyping platforms (HTPPs) have been studied. However, existing automatic phenotyping methods encounter occlusion problems in fields. This paper presents a new in-field interactive cognition phenotyping paradigm. An active interactive cognition method is proposed to remove occlusion and overlap for better detectable quasi-structured environment construction with a field phenotyping robot. First, a humanoid robot equipped with image acquiring sensory devices is designed to contain an intuitive remote control for field phenotyping manipulations. Second, a bio-inspired solution is introduced to allow the phenotyping robot to mimic the manual phenotyping operations. In this way, automatic high-throughput phenotyping of the full growth period is realized and a large volume of tiller counting data is availed. Third, an attentional residual network (AtResNet) is proposed for rice tiller number recognition. The in-field experiment shows that the proposed method achieves approximately 95% recognition accuracy with the interactive cognition phenotyping platform. This paper opens new possibilities to solve the common technical problems of occlusion and observation pose in field phenotyping.

The working environment of fruit-picking robots is highly complex, involving numerous obstacles such as branches. Sampling-based algorithms like Rapidly Exploring Random Trees (RRTs) are faster but suffer from low success rates and poor path quality. Deep reinforcement learning (DRL) has excelled in high-degree-of-freedom (DOF) robot path planning, but typically requires substantial computational resources and long training cycles, which limits its applicability in resource-constrained and large-scale agricultural deployments. However, picking robot agents trained by DRL underperform because of the complexity and dynamics of the picking scenes. We propose a real-time policy reconstruction method based on experience retrieval to augment an agent trained by DRL. The key idea is to optimize the agent’s policy during inference rather than retraining, thereby reducing training cost, energy consumption, and data requirements, which are critical factors for sustainable agricultural robotics. We first use Soft Actor–Critic (SAC) to train the agent with simple picking tasks and less episodes. When faced with complex picking tasks, instead of retraining the agent, we reconstruct its policy by retrieving experience from similar tasks and revising action in real time, which is implemented specifically by real-time action evaluation and rejection sampling. Overall, the agent evolves into an augment agent through policy reconstruction, enabling it to perform much better in complex tasks with narrow passages and dense obstacles than the original agent. We test our method both in simulation and in the real world. Results show that the augment agent outperforms the original agent and sampling-based algorithms such as BIT* and AIT* in terms of success rate (+133.3%) and path quality (+60.4%), demonstrating its potential to support reliable, scalable, and sustainable fruit-picking automation.

Robotic harvesters have the potential to substantially reduce the physical workload of agricultural laborers. However, in complex agricultural environments, traditional single-arm robot path planning methods often struggle to accomplish fruit harvesting tasks due to the presence of collision avoidance requirements and orientation constraints during grasping. In this work, we design a dual-arm tomato harvesting robot and propose a reinforcement learning-based cooperative control algorithm tailored to the dual-arm system. First, a deep learning-based semantic segmentation network is employed to extract the spatial locations of tomatoes and branches from sensory data. Building upon this perception module, we develop a reinforcement learning-based cooperative path planning approach to address inter-arm collision avoidance and end-effector orientation constraints during the harvesting process. Furthermore, a task-driven policy network architecture is introduced to decouple the complex harvesting task into structured subproblems, thereby enabling more efficient learning and improved performance. Simulation and experimental results demonstrate that the proposed method can generate collision-free harvesting trajectories that satisfy dual-arm orientation constraints, significantly improving the tomato harvesting success rate.