Accurate recognition of human pose and prediction of human motion intention are essential for exoskeleton robots to provide effective assistance. In recent years, inertial measurement units (IMUs) have been widely used to estimate human pose due to their stability and economy. However, physical sensors are inherently subject to time delay relative to human movement, which makes it challenging to meet the demand for real-time control of power-assisted exoskeletons. To address this issue, we propose a data-driven approach called motion forecasting network (MoFCNet) that leverages historical IMU readings to forecast the 3D pose of human in future frames. Our approach consists of two stages, state variables forecasting and pose regression. In stage I, we forecast the IMU data, joint positions and velocities. In stage II, we regress the future human poses from the forecasting results of stage I. Besides, we design a basic stack that can be applied to both stages. The mean joint rotation error on DIP-IMU and TotalCapture datasets are $\bm {10.34}$ and $\bm {16.11}$ degrees, respectively. When focusing solely on the first frame of the prediction results, these errors decrease to 9.13 and 14.50 degrees. Experimental results indicate that our method not only outperforms the forecasting baseline methods but also achieves comparable performance to the sparse IMU-based human pose estimation algorithm.
2 August 2023 in IEEE Robotics and Automation Letters
We introduce a new task, called Class Incremental Robotic Pick-and-Place (CIRPAP), which calls for the capacity to learn to pick and place new categories of objects while retaining the skill of dealing with the previously learned ones. CIRPAP faces three challenges: catastrophic forgetting, few-shot learning, and robust picking in cluttered environments. To address the challenges of catastrophic forgetting and few-shot learning, we propose a novel CIRPAP framework that is built on Incremental Few-Shot Object Detection (iFSD). Specifically, with fixed pre-trained Transformer-like object detection models, we only fine-tune the additional adapter modules, which is called adapter-tuning. To address the challenge of robust picking in cluttered environments, we also utilize multiview fusion to integrate object detection and grasp prediction results. As for iFSD evaluation, experiments show that our adapter-tuning-based approach outperforms state-of-the-art methods on COCO and our dataset. As for full CIRPAP system evaluation, experimental results on a real robotic platform demonstrate the effectiveness of our proposed framework.
This study addresses the challenge of efficiently peeling pineapples, which have a distinct elliptical form, thick skin, and small eyes that are difficult to detect with conventional automated methods. This results in significant flesh waste. To improve the process, we developed an integrated system combining an enhanced BlendMask method, termed SAAF-BlendMask, and a Pose Correction Planning (PCP) method. SAAF-BlendMask improves the detection of small pineapple eyes, while PCP ensures accurate posture adjustment for precise path planning. The system uses 3D vision and deep learning technologies, achieving an average precision (AP) of 73.04% and a small object precision (APs) of 62.54% in eye detection, with a path planning success rate reaching 99%. The fully automated electromechanical system was tested on 110 real pineapples, demonstrating a reduction in flesh waste by 11.7% compared to traditional methods. This study highlights the potential of advanced machine vision and robotics in enhancing the efficiency and precision of food processing.
3D instance segmentation is a fundamental task in sensor data processing for robot manipulation. However, recent works in this field did not pay much attention to 3D instance segmentation under cluttered conditions. Specifically, current widely-used 3D instance segmentation datasets mainly focus on indoor scenes, and they often reconstruct the whole 3D scene by multiple depth sensors. Therefore, there are no occluded or cluttered areas in these datasets. Moreover, we explore prevailing 3D instance segmentation models and find that the traditional heuristic algorithms in these models severely harm their performance in cluttered scenes. To tackle these issues, we first build a monocular depth sensor system to collect a new dataset named Cluttered Objects (CO), which is tailored for robotic manipulation under cluttered conditions. Then, we propose Instance-Guided Net (IGN) to segment targets in these complex areas. In our IGN, the point-wise instance information is fully exploited to make all heuristic algorithms task-oriented. The keys to IGN are Dual-Instance-Guided Block (D-IGB), Instance-Guided Upsampler (IGUS), and Instance-Guided Downsampler (IGDS). In D-IGB, we build the instance-aware receptive field and dilated receptive field to gather intra-instance information and long-range context simultaneously. In IGUS, the instance-aware receptive field is applied to realize task-oriented interpolation. Moreover, we propose IGDS to retain more beneficial foreground features by eliminating background points. Extensive experiments on CO and other public datasets show the effectiveness of our IGN.
Class incremental learning(CIL) has attracted much attention, but most existing related works focus on fine-tuning the entire representation model, which inevitably results in much catastrophic forgetting. In the contrast, with a semantic-rich pre-trained representation model, parameter-additional-tuning (PAT) only changes very few parameters to learn new visual concepts. Recent studies have proved that PAT-based CIL can naturally avoid fighting against forgetting by replaying or distilling like most of the existing methods. However, we find that PAT-based CIL still faces serious semantic drift, the high-level forgetting problem caused by classifier learning bias at different learning phases, which significantly reduces the performance of PAT-based CIL. To address this problem, we propose Incremental Prototype Tuning (IPT), a simple but effective method that tunes category prototypes for classification and learning example prototypes to compensate for semantic drift. Extensive experiments demonstrate that our method can effectively compensate for semantic drift. Combined with well-pre-trained Vit backbones and other PAT methods, IPT surpasses the state-of-the-art baselines on mainstream incremental learning benchmarks.
The study addresses the challenge of efficiently peeling pineapples, which have a distinct elliptical form, thick skin, and small eyes that are difficult to detect with conventional automated methods. This results in significant flesh waste. To improve the process, the researchers developed an integrated system combining an enhanced BlendMask method, termed SAAF-BlendMask, and a PCP method. SAAF-BlendMask improves the detection of small pineapple eyes, while PCP ensures accurate posture adjustment for precise path planning. The system uses 3D vision and deep learning technologies, achieving an average precision (AP) of 73.04% and a small object precision (APs) of 62.54% in eye detection, with a path planning success rate reaching 99%. The fully automated electromechanical system was tested on 110 real pineapples, demonstrating a reduction in flesh waste by 11.7% compared to traditional methods. This study highlights the potential of advanced machine vision and robotics in enhancing the efficiency and precision of food processing.