Bionic bimanual robot teleoperation can transfer the grasping and manipulation skills of human dual hands to the bionic bimanual robots to realize natural and flexible manipulation. The motion capture of dual hands plays an important role in the teleoperation. The motion information of dual hands can be captured through the hand detection, localization, and pose estimation and mapped to the bionic bimanual robots to realize the teleoperation. However, although the motion capture technology has achieved great achievements in recent years, visual dual-hand motion capture is still a great challenge. So, this work proposed a dual-hand detection method and a 3-dimensional (3D) hand pose estimation method based on body and hand biological inspiration to achieve convenient and accurate monocular dual-hand motion capture and bionic bimanual robot teleoperation. First, a dual-hand detection method based on body structure constraints is proposed, which uses a parallel structure to combine hand and body relationship features. Second, a 3D hand pose estimation method with bone-constraint loss from single RGB images is proposed. Then, a bionic bimanual robot teleoperation method is designed by using the proposed hand detection and pose estimation methods. Experiment results on public hand datasets show that the performances of the proposed hand detection and 3D hand pose estimation outperform state-of-the-art methods. Experiment results on a bionic bimanual robot teleoperation platform shows the effectiveness of the proposed teleoperation method.

This work presents a novel RGB-D SLAM approach to simultaneously segment, track and reconstruct the static background and large dynamic rigid objects that can occlude major portions of the camera view. Previous approaches treat dynamic parts of a scene as outliers and are thus limited to a small amount of changes in the scene, or rely on prior information for all objects in the scene to enable robust camera tracking. Here, we propose to treat all dynamic parts as one rigid body and simultaneously segment and track both static and dynamic components. We, therefore, enable simultaneous localisation and reconstruction of both the static background and rigid dynamic components in environments where dynamic objects cause large occlusion. We evaluate our approach on multiple challenging scenes with large dynamic occlusion. The evaluation demonstrates that our approach achieves better motion segmentation, localisation and mapping without requiring prior knowledge of the dynamic object's shape and appearance.

This work presents a novel dense RGB-D SLAM approach for dynamic planar environments that enables simultaneous multi-object tracking, camera localisation and background reconstruction. Previous dynamic SLAM methods either rely on semantic segmentation to directly detect dynamic objects; or assume that dynamic objects occupy a smaller proportion of the camera view than the static background and can, therefore, be removed as outliers. With the aid of camera motion prior, our approach enables dense SLAM when the camera view is largely occluded by multiple dynamic objects. The dynamic planar objects are separated by their different rigid motions and tracked independently. The remaining dynamic non-planar areas are removed as outliers and not mapped into the background. The evaluation demonstrates that our approach outperforms the state-of-the-art methods in terms of localisation, mapping, dynamic segmentation and object tracking. We also demonstrate its robustness to large drift in the camera motion prior.

Visual perception is critical in robotic operations, particularly in collaborative and autonomous robot systems. Through efficient visual systems, robots can acquire and process environmental information in real‐time, recognise objects, assess spatial relationships, and make adaptive decisions. This review aims to provide a comprehensive overview of the latest advancements in the field of vision as applied to robotic perception, focusing primarily on visual applications in the areas of object perception, self‐perception, human–robot collaboration, and multi‐robot collaboration. By summarising the current state of development and analysing the challenges and opportunities that remain in these areas, this paper offers a thorough examination of the integration of visual perception with operational robotics. It further inspires future research and drives the application and development of visual perception across various robotic domains, enabling operational robots to better adapt to complex environments and reliably accomplish tasks.

RGB-D Simultaneous Localization and Mapping (SLAM) in indoor environments is a hot topic in computer vision and robotics communities, and the dynamic environment is a remaining problem. Dynamic environments, which are often caused by dynamic humans in indoor environments, usually lead to the camera pose tracking method failure, feature association error or loop closure failure. In this paper, we propose a robust dense RGB-D SLAM method which efficiently detects humans and fast reconstructs the static backgrounds in the dynamic human environments. By using the deep learning-based human body detection method, we first quickly recognize the human body joints in the current RGB frame, even when the body is occluded. We then apply graph-based segmentation on the 3D point clouds, which separates the detected moving humans from the static environments. Finally, the left static environment is aligned with a state-of-the-art frame-to-model scheme. Experimental results on common RGB-D SLAM benchmark show that the proposed method achieves outstanding performance in dynamic environments. Moreover, it is even comparable to the performance of the related state-of-the-art methods in static environments.