Mobile robot autonomous navigation in large-scale environments with crowded dynamic objects and static obstacles is still an essential yet challenging task. Recent works have demonstrated the potential of using deep reinforcement learning to enable autonomous navigation in crowds. However, only considering the human-robot interactions results in short-sighted and unsafe behaviors, and they typically use hand-crafted features and assume the global observation range, leading to large performance declines in large-scale crowded environments. Recent advances have shown the power of graph neural networks to learn local interactions among surrounding objects. In this paper, we consider autonomous navigation task in large-scale environments with crowded static and dynamic objects (such as humans). Particularly, local interactions among dynamic objects are learned for better-understanding their moving tendency and relational graph learning is introduced for aggregating both the object-object interactions and object-robot interactions. In addition, local observations are transformed into graphical inputs to achieve the scalability to various number of surrounding dynamic objects and various static obstacle patterns, and the globally guided reinforcement learning strategy is introduced to achieve the fixed-sized learning model even in large-scale complex environments. Simulation results validate our generalizability to various environments and advanced performance compared with existing works in large-scale crowded environments. In particular, our method with only local observations performs better than the benchmarks with global complete observability. Finally, physical robotic experiments demonstrate our effectiveness and practical applicability in real scenarios.

Visual servoing and vibration suppression of spatial flexible-link manipulators with a fixed camera setup are addressed in this article. The singular perturbation method is adopted to decouple the dynamic equations of the flexible manipulator; hence, two subsystems that represent the rigid robot motion and flexible-link vibration are obtained, respectively. Then, for the slow subsystem related to the rigid motion, an image-based controller is designed to converge the image errors with the consideration of compensating for the errors of approximating the Jacobian matrix. For the fast subsystem corresponding to the elastic vibration, to eliminate the requirements of measuring the vibration states, an observer is designed to estimate the fast states and then a feedback controller of the fast subsystem is presented to suppress the vibration of the flexible manipulator by using the estimation values. The closed-loop stabilities of the slow and fast subsystem are both proved by employing the Lyapunov theory. Numerical simulations demonstrate the effectiveness of the proposed controller, which shows that the image errors approach zero with the vibration of the flexible manipulator damped out simultaneously.

3-D scene flow characterizes how the points at the current time flow to the next time in the 3-D Euclidean space, which possesses the capacity to infer autonomously the nonrigid motion of all objects in the scene. The previous methods for estimating scene flow from images have limitations, which split the holistic nature of 3-D scene flow by estimating optical flow and disparity separately. Learning 3-D scene flow from point clouds also faces the difficulties of the gap between synthesized and real data and the sparsity of LiDAR point clouds. In this article, the generated dense depth map is utilized to obtain explicit 3-D coordinates, which achieves direct learning of 3-D scene flow from 2-D images. The stability of the predicted scene flow is improved by introducing the dense nature of 2-D pixels into the 3-D space. Outliers in the generated 3-D point cloud are removed by statistical methods to weaken the impact of noisy points on the 3-D scene flow estimation task. Disparity consistency loss is proposed to achieve more effective unsupervised learning of 3-D scene flow. The proposed method of self-supervised learning of 3-D scene flow on real-world images is compared with a variety of methods for learning on the synthesized dataset and learning on LiDAR point clouds. The comparisons of multiple scene flow metrics are shown to demonstrate the effectiveness and superiority of introducing pseudo-LiDAR point cloud to scene flow estimation.

In this article, to solve trajectory tracing problem and vibration suppression for a planar two-link rigid-flexible manipulator subject to joint-velocity measurement noise, a novel uncalibrated visual servoing control is proposed. To begin with, the manipulator’s dynamic model is established by the assumed mode method (AMM). On this basis, based on the singular perturbation theory, two subsystem controllers are designed, one is slow subsystem controller, and the other one is fast subsystem controller. In the slow subsystem, to cope with the complication of the camera calibration, an adaptive algorithm is formulated to evaluate the parameters of a fixed camera online. Aiming to overcome the challenge that exact joint-velocity measurement may be disturbed by external noise, a nonlinear sliding observer is developed to estimate the state of joint velocity accurately. The asymptotic convergence of image tracking error is proved by means of Lyapunov analysis. Additionally, for the purpose of restraining the flexible beam’s elastic vibration, a linear quadratic regulator (LQR) approach is adopted in the fast subsystem control design. The realistic comparing simulation experiments are presented to demonstrate the performance of the proposed controller.

As one of the important tasks in the field of robotics and machine vision, visual odometry provides tremendous help for various applications such as navigation, location, etc. Conventionally, the task of visual odometry mainly relies on the input of continuous images. However, it is very complicated for the odometry network to learn the epipolar geometry information provided by the images. Since the 6 degree of freedom (DoF) poses transformation occurs in 3D space and learning poses from 3D point clouds is more straightforward, this paper introduces the concept of pseudo-LiDAR to the odometry task. The pseudo-LiDAR point cloud is formed by back-projecting the depth map generated from the image into 3D space. Due to the limitation of calculation power, most current algorithms based on the point cloud need to sample 8192 points from the point cloud as input, but such an approach makes the rich point cloud information in the pseudo-LiDAR point cloud not fully utilized. To address this problem, a projection-aware algorithm is adopted, which achieves efficient point cloud learning and improves the accuracy of the network while preserving the 3D structure information in the pseudo-LiDAR point cloud. Finally, an image-only 2D-3D fusion module is proposed to enhance the pseudo-LiDAR point features using information such as the texture and color of the images. Through multimodal fusion, the network achieves a deeper understanding of the environment. Experiments on the KITTI dataset prove the effectiveness of our method. The source code will be open-sourced at https://github.com/IRMVLab/Pseudo-LiDAR-for-Visual-Odometry.

Due to GPS restrictions, an inertial sensor is usually used to estimate the location of indoor mobile robots. However, it is difficult to achieve high-accuracy localization and control by inertial sensors alone. In this paper, a new method is proposed to estimate an indoor mobile robot pose with six degrees of freedom based on an improved 3D-Normal Distributions Transform algorithm (3D-NDT). First, point cloud data are captured by a Kinect sensor and segmented according to the distance to the robot. After the segmentation, the input point cloud data are processed by the Approximate Voxel Grid Filter algorithm in different sized voxel grids. Second, the initial registration and precise registration are performed respectively according to the distance to the sensor. The most distant point cloud data use the 3D-Normal Distributions Transform algorithm (3D-NDT) with large-sized voxel grids for initial registration, based on the transformation matrix from the odometry method. The closest point cloud data use the 3D-NDT algorithm with small-sized voxel grids for precise registration. After the registrations above, a final transformation matrix is obtained and coordinated. Based on this transformation matrix, the pose estimation problem of the indoor mobile robot is solved. Test results show that this method can obtain accurate robot pose estimation and has better robustness.