1 August 2024 in IEEE Transactions on Automation Science and Engineering
The creation of a metric-semantic map, which encodes human-prior knowledge, represents a high-level abstraction of environments. However, constructing such a map poses challenges related to the fusion of multi-modal sensor data, the attainment of real-time mapping performance, and the preservation of structural and semantic information consistency. In this paper, we introduce an online metric-semantic mapping system that utilizes LiDAR-Visual-Inertial sensing to generate a global metric-semantic mesh map of large-scale outdoor environments. Leveraging GPU acceleration, our mapping process achieves exceptional speed, with frame processing taking less than $7ms$ , regardless of scenario scale. Furthermore, we seamlessly integrate the resultant map into a real-world navigation system, enabling metric-semantic-based terrain assessment and autonomous point-to-point navigation within a campus environment. Through extensive experiments conducted on both publicly available and self-collected datasets comprising $24$ sequences, we demonstrate the effectiveness of our mapping and navigation methodologies. Note to Practitioners —This paper tackles the challenge of autonomous navigation for mobile robots in complex, unstructured environments with rich semantic elements. Traditional navigation relies on geometric analysis and manual annotations, struggling to differentiate similar structures like roads and sidewalks. We propose an online mapping system that creates a global metric-semantic mesh map for large-scale outdoor environments, utilizing GPU acceleration for speed and overcoming the limitations of existing real-time semantic mapping methods, which are generally confined to indoor settings. Our map integrates into a real-world navigation system, proven effective in localization and terrain assessment through experiments with both public and proprietary datasets. Future work will focus on integrating kernel-based methods to improve the map’s semantic accuracy.
19 December 2022 in IEEE Robotics and Automation Letters
Having good knowledge of terrain information is essential for improving the performance of various downstream tasks on complex terrains, especially for the locomotion and navigation of legged robots. We present a novel framework for neural urban terrain reconstruction with uncertainty estimations. It generates dense robot-centric elevation maps online from sparse LiDAR observations. We design a novel pre-processing and point features representation approach that ensures high robustness and computational efficiency when integrating multiple point cloud frames. A generative Bayesian model then recovers the detailed terrain structures while simultaneously providing the pixel-wise reconstruction uncertainty. We evaluate the proposed pipeline through both simulation and real-world experiments. Our approach achieves high-quality terrain reconstruction with real-time performance on a mobile platform, and the uncertainty estimates may further benefit the downstream tasks of legged robots.
17 May 2024 in IEEE Robotics and Automation Letters
Global point clouds that correctly represent the static environment features can facilitate accurate localization and robust path planning. However, dynamic objects introduce undesired ‘ghost’ tracks that are mixed up with the static environment. Existing dynamic removal methods normally fail to balance the performance in computational efficiency and accuracy. In response, we present ‘BeautyMap’ to efficiently remove the dynamic points while retaining static features for high-fidelity global maps. Our approach utilizes a binary-encoded matrix to efficiently extract the environment features. With a bit-wise comparison between matrices of each frame and the corresponding map region, we can extract potential dynamic regions. Then we use coarse to fine hierarchical segmentation of the $z$ -axis to handle terrain variations. The final static restoration module accounts for the range-visibility of each single scan and protects static points out of sight. Comparative experiments underscore BeautyMap's superior performance in both accuracy and efficiency against other dynamic points removal methods.
16 May 2024 in IEEE/ASME Transactions on Mechatronics
Navigation in complex 3-D scenarios requires appropriate environment representation for efficient scene understanding and trajectory generation. We propose a highly efficient and extensible global navigation framework based on a tomographic understanding of the environment to navigate ground robots in multilayer structures. Our approach generates tomogram slices using the point cloud map to encode the geometric structure as ground and ceiling elevations. Then, it evaluates the scene traversability considering the robot's motion capabilities. Both the tomogram construction and the scene evaluation are accelerated through parallel computation. Our approach further alleviates the trajectory generation complexity compared with planning in 3-D spaces directly. It generates 3-D trajectories by searching through multiple tomogram slices and separately adjusts the robot height to avoid overhangs. We evaluate our framework in various simulation scenarios and further test it in the real world on a quadrupedal robot. Our approach reduces the scene evaluation time by three orders of magnitude and improves the path planning speed by three times compared with existing approaches, demonstrating highly efficient global navigation in various complex 3-D environments.
20 September 2024 in IEEE/ASME Transactions on Mechatronics
Mobile robots, such as quadrupedal and vehicular robots, are known for their high-speed movement and operation in environments with wide dynamic ranges. This property challenges the existing camera capture methods for visual applications, especially the visual simultaneous localization and mapping (SLAM) task, which requires a strong temporal continuity. Due to the limitations imposed by the camera hardware's control frequency and delay, camera exposure control methods cannot rapidly and stably publish high-quality images preventing oversaturation, background noise, and motion blur. In this article, we propose a novel image acquisition framework called VirCap, introducing image bracketing capture patterns to preserve more dynamic range information than a single image capture. By leveraging image photometric synthetic technology, VirCap enables a virtual camera exposure control (exposure time and analog gain), effectively decoupling the camera interaction from the control loop and facilitating more frequent exposure updates than traditional camera control methods. An exposure allocation strategy is also developed to balance motion blur and background noise, allowing VirCap to synthesize images of optimal quality that consider the robot's self-motion. Extensive experiments are conducted to demonstrate the efficiency and resilience of VirCap under extreme operating conditions for different visual SLAM systems.
21 August 2025 in IEEE Transactions on Instrumentation and Measurement
To address the challenges of hand-eye calibration for Intelligent Guided Vehicles operating in degenerated environments such as container ports, this paper presents an innovative solution. Conventional hand-eye calibration techniques have difficulty accurately estimating camera extrinsic parameters, especially height, in flat 2D scenes. This issue is further compounded when IGVs carry containers of varying weights, causing dynamic changes in the sensor’s extrinsic parameters. To overcome this difficulty, we propose an IGV extrinsic parameter variation estimation module and a constrained global optimal hand-eye calibration method.We develop a mathematical mapping model relating the loaded container’s weight to the change in sensor extrinsic parameters. The model estimates and compensates for sensor height and pitch angle changes by reading the loaded container’s weight, solving the degeneration issue of hand-eye calibration in 2D environments. We create a new solver to handle the problem of hand-eye calibration frequently converging to local optima under degenerated motion conditions. The solver finds the global optimum for hand-eye calibration under height or pitch angle constraints, and works with unknown scale, greatly enhancing calibration accuracy and applicability. Simulations and real-world tests have validated the proposed method, showing superior performance in extrinsic parameter estimation accuracy and environmental adaptability compared to state-of-the-art techniques.