13 May 2024 in IEEE Robotics and Automation Letters
We introduce an integrated precise LiDAR, Inertial, and Visual (LIV) multimodal sensor fused mapping system that builds on the differentiable Gaussians to improve the mapping fidelity, quality, and structural accuracy. Notably, this is also a novel form of tightly coupled map for LiDAR-visual-inertial sensor fusion. This system leverages the complementary characteristics of LiDAR and visual data to capture the geometric structures of large-scale 3D scenes and restore their visual surface information with high fidelity. The initialization for the scene's surface Gaussians and the sensor's poses of each frame are obtained using a LiDAR-inertial system with the feature of size-adaptive voxels. Then, we optimized and refined the Gaussians using visual-derived photometric gradients to optimize their quality and density. Our method is compatible with various types of LiDAR, including solid-state and mechanical LiDAR, supporting both repetitive and non-repetitive scanning modes. Bolstering structure construction through LiDAR and facilitating real-time generation of photorealistic renderings across diverse LIV datasets. It showcases notable resilience and versatility in generating real-time photorealistic scenes potentially for digital twins and virtual reality, while also holding potential applicability in real-time SLAM and robotics domains. We release our software and hardware and self-collected datasets on Github to benefit the community.
20 March 2025 in IEEE/ASME Transactions on Mechatronics
Loop closing is a core module of simultaneous localization and mapping (SLAM) in robotics. It can be achieved through different sensors, of which LiDAR is dominant in large-scale scenes. The LiDAR-based loop closing involves two subtasks: loop closure detection and point cloud registration. Loop closure detection is challenged by large viewpoint changes such as reverse loops, and point cloud registration in Euclidean space is affected by noise and sparsity of point clouds. To solve these problems, we propose to jointly leverage SE(3)-equivariant and -invariant features. Specifically, we first extract equivariant features and map them into invariant features. Then, invariant features are aggregated into global descriptors for loop closure detection. After that, equivariant and invariant features are both applied to point cloud registration. Our strategy has two main advantages. First, our equivariance-to-invariance mapping decouples geometry and pose information in the point cloud. Accordingly, invariant features exhibit high robustness to viewpoint changes by purely encoding the geometry information. Second, equivariant features in latent space, which encode the pose information, are more robust to noise than the points in Euclidean space for pose estimation. Extensive experiments show that our method outperforms state-of-the-art approaches in both accuracy and robustness.
11 August 2025 in IEEE Transactions on Instrumentation and Measurement
The localization method based on aerial images and satellite maps is an effective solution for unmanned aerial vehicles (UAVs) in Global Navigation Satellite Systems (GNSS) denied conditions. However, in addition to differences in scale and perspective, the difficulty of matching increases due to changes in lighting and terrain. At the same time, the search area of the satellite map relies on previous matching results, so previous errors can easily cause cascading effects, significantly reducing the accuracy of localization. To overcome these challenges, a multifactor tightly coupled visual localization method is proposed in this article, which avoids the interference of single-modal outliers by introducing multifactor joint optimization. Second, a search strategy on the satellite map is proposed, which continuously updates the candidate area based on the status of the UAV to improve the efficiency of the search and the accuracy of matching. Finally, an updated method of satellite map is designed to avoid matching failures caused by lagging satellite map updates. Multiple times of experiments are conducted using two different types of UAVs in different conditions, ranges, altitudes, and speeds to verify the effectiveness of the proposed method. Comparison with the state-of-the-art visual localization method shows that the proposed method improves the accuracy of localization by 27.1% and the efficiency of processing by 30.6%.
5 March 2025 in IEEE Transactions on Instrumentation and Measurement
Navigating in unknown environments with multiple and irregular obstacles is a crucial task for autonomous mobile systems, such as autonomous driving vehicles and mobile robots. However, most of the existing path planners can only effectively solve this problem under the small-scale precise perception conditions, and cannot handle the large-scale and high-dynamic environments. In this paper, we propose a new efficient path planner called IM Planner, which concentrates on the updating and utilization of Incremental patch Map (IM). IM adopts a hierarchical data structure, combining multiple adjacent girds into patches as the basic unit, which facilitates memory management and query operations, and provides accurate obstacle information. After partial sensor observations arrival, IM is continuously updated in patch blocks. At the same time, tightly coupled with IM, the non-optimized search algorithm utilizes a bidirectional strategy, and integrates a fast and accurate collision detection model, which can generate safe, kinematically feasible, and smooth trajectories without post optimization. Extensive experimental results show that IM Planner achieves enhanced efficiency and safety performance among the offline parking benchmark and the online CARLA simulation. The code are available at https://github.com/chenghuang66/IM-Planner.
12 March 2026 in IEEE Transactions on Image Processing
Generating dynamic scenes from images has gained increasing attention. Existing methods have two major limitations: (1) They can hardly handle sparse images which exhibit limited geometry constraints and insufficient motion; (2) They struggle to maintain spatial-temporal consistency when rendering multi-view videos. To address these limitations, we propose SCSV, a spatial-temporal consistent dynamic scene generation method from sparse views. Our method consists of two stages: scene reconstruction and scene expansion, both of which decouple background and foreground. In the scene reconstruction stage, we first interpolate a set of images between the input images based on a video generation model, followed by the optimization of the scene Gaussian from the interpolated and input images. To improve the spatial-temporal consistency of the reconstructed scene, we propose an uncertainty-aware Gaussian training approach, which introduces adaptive weights of images and pixels. In the scene expansion stage, for background, we render novel views and refine them with a geometry-aware diffusion process. These refined images are then used to incrementally add the Gaussians. As to foreground, we generate human motion according to previous motion, enabling temporal coherent generation of motion. To further enhance the physical plausibility, we integrate the expanded foreground into the background using a gravity-aware alignment. Experiments on NeuMan, Bonn, and EMDB datasets demonstrate that our SCSV achieves superior performance compared to state-of-the-art methods. The code will be released upon acceptance.