Traditionally, road detection approaches mostly capitalize on RGB images, 3D LiDAR point cloud or their fusion. However, RGB camera is sensitive to light conditions, while LiDAR point cloud is sparse compared with dense image pixels. In this work, a new hybrid image dataset is provided for the task of road detection based on cameras. In this dataset, the hybrid images are acquired by an optically aligned hybrid imaging device, consisting of a far-infrared (FIR) imager and an RGB camera to output pixel-wise registration of thermal and RGB frames. Then we investigate on three methods based on fully convolutional neural network (F-CNN) to demonstrate the advantages by fusing RGB-FIR images in road detection. First, a middle-fusion based model is built, where the output feature maps of encoder branches from RGB and FIR images are directly concatenated into a single-fusion branch as the decoder. Next, the originally discarded layers after fusion operation for both RGB and FIR branches are recovered as the mimic branches to imitate the distributions of the fusion outputs, which constitutes an extended cross model (ECM). Moreover, the outputs of mimic branches at different scales are also used to imitate the corresponding outputs in the fusion branch, called a hierarchical cross model (HCM). The experimental results demonstrate the effectiveness and efficiency of our fusion strategies.
In this work, a novel paradigm is proposed to build a hybrid RGB/FIR camera, which outputs pixel-wisely aligned and temporally synchronized RGB and FIR frames simultaneously. Compared with the existing schemes, we innovate in several ways to make it compact, practical, and extendable for real-world applications. First, we propose a new structure layout design of the hybrid camera, and we also design a removable adjuster to detach the hybrid camera from the base platform. For RGB and FIR image registration, we have achieved a pixel-wise spatial registration of the FIR and RGB frames by a two-step alignment, where we propose a ghost-effect removal scheme to remove the ghost effect from the rolling-shutter-based FIR camera. Finally, multiple hybrid cameras are extended to a hybrid camera array, through which we can obtain wide-view spatially aligned FIR, RGB and disparity images simultaneously. The experimental results demonstrate the accuracy of the hybrid camera and the effectiveness of the designed stereo wide-view hybrid camera system.
19 November 2025 in IEEE Robotics and Automation Letters
Semantic Scene Completion (SSC) is a task that simultaneously predicts the occupancy and semantic labels of the environment. Compared with separate processing, SSC leverages the coupled nature of scene completion and semantic segmentation. Although this multitask integration can utilize complementarity and correlation between tasks, it also increases the training difficulty. To address this, in this letter, we propose a Semantic Decoupling based Semantic Scene Completion (SD-SSC) network from a single depth image. The semantic segmentation task is decoupled from the semantic scene completion task, and we use 2D and 3D semantic supervision to simplify the scene completion task and improve SSC performance. Specifically, our network first performs 2D semantic segmentation on the depth image and transforms features into 3D voxel space as semantic priors. Then, the 3D SSC is performed based on the voxel features and the flipped Truncated Signed Distance Field (f-TSDF). We use multi-scale 3D semantic supervision to further enhance the semantic information and fuse semantic and geometric features through the Planar Attention Fusion Module (PAFM) to obtain accurate SSC results. The proposed SD-SSC network achieves state-of-the-art performance on the NYU dataset (51.1% mIoU) and the NYUCAD dataset (61.9% mIoU) among all single depth-image based methods. It is even better than most RGB-D fusion-based SSC methods.