Semantic segmentation and stereo matching are two essential components of 3D environmental perception systems for autonomous driving. Nevertheless, conventional approaches often address these two problems independently, employing separate models for each task. This approach poses practical limitations in real-world scenarios, particularly when computational resources are scarce or real-time performance is imperative. Hence, in this article, we introduce S $^{3}$ M-Net, a novel joint learning framework developed to perform semantic segmentation and stereo matching simultaneously. Specifically, S $^{3}$ M-Net shares the features extracted from RGB images between both tasks, resulting in an improved overall scene understanding capability. This feature sharing process is realized using a feature fusion adaption (FFA) module, which effectively transforms the shared features into semantic space and subsequently fuses them with the encoded disparity features. The entire joint learning framework is trained by minimizing a novel semantic consistency-guided (SCG) loss, which places emphasis on the structural consistency in both tasks. Extensive experimental results conducted on the vKITTI2 and KITTI datasets demonstrate the effectiveness of our proposed joint learning framework and its superior performance compared to other state-of-the-art single-task networks. Our project webpage is accessible at mias.group/S3M-Net.
7 July 2025 in IEEE Transactions on Automation Science and Engineering
Semantic segmentation and stereo matching, respectively analogous to the ventral and dorsal streams in our human brain, are two key components of autonomous driving perception systems. Addressing these two tasks with separate networks is no longer the mainstream direction in developing computer vision algorithms, particularly with the recent advances in large vision models and embodied artificial intelligence. The trend is shifting towards combining them within a joint learning framework, especially emphasizing feature sharing between the two tasks. The major contributions of this study lie in comprehensively tightening the coupling between semantic segmentation and stereo matching. Specifically, this study makes three key contributions: (1) a tightly coupled, gated feature fusion strategy, (2) a hierarchical deep supervision strategy, and (3) a coupling tightening loss function. The combined use of these technical contributions results in TiCoSS, a state-of-the-art joint learning framework that simultaneously tackles semantic segmentation and stereo matching. Through extensive experiments on the KITTI, vKITTI2, and Cityscapes datasets, along with both qualitative and quantitative analyses, we validate the effectiveness of our developed strategies and loss function. Our approach demonstrates superior performance compared to prior arts, with a notable increase in mean intersection over union by over 9%.
23 October 2025 in IEEE Transactions on Image Processing
Despite the impressive performance achieved by data-fusion networks with duplex encoders for visual semantic segmentation, they become ineffective when spatial geometric data are not available. Implicitly infusing the spatial geometric prior knowledge acquired by a data-fusion teacher network into a single-modal student network is a practical, albeit less explored research avenue. This article delves into this topic and resorts to knowledge distillation approaches to address this problem. We introduce the Learning to Infuse “X” (LIX) framework, with novel contributions in both logit distillation and feature distillation aspects. We present a mathematical proof that underscores the limitation of using a single, fixed weight in decoupled knowledge distillation and introduce a logit-wise dynamic weight controller as a solution to this issue. Furthermore, we develop an adaptively-recalibrated feature distillation algorithm, including two novel techniques: feature recalibration via kernel regression and feature consistency quantification via centered kernel alignment. Extensive experiments conducted with intermediate-fusion and late-fusion networks across various public datasets provide both quantitative and qualitative evaluations, demonstrating the superior performance of our LIX framework when compared to other state-of-the-art approaches. Source code is available at https://mias.group/LIX.
15 October 2025 in IEEE Robotics and Automation Letters
Robotic pushing is a fundamental manipulation task that requires tactile feedback to capture subtle contact forces and dynamics between the end-effector and the object. However, real tactile sensors often face hardware limitations such as high costs and fragility, and deployment challenges involving calibration and variations between different sensors, while vision-only policies struggle with satisfactory performance. Inspired by humans' ability to infer tactile states from vision, we propose ViTacGen, a novel robot manipulation framework designed for visual robotic pushing with vision-to-touch generation in reinforcement learning to eliminate the reliance on high-resolution real tactile sensors, enabling effective zero-shot deployment on visual-only robotic systems. Specifically, ViTacGen consists of an encoder-decoder vision-to-touch generation network that generates contact depth images, a standardized tactile representation, directly from visual image sequence, followed by a reinforcement learning policy that fuses visual-tactile data with contrastive learning based on visual and generated tactile observations. We validate the effectiveness of our approach in both simulation and real world experiments, demonstrating its superior performance and achieving a success rate of up to 86%. Code and data are available on https://robot-perception-lab.github.io/vitacgen-website/.
16 June 2025 in IEEE Transactions on Instrumentation and Measurement
Freespace detection is essential for autonomous robotic systems, particularly self-driving vehicles that depend on precise environmental perception for safe navigation. State-of-the-art methods typically use dual encoders to extract features from RGB images and other data sources, which are then fused and decoded for freespace prediction. This article presents SG-RoadSeg+, which incorporates the transformed disparity-azimuth-zenith (TAZ) map and a novel freespace disparity projection linearity (FDPL) loss function. The key advancement is the TAZ map, which integrates transformed disparity and surface normal maps to enrich spatial understanding, necessary for accurate freespace detection. Additionally, the FDPL loss function optimizes the linear projection of the disparity during training, ensuring precise environmental measurements. Moreover, our data augmentation technique, using random perspective transformations, primarily enhances the system’s generalization to real-world environments, enabling it to handle complex and dynamic scenarios more effectively. Our previous model, SG-RoadSeg, uses a deep stereo encoder, resulting in strong freespace detection performance. However, its reliance on a single estimated disparity map limits the available spatial information, potentially leading to inaccuracies. Comprehensive experiments on the KITTI Road and Semantics datasets show that SG-RoadSeg+ outperforms the previous version, with improvements of up to 0.73% in IoU and 0.51% in F-score. Notably, applying the TAZ map to other feature-fusion models improves F-score by 3.27% and IoU by 6.44%, highlighting its effectiveness in boosting freespace detection accuracy.
1 September 2026 in IEEE Robotics and Automation Letters