19 August 2024 in IEEE Transactions on Intelligent Vehicles
We study the problem of self-supervised 3D scene flow estimation from real large-scale raw point cloud sequences. The problem is crucial to various automotive tasks like trajectory prediction, object detection, and scene reconstruction. In the absence of ground truth scene flow labels, contemporary approaches concentrate on deducing and optimizing flow across sequential pairs of point clouds by incorporating structure-based regularization on flow and object rigidity. The rigid objects are estimated by a variety of 3D spatial clustering methods. While state-of-the-art methods successfully capture overall scene motion using the Neural Prior structure, they encounter challenges in discerning multi-object motions. We identified the structural constraints and the use of large and strict rigid clusters as the main pitfall of the current approaches, and we propose a novel clustering approach that allows for a combination of overlapping soft clusters and non-overlapping rigid clusters. Flow is then jointly estimated with progressively growing non-overlapping rigid clusters together with fixed-size overlapping soft clusters. We evaluate our method on multiple datasets with LiDAR point clouds, demonstrating superior performance over the self-supervised baselines and reaching new state-of-the-art results. Our method excels in resolving flow in complicated dynamic scenes with multiple independently moving objects close to each other, including pedestrians, cyclists, and other vulnerable road users. Our codes are publicly available on https://github.com/ctu-vras/let-it-flow .
21 February 2024 in arXiv
This work investigates learning pixel-wise semantic image segmentation in urban scenes without any manual annotation, just from the raw non-curated data collected by cars which, equipped with cameras and LiDAR sensors, drive around a city. Our contributions are threefold. First, we propose a novel method for cross-modal unsupervised learning of semantic image segmentation by leveraging synchronized LiDAR and image data. The key ingredient of our method is the use of an object proposal module that analyzes the LiDAR point cloud to obtain proposals for spatially consistent objects. Second, we show that these 3D object proposals can be aligned with the input images and reliably clustered into semantically meaningful pseudo-classes. Finally, we develop a cross-modal distillation approach that leverages image data partially annotated with the resulting pseudo-classes to train a transformer-based model for image semantic segmentation. We show the generalization capabilities of our method by testing on four different testing datasets (Cityscapes, Dark Zurich, Nighttime Driving and ACDC) without any finetuning, and demonstrate significant improvements compared to the current state of the art on this problem. See project webpage https://vobecant.github.io/DriveAndSegment/ for the code and more.