Prof. Luigi Di Stefano

Camera relocalisation is a key computer vision problem. Common techniques match the current image against keyframes with known poses, directly regress the pose, or estimate the pose using camera-world correspondences. Regression forests have become popular for establishing correspondences. They are accurate, but previously needed to be trained offline on the target scene, preventing relocalisation in new environments. Recently, we showed how to adapt a pre-trained forest to a new scene online. The adapted forests perform similarly to offline forests, and can estimate the pose in near real time. Here, we extend this work to perform significantly better and run fully in real time: instead of simply accepting the pose produced by RANSAC, we score its final few hypotheses and select the best one; we chain several instances of our relocaliser (with different parameters) in a cascade, trying fast, less accurate relocalisation first, and falling back to slow, accurate relocalisation if needed; finally, we tune the hyperparameters of our relocalisers to achieve effective overall performance. We achieve state-of-the-art performance on the well-known 7-Scenes and Stanford 4 Scenes benchmarks. We visualise the internal behaviour of our forests, and show how to remove the need to train them offline on a generic scene.

12 September 2019 in IEEE Transactions on Pattern Analysis and Machine Intelligence

Journal Article

Unsupervised Domain Adaptation for Depth Prediction from Images

State-of-the-art methods to infer dense and accurate depth measurements from images rely on deep CNN models trained in an end-to-end fashion on a significant amount of data. However, despite the outstanding performance achieved, these frameworks suffer a drastic drop in accuracy when dealing with unseen environments much different, concerning appearance (e.g., synthetic vs. real) or context (e.g., indoor vs. outdoor), from those observed during the training phase. Such domain shift issue is usually softened by fine-tuning on smaller sets of images with depth labels acquired in the target domain with active sensors (e.g., LiDAR). However, relying on such supervised labeled data is seldom feasible in practical applications. Therefore, we propose an effective unsupervised domain adaptation technique enabling to overcome the domain shift problem without requiring any groundtruth label. Our method, deploying much more accessible to obtain stereo pairs, leverages traditional and not learning-based stereo algorithms to produce disparity/depth labels and on confidence measures to assess their degree of reliability. With these cues, we can fine-tune deep models through a novel confidence-guided loss function, neglecting the effect of outliers gathered from the output of conventional stereo algorithms.

28 April 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence

Journal Article

Continual Adaptation for Deep Stereo

Depth estimation from stereo images is carried out with unmatched results by convolutional neural networks trained end-to-end to regress dense disparities. Like for most tasks, this is possible if large amounts of labelled samples are available for training, possibly covering the whole data distribution encountered at deployment time. Being such an assumption systematically unmet in real applications, the capacity of adapting to any unseen setting becomes of paramount importance. Purposely, we propose a continual adaptation paradigm for deep stereo networks designed to deal with challenging and ever-changing environments. We design a lightweight and modular architecture, Modularly ADaptive Network (MADNet), and formulate Modular ADaptation algorithms (MAD, MAD++) which permit efficient optimization of independent sub-portions of the entire network. In our paradigm, the learning signals needed to continuously adapt models online can be sourced from self-supervision via right-to-left image warping or from traditional stereo algorithms. With both sources, no other data than the input images being gathered at deployment time are needed. Thus, our network architecture and adaptation algorithms realize the first real-time self-adaptive deep stereo system and pave the way for a new paradigm that can facilitate practical deployment of end-to-end architectures for dense disparity regression.

11 October 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence

Journal Article

Booster: A Benchmark for Depth From Images of Specular and Transparent Surfaces

Pierluigi Zama Ramirez

Estimating depth from images nowadays yields outstanding results, both in terms of in-domain accuracy and generalization. However, we identify two main challenges that remain open in this field: dealing with non-Lambertian materials and effectively processing high-resolution images. Purposely, we propose a novel dataset that includes accurate and dense ground-truth labels at high resolution, featuring scenes containing several specular and transparent surfaces. Our acquisition pipeline leverages a novel deep space-time stereo framework, enabling easy and accurate labeling with sub-pixel precision. The dataset is composed of 606 samples collected in 85 different scenes, each sample includes both a high-resolution pair (12 Mpx) as well as an unbalanced stereo pair (Left: 12 Mpx, Right: 1.1 Mpx), typical of modern mobile devices that mount sensors with different resolutions. Additionally, we provide manually annotated material segmentation masks and 15 K unlabeled samples. The dataset is composed of a train set and two test sets, the latter devoted to the evaluation of stereo and monocular depth estimation networks. Our experiments highlight the open challenges and future research directions in this field.

18 September 2019 in IEEE Transactions on Automation Science and Engineering

Journal Article

Semiautomatic Labeling for Deep Learning in Robotics

In this article, we propose an augmented reality semiautomatic labeling (ARS), a semiautomatic method which leverages on moving a 2-D camera by means of a robot, proving precise camera tracking, and an augmented reality pen (ARP) to define initial object bounding box, to create large labeled data sets with minimal human intervention. By removing the burden of generating annotated data from humans, we make the deep learning technique applied to computer vision, which typically requires very large data sets, truly automated and reliable. With the ARS pipeline, we created two novel data sets effortlessly, one on electromechanical components (industrial scenario) and other on fruits (daily-living scenario) and trained two state-of-the-art object detectors robustly, based on convolutional neural networks, such as you only look once (YOLO) and single shot detector (SSD). With respect to conventional manual annotation of 1000 frames that takes us slightly more than 10 h, the proposed approach based on ARS allows to annotate 9 sequences of about 35,000 frames in less than 1 h, with a gain factor of about 450. Moreover, both the precision and recall of object detection is increased by about 15% with respect to manual labeling. All our software is available as a robot operating system (ROS) package in a public repository alongside with the novel annotated data sets.

About Me

Luigi Di Stefano is Full Professor at the Department of Computer Science and Engineering (DISI) of the University of Bologna, where he leads the Computer Vision Laboratory (CVLab). Luigi Di Stefano teaches Computer Architecture for undergraduate students and Computer Vision and Image Processing for master students, both at the School of Engineering of the University of Bologna. He also teaches Computer Vision within the Master in Digital Technology Management – Artificial Intelligence Track at the BBS (Bologna Business School). His research interests are focused on computer vision and machine/deep learning. In these fields, he has coordinated many research projects funded by public grants and private companies and he is author of more than 160 papers in renowned international journals and conferences as well as of several patents. He has published in top-tier journals and conferences such as IEEE PAMI, IJCV, IEEE TIP, CVPR, ICCV, ECCV and NeurIPS. He was a winner of the Runner-Up Paper Award of the 2011 3DIMPVT Conference. He has given invited lectures in PhD schools and many invited talks at academic institution and companies, such as: ETH-Zurich, IIT-Genoa, TUW-Vienna, Huawei, STM, Datalogic, Pirelli, Lamborghini, Ferrari, FCA, SACMI. In 2009-2011 and 2015-2017 he has been a member of the Board of Directors of Datalogic SpA. Since 2011 he is scientific consultant for Pirelli Tyres in the area of computer vision. In 2020 he co-founded the Start-up company eyecan.ai.

Prof. Luigi Di Stefano

Prof. Luigi Di Stefano

Research Focus & Metrics

Academic Career Length

Publication Type Breakdown

Representative Publications

About Me