Cross-modality images that combine visible-infrared spectra can provide complementary information for object detection. In particular, they are well-suited for autonomous vehicle applications in dark environments with limited illumination. However, it is time-consuming to acquire a large number of pixel-aligned visible-thermal image pairs, and real-time alignment is challenging in practical driving systems. Furthermore, the quality of visible-spectrum images can be adversely affected by complex environmental conditions. In this paper, we propose a novel neural network called TIRDet, which only utilizes Thermal InfraRed (TIR) images for mono-modality object detection. To compensate for the lacked visible-band information, we adopt a prior Thermal-To-Visible (T2V) translation model to obtain the translated visible images and the latent T2V codes. In addition, we introduce a novel attention-based Cross-Modality Aggregation (CMA) module, which can augment the modality-translation awareness of TIRDet by preserving the T2V semantic information. Extensive experiments on FLIR and LLVIP datasets demonstrate that our TIRDet significantly outperforms all mono-modality detection methods based on thermal images, and it even surpasses most State-Of-The-Art (SOTA) multispectral methods using visible-thermal image pairs. Code is available at https://github.com/zeyuwang-zju/TIRDet
Event-based vision sensors show great promise for use in embedded applications requiring low-latency passive sensing at a low computational cost. In this paper, we present an event-based algorithm that relies on an Extended Kalman Filter for 6-Degree of Freedom sensor pose estimation. The algorithm updates the sensor pose event-by-event with low latency (worst case of less than 2 μs on an FPGA). Using a single handheld sensor, we test the algorithm on multiple recordings, ranging from a high contrast printed planar scene to a more natural scene consisting of objects viewed from above. The pose is accurately estimated under rapid motions, up to 2.7 m/s. Thereafter, an extension to multiple sensors is described and tested, highlighting the improved performance of such a setup, as well as the integration with an off-the-shelf mapping algorithm to allow point cloud updates with a 3D scene and enhance the potential applications of this visual odometry solution.
Here we present a novel bio-inspired visual processing system, which enables a robot to locate and follow a target, using an artificial compound eye called CurvACE. This visual sensor actively scanned the environment at an imposed frequency (50Hz) with an angular scanning amplitude of 4.2° and succeeded in locating a textured cylindrical target with hyperacuity, i.e. much finer resolution than the coarse inter-receptor angle of the compound eye. Equipped with this small, lightweight visual scanning sensor, a Mecanum-wheeled mobile robot named ACEbot was able to follow a target at a constant distance by localizing the right and left edges of the target. The localization of the target's contrasted edges is based on a bio-inspired summation of Gaussian receptive fields in the visual system. By means of its auto-adaptive pixels, ACEbot consistently achieved similar pursuit performances under various lighting conditions with a high level of repeatability. The robotic pursuit pattern mimics finely the behavior of the male fly Syritta Pipens L. while pursuing the female. The high similarity in the trajectories as well as the biomimicry of the visual system provides strong support for the hypothesis that flies do maintain center the target and constant its subtended angle during smooth pursuit. Moreover, we discuss the fact that such simple strategy can also provide a trajectory compatible with motion camouflage.
Oeil composé artificiel doté d'hypercuité : applications robotiques à la stabilisation et à la poursuite: Inspirés par les propriétés optiques des yeux composés de la mouche et par l’observation de micromouvements périodiques de sa rétine, différents capteurs visuels ont montré qu’il était possible de localiser un contraste très précisément. Ce fut les premières démonstrations d’hyperacuité visuelle de l’œil de la mouche.Dans cette thèse, un œil composé artificiel, doté d'un large champ de vision, est utilisé. Grâce à un nouvel algorithme de fusion des données visuelles, cet œil peut mesurer le déplacement d’un robot et lui permettre de se stabiliser au-dessus d’un environnement texturé. Localiser un contraste de manière linéaire sur l'ensemble du champ visuel demeure toutefois difficile. Un deuxième algorithme permet d’améliorer la localisation d’une barre grâce à un étalonnage, mais montre une certaine dépendance aux variations de contraste et de luminosité ambiante.Afin d'éviter ce processus d’étalonnage, un troisième algorithme qui s’appuie sur les travaux d'Heiligenberg et Baldi, a été proposé pour localiser deux contrastes. Ces auteurs ont montré que la somme pondérée de plusieurs capteurs ayant un champ récepteur gaussien pouvait fournir une estimation linéaire de la position d'un stimulus. Nous avons, pour la première fois, appliqué une variante de ce principe à un œil composé artificiel. Cet œil, une fois monté sur un robot permet de suivre une cible à une distance constante.Finalement, un œil composé artificiel dont la résolution intrinsèque est faible, peut être doté d’une hyperacuité visuelle et permettre de suivre une cible avec précision. Ces travaux ont ainsi conduit à proposer des stratégies bio-inspirées pour la localisation et la poursuite de cible.