Ubiquitous low cost multi-rotor and fixed wing drones or unmanned aerial vehicles (UAVs) have accelerated the need for reliable, robust, and scalable Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines suitable for a variety of flightpath trajectories especially in degraded environments. Feature tracking being a core part of SfM and MVS, is essential for multiview scene modeling and perception, but difficult to evaluate in large scale datasets due to the lack of sufficient ground-truth. For large-scale aerial imagery, accurate camera orientation and dense 3D point cloud accuracy can be used to assess the impact of accurate feature localization and track length. We propose a novel view simulation (or synthesis) framework which generates visually realistic new unseen camera views for feature detection using known high fidelity camera poses for modeling. Seven state-of-the-art local handcrafted and learning-based features are quantitatively evaluated for robustness and matchability within the SfM and MVS pipelines using the open source COLMAP software. Our experimental results provide performance rankings of each feature, using twelve different evaluation metrics across three synthetic city-wide aerial image sequences. We show that recent learned features, SuperPoint and LF-Net, have not only reached the quality of the best handcrafted features like SIFT and SURF, but now outperform them in terms of more accurate 3D camera pose estimates and longer feature tracks. SuperPoint produces 1.51 meter average position error and 0.03° average angular error, while SIFT remains competitive (third best for pose and second overall) with 1.78 meter and 0.11° errors respectively.

Multi-modal systems extract information about the environment using specialized sensors that are optimized based on the wavelength of the phenomenology and material interactions. To maximize the entropy, complementary systems operating in regions of non-overlapping wavelengths are optimal. VIS-IR (Visible-Infrared) systems have been at the forefront of multi-modal fusion research and are used extensively to represent information in all-day all-weather applications. Prior to image fusion, the image pairs have to be properly registered and mapped to a common resolution palette. However, due to differences in the device physics of image capture, information from VIS-IR sensors cannot be directly correlated, which is a major bottleneck for this area of research. In the absence of camera metadata, image registration is performed manually, which is not practical for large datasets. Most of the work published in this area assumes calibrated sensors and the availability of camera metadata providing registered image pairs, which limits the generalization capability of these systems. In this work, we propose a novel end-to-end pipeline termed DeepFusion for image registration and fusion. Firstly, we design a recursive crop and scale wavelet spectral decomposition (WSD) algorithm for automatically extracting the patch of visible data representing the thermal information. After data extraction, both the images are registered to a common resolution palette and forwarded to the DNN for image fusion. The fusion performance of the proposed pipeline is compared and quantified with state-of-the-art classical and DNN architectures for open-source and custom datasets demonstrating the efficacy of the pipeline. Furthermore, we also propose a novel keypoint-based metric for quantifying the quality of fused output.

We investigate combinations of 3D-2D matches of points and lines for a single calibrated camera that imply a nontrivial constraint on the camera’s position. Specifically, we address the minimal cases for which n point matches and m line matches constrain the camera center to lie on an algebraic surface in \(\mathbb {R}^3\). For two points \((m,n) = (0,2)\), the constraint is well-known to be a self-intersecting torus. We complete the classification problem by deriving explicit, general equations for the cases \((m,n) \in \{ (2,0), (1, 1) \}\) for two lines and one point, one line, respectively. We also report preliminary experiments investigating the suitability of these constraints for pose estimation tasks.