In this paper, we present a novel onboard robust visual algorithm for long-term arbitrary 2D and 3D object tracking using a reliable global-local object model for unmanned aerial vehicle (UAV) applications, e.g., autonomous tracking and chasing a moving target. The first main approach in this novel algorithm is the use of a global matching and local tracking approach. In other words, the algorithm initially finds feature correspondences in a way that an improved binary descriptor is developed for global feature matching and an iterative Lucas–Kanade optical flow algorithm is employed for local feature tracking. The second main module is the use of an efficient local geometric filter (LGF), which handles outlier feature correspondences based on a new forward-backward pairwise dissimilarity measure, thereby maintaining pairwise geometric consistency. In the proposed LGF module, a hierarchical agglomerative clustering, i.e., bottom-up aggregation, is applied using an effective single-link method. The third proposed module is a heuristic local outlier factor (to the best of our knowledge, it is utilized for the first time to deal with outlier features in a visual tracking application), which further maximizes the representation of the target object in which we formulate outlier feature detection as a binary classification problem with the output features of the LGF module. Extensive UAV flight experiments show that the proposed visual tracker achieves real-time frame rates of more than thirty-five frames per second on an i7 processor with 640 × 512 image resolution and outperforms the most popular state-of-the-art trackers favorably in terms of robustness, efficiency and accuracy.

This article presents a novel outlier rejection approach for feature-based visual odometry. The proposed approach is based on an empirical observation that shows that some 2D–3D correspondences with very low reprojection error can cause a high error in pose estimation. This work exploits such observations for odometry when a stereo camera is available. We argue that an explicit pose error measure is desired over that of implicit reprojection – whenever the former is possible – to classify correspondences into inliers versus outliers for the robust long-term odometry. To explicitly measure the plausible pose error, we derive bounds on the individual pose parameters with the help of the known orientation of stereo cameras. In this process, we formulate our bounds using the sum-of-square polynomials, which allow us to test whether a given correspondence satisfies any solution within the expected bounds. If the correspondence does not satisfy bounds for any parameter, it is considered to be an outlier. We implemented and tested the proposed method for the unmanned aerial vehicle (UAV) indoor navigation. The experiments from both benchmark evaluations (EuRoC and KITTI) and UAV onboard tests indicate that the inlier group refined by the proposed method significantly improves odometry estimation compared to the traditional outlier rejection methods. In fact, the proposed method performs as accurately as inertial measurement unit (IMU) aided methods in the state of the art. ROS project page (open-source code with demo video) is available at: https://github.com/arclab-hku/SOPVO

This work addresses the loop closure detection issue by matching the local pose graphs for semantic visual SLAM. We propose a deep feature matching-based keyframe retrieval approach. The proposed method treats the local navigational maps as images. Thus, the keyframes may be considered keypoints of the map image. The descriptors of the keyframes are extracted using a convolutional neural network. As a result, we convert the loop closure detection problem to a feature matching problem so that we can solve the keyframe retrieval and pose graph matching concurrently. This process in our work is carried out by modified deep feature matching (DFM). The experimental results on the KITTI and Oxford RobotCar benchmarks show the feasibility and capabilities of accurate loop closure detection and the potential to extend to multiagent applications.

The sectional insulator with arcing horns is indispensable for electrified railways’ overhead contact system (OCS). When these arcing horns are damaged, broken, or detached, arcing can occur due to unstable contact between the pantograph and catenary, potentially causing burns and damage to OCS components or even complete system failure. Unfortunately, no specialized technique or method for detecting arcing horn defects exists. To address this critical issue, this paper proposes a novel CTBM-DAHD network that utilizes CNN-Transformer bridge mode to recognize arcing horn defects in realistic application scenarios, such as rainy, foggy, sunny, and night-time conditions. Notably, the network can accurately detect obscured and long-range minor arcing horn defects, significantly improving recall and precision of defect recognition. The experimental findings demonstrate that, compared to the state-of-the-art networks, the CTBM-DAHD network achieves outstanding performance while maintaining lower computational costs and a reduced number of weight parameters. It surpasses the best CNN-Transformer bridge fusion network by 3.5% and outperforms the dual-branch vision transformer by 1.5%. Moreover, the CTBM-DAHD network has been successfully deployed on over 500 high-speed trains in China, detecting more than 1,000 arcing horn defects. These results affirm its effectiveness in recognizing arcing horn defects in complex and natural environments.

This article tackles the visual localization of unmanned aerial vehicles (UAVs) in the presence of multisource and cross-view images are involved. We present a lightweight end-to-end scene graph encoding and matching network that finds the best matches for the airborne camera views from the reference image maps. The scene graph addresses the challenges of encoding the semantic scene by aggregating the image convolutional features into global and structured semiglobal descriptors. The principal contributions of this article are as follows: First, we develop a new network architecture that embeds a nonlocal block and a modified vector of locally aggregated descriptors network (NetVLAD) into a backbone convolutional neural network. The main component of the modified NetVLAD is a cluster similarity masking graph (CSMG) encoder, which is proposed to replace the feature-cluster residuals computing in NetVLAD with cluster consensus feature aggregation and structure-aware scene graph extraction. In addition, a global descriptor is extracted by a nonlocal block to label each image with a discriminative global feature descriptor. Second, we develop a new triplet loss for the network training procedure to learn the features at different semantic levels. The proposed global descriptor and CSMG encoder are trained together according to a weighted sum of cosine triplet losses. Third, the global descriptor from the nonlocal block and semiglobal descriptor from the CSMG encoder work hierarchically for coarse-to-fine image retrieval and can achieve real-time efficiency and favorable accuracy of image searching and matching from the reference image map. We train and test the model on two challenging benchmark datasets. We also test the pretrained model on a dataset collected by a fixed-wing UAV to further evaluate the model's generalizability. The benchmark evaluations and ablation experiments show that the developed method outperforms state-of-the-art methods and achieves superior performance in the real-time matching of UAV images and reference image maps for UAV visual localization. Open-source code is available on GitHub.