Unmanned Aerial Vehicles (UAVs) have been widely applied in military and civilian fields due to their flexibility and effectiveness. As a vital component of UAVs, the vision system has taken on great significance in different applications (e.g., autonomous landing, traffic surveillance, and disaster rescue) to attract widespread attention in recent years. Therefore, the automatic understanding of visual data collected from these air platforms becomes urgently needed in UAV systems. In this review, we revisit and summarize the recent techniques and developments for several typical UAV applications, including object detection, object tracking, and semantic segmentation. In addition, we also highlight the difficulties and subsequent orientations from different perspectives, which may stimulate future research and applications in the UAV vision era.

The semantic segmentation of remote sensing images faces two major challenges: high inter-class similarity and interference from ubiquitous shadows. In order to address these issues, we develop a novel edge loss reinforced semantic segmentation network (ERN) that leverages the spatial boundary context to reduce the semantic ambiguity. The main contributions of this paper are as follows: (1) we propose a novel end-to-end semantic segmentation network for remote sensing, which involves multiple weighted edge supervisions to retain spatial boundary information; (2) the main representations of the network are shared between the edge loss reinforced structures and semantic segmentation, which means that the ERN simultaneously achieves semantic segmentation and edge detection without significantly increasing the model complexity; and (3) we explore and discuss different ERN schemes to guide the design of future networks. Extensive experimental results on two remote sensing datasets demonstrate the effectiveness of our approach both in quantitative and qualitative evaluation. Specifically, the semantic segmentation performance in shadow-affected regions is significantly improved.

Semantic segmentation and height estimation play fundamental roles in the scene understanding of remote sensing images with their wide variety of aerial applications. Recently, deep convolutional neural networks (DCNNs) have achieved state-of-the-art performance in both tasks. However, DCNNs-based methods learn to accumulate contextual information over large receptive fields while lose the local detailed information, resulting in blurry object boundaries. The complicated ground object distribution and low inter-class variance further aggravate the difficulty in generating accurate predictions. To address the above issues, we propose a novel Boundary-Aware Multi-Task Learning (BAMTL) framework to perform three tasks, semantic segmentation, height estimation, and boundary detection, within a unified model. The boundary detection is employed as an auxiliary task to regularize the other two master tasks at both the feature space and output space. We present a Boundary Attentive Module (BAM) to build the cross-task interaction for master tasks, which enforce the networks to filter out the confident area and focus on learning the high-frequency details. We then introduce a Boundary Regularized Loss (BRL) term to further refine the prediction maps to be locally consistent while preserving boundary structures. With these formulations, our model improves the performance of both segmentation and height tasks, especially along the boundaries. Experimental results on two publicly available remote sensing datasets demonstrate that the proposed approach performs favorably against the state-of-the-art methods.

With the promising techniques and mobile applications of the Internet of Things (IoT) in the fifth-generation (5G) wireless networks, the mobile edge computing (MEC) paradigm has been widely utilized to support the massive data processing and meet the computation demand of mobile devices. However, the service coverage is limited by the static servers. Therefore, the unmanned aerial vehicles (UAVs) with high mobility are deployed as aerial base stations (BS) to assist ground users to offload the task and process the data on edge. In this paper, we introduce a decentralized user allocation and dynamic service scheme to assign the UAV deployment for a multi-UAV MEC system, which incorporates the deep reinforcement learning (DRL) technology into a two-layer training framework. The continuous action, including the UAV trajectory and bit allocation of offloading, is first optimized in the lower layer, while the upper one concentrates on the discrete action space for the user allocation. In particular, we develop a constraint reward function that considers the communication quality in the lower layer to accomplish efficient communication. The well-trained neural network is embedded in the upper layer to provide rapid feedback for the upper environment. Experimental results demonstrate that the proposed approach achieves superior performance than the baseline methods in terms of the user energy consumption.

In this paper, we find that the conventional convolution operation becomes the bottleneck for extremely efficient binary neural networks (BNNs). To address this issue, we open up a new direction by introducing a reshaped point-wise convolution (RPC) to replace the conventional one to build BNNs. Specifically, we conduct a point-wise convolution after rearranging the spatial information into depth, with which at least $2.25\times $ computation reduction can be achieved. Such an efficient RPC allows us to explore more powerful representational capacity of BNNs under a given computation complexity budget. Moreover, we propose to use a balanced activation (BA) to adjust the distribution of the scaled activations after binarization, which enables significant performance improvement of BNNs. After integrating RPC and BA, the proposed network, dubbed as RB-Net, strikes a good trade-off between accuracy and efficiency, achieving superior performance with lower computational cost against the state-of-the-art BNN methods. Specifically, our RB-Net achieves 66.8% Top-1 accuracy with ResNet-18 backbone on ImageNet, exceeding the state-of-the-art Real-to-Binary Net (65.4%) by 1.4% while achieving more than $3\times $ reduction (52M vs. 165M) in computational complexity.

Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. Typical MTL methods are jointly trained with the complete multitude of ground-truths for all tasks simultaneously. However, one single dataset may not contain the annotations for each task of interest. To address this issue, we propose the Semi-supervised Multi-Task Learning (SemiMTL) method to leverage the available supervisory signals from different datasets, particularly for semantic segmentation and depth estimation tasks. To this end, we design an adversarial learning scheme in our semi-supervised training by leveraging unlabeled data to optimize all the task branches simultaneously and accomplish all tasks across datasets with partial annotations. We further present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets. Finally, we demonstrate the effectiveness of the proposed method to learn across different datasets on challenging street view and remote sensing benchmarks.