Artificial intelligence (AI), owing to recent breakthroughs in deep learning, has revolutionized applications and services in almost all technology domains including aerospace. AI and deep learning rely on huge amounts of training data that are mostly generated at the network edge by Internet of Things (IoT) devices and sensors. Bringing the sensed data from the edge of a distributed network to a centralized cloud is often infeasible because of the massive data volume, limited network bandwidth, and real-time application constraints. Consequently, there is a desire to push AI frontiers to the network edge toward utilizing the enormous amount of data generated by IoT devices near the data source. The merger of edge computing and AI has engendered a new discipline, that is, AI at the edge or edge intelligence. To help AI make sense of gigantic data at the network edge, data fusion is of paramount significance and goes hand in hand with AI. This article focuses on data fusion and AI at the edge. In this article, we propose a framework for data fusion and AI processing at the edge. We then provide a comparative discussion of different data fusion and AI models and architectures. We discuss multiple levels of fusion and different types of AI, and how different types of AI align with different levels of fusion. We then highlight the benefits of combining data fusion with AI at the edge. The methods of AI and data fusion at the edge detailed in this article are applicable to many application domains including aerospace systems. We evaluate the effectiveness of combined data fusion and AI at the edge using convolutional neural network models and multiple hardware platforms suitable for edge computing. Experimental results reveal that combining AI with data fusion can impart a speedup of 9.8× while reducing energy consumption up to 88.5% over AI without data fusion. Furthermore, results demonstrate that data fusion either maintains or improves the accuracy of AI in most cases. For our experiments, data fusion imparts a maximum improvement of 15.8% in accuracy to AI.

Urban surveillance , of which airborne urban surveillance is a vital constituent, provides situational awareness (SA) and timely response to emergencies. The significance and scope of urban surveillance has increased manyfold in recent years due to the proliferation of unmanned aerial vehicles (UAVs), Internet of things (IoTs), and multitude of sensors. In this article, we propose FogSurv—a fog-assisted surveillance architecture and framework leveraging artificial intelligence (AI) and information/data fusion for enabling real-time SA and monitoring. We also propose an AI- and data-driven information fusion model for FogSurv to help provide (near) real-time SA, threat assessment, and automated decision-making. We further present a latency model for AI and information fusion processing in FogSurv. We then discuss several use cases of FogSurv that can have a huge impact on multifarious fronts of national significance ranging from safeguarding national security to monitoring of critical infrastructures. We conduct an extensive set of experiments to demonstrate that FogSurv using AI and data fusion help provide near real-time inferences and SA. Experimental results demonstrate that FogSurv provides a latency improvement of 37% on average over cloud architectures for the selected benchmarks. Results further indicate that combining AI with data fusion as in FogSurv can provide a speedup of up to $9.8\times $ over AI without data fusion while also maintaining or improving the inference accuracy. Additionally, results show that AI combined with fusion of different image modalities obtained through UAVs in FogSurv results in improved average precision of target detection for surveillance as compared to AI without data fusion for different target scales and environment complexity.

Off-the-shelf embedded systems have adopted heterogeneous multi-core processors which have high-performance big cores and low-power small cores. Though there are two different types of cores in heterogeneous multi-core processors, conventional DVFS (Dynamic Voltage and Frequency Scaling)-based DTM (Dynamic Thermal Management) techniques do not utilize the different types of cores to cool down hot cores. Rather, they primarily reduce the voltage and frequency of the hot cores, leading to performance degradation. In this paper, we propose a novel adaptive DTM framework for heterogeneous multi-core processors, which utilizes the big and small cores to prevent performance degradation. Our proposed framework exploits two migration-based DTM techniques: 1) a technique (denoted as $\text{Migration}_{\text{big}\longleftrightarrow \text{big}}$ }) that migrates applications from hot big cores (big cores whose temperature is above a pre-defined threshold) to cold big cores (big cores whose temperature is below the threshold) and 2) a technique (denoted as $\text{Migration}_{\text{big}\longleftrightarrow \text{small}}$ ) that migrates all applications from the big cores to the small cores. In case of thermal emergency of the big cores, our proposed framework checks the number of cold big cores. When there exist available cold big cores, our proposed framework employs $\text{Migration}_{\text{big}\longleftrightarrow \text{big}}$ to cool down the hot big cores while not reducing the big core frequency. On the other hand, when there does not exist any available cold big core, our proposed framework employs one between $\text{Migration}_{\text{big}\longleftrightarrow \text{small}}$ and a DVFS-based DTM technique, which is expected to result in better performance. In our experiments on an embedded development board, our proposed framework improves the average performance by 8.9%, compared to ARM's DVFS-based IPA (Intelligent Power Allocation), satisfying thermal constraints. Our framework also improves the average performance by 10.4%, compared to a state-of-the-art predictive DVFS-based DTM technique.

To improve mobile experience of users, recent mobile devices have adopted powerful processing units (CPUs and GPUs). Unfortunately, the processing units often consume a considerable amount of energy, which in turn shortens battery life of mobile devices. For energy reduction of the processing units, mobile devices adopt energy management techniques based on software, especially OS (Operating Systems), as well as hardware. In this survey paper, we summarize recent OS-level energy management techniques for mobile processing units. We categorize the energy management techniques into three parts, according to main operations of the summarized techniques: 1) techniques adjusting power states of processing units, 2) techniques exploiting other computing resources, and 3) techniques considering interactions between displays and processing units. We believe this comprehensive survey paper will be a useful guideline for understanding recent OS-level energy management techniques and developing more advanced OS-level techniques for energy-efficient mobile processing units.

Convolutional neural networks (CNNs) are widely deployed for many artificial intelligence (AI) applications, such as object detection and image classification. Due to the burgeoning revolution in edge AI, CNN hardware accelerators are also being employed in resource-constrained edge devices for achieving better performance and energy efficiency at the edge. Although CNN accelerators enable fast and energy-efficient CNN inference at the edge, the remaining hardware resources on the edge devices except for the CNN accelerator remain idle, which could otherwise be utilized for attaining even better performance and energy efficiency for CNN inferences. In this paper, we propose a CPU-accelerator co-scheduling technique for convolution (CONV) layer operations of CNN inferences in resource-constrained edge devices. Our proposed co-scheduling technique exploits an inherent parallelism in CNN output channels, that is, the operations for generating different output channels in a CONV layer can be executed in parallel. For load balancing between the CPU and the CNN accelerator, we also propose a simple, yet accurate latency model for CONV layer operations in the CPU and the accelerator. Based on the latency estimation of CONV layer operations provided by our proposed model, we distribute the tasks to the CPU and the CNN accelerator in a load-balance manner to minimize the idle period during the CONV layer operations in both the CPU and the CNN accelerator. We implement our proposed hardware/software (HW/SW) co-scheduling technique in various field-programmable gate array system-on-chip (FPGA-SoC) platforms as a proof-of-concept. Experimental results indicate that our proposed co-scheduling technique improves system performance by 1.18×– 2.00× with energy reduction of 14.9% – 49.7% as compared to the accelerator-only execution.

Many recent research efforts have exploited data sparsity for the acceleration of convolutional neural network (CNN) inferences. However, the effects of data transfer between main memory and the CNN accelerator have been largely overlooked. In this work, the authors propose a CNN acceleration technique that leverages hardware/software co‐design and exploits the sparsity in input feature maps (IFMs). On the software side, the authors' technique employs a novel lossless compression scheme for IFMs, which are sent to the hardware accelerator via direct memory access. On the hardware side, the authors' technique uses a CNN inference accelerator that performs convolutional layer operations with their compressed data format. With several design optimization techniques, the authors have implemented their technique in a field‐programmable gate array (FPGA) system‐on‐chip platform and evaluated their technique for six different convolutional layers in SqueezeNet. Results reveal that the authors' technique improves the performance by 1.1×–22.6× while reducing energy consumption by 47.7%–97.4% as compared to the CPU‐based execution. Furthermore, results indicate that the IFM size and transfer latency are reduced by 34.0%–85.2% and 4.4%–75.7%, respectively, compared to the case without data compression. In addition, the authors' hardware accelerator shows better performance per hardware resource with less than or comparable power consumption to the state‐of‐the‐art FPGA‐based designs.