In recent years, convolutional neural network (CNN)-based methods have been widely used for optical remote sensing object detection and have shown excellent performance. Some aerospace systems, such as satellites or aircrafts, need to adopt these methods to observe objects on the ground. Due to the limited budget of the logical resources and power consumption in these systems, an embedded device is a good choice to implement the CNN-based methods. However, it is still a challenge to strike a balance between performance and power consumption. In this paper, we propose an efficient hardware-implementation method for optical remote sensing object detection. Firstly, we optimize the CNN-based model for hardware implementation, which establishes a foundation for efficiently mapping the network on a field-programmable gate array (FPGA). In addition, we propose a hardware architecture for the CNN-based remote sensing object detection model. In this architecture, a general processing engine (PE) is proposed to implement multiple types of convolutions in the network using the uniform module. An efficient data storage and access scheme is also proposed, and it achieves low-latency calculations and a high memory bandwidth utilization rate. Finally, we deployed the improved YOLOv2 network on a Xilinx ZYNQ xc7z035 FPGA to evaluate the performance of our design. The experimental results show that the performance of our implementation on an FPGA is only 0.18% lower than that on a graphics processing unit (GPU) in mean average precision (mAP). Under a 200 MHz working frequency, our design achieves a throughput of 111.5 giga-operations per second (GOP/s) with a 5.96 W on-chip power consumption. Comparison with the related works demonstrates that the proposed design has obvious advantages in terms of energy efficiency and that it is suitable for deployment on embedded devices.

Scene classification has become an active research area in remote sensing (RS) image interpretation. Recently, Transformer-based methods have shown great potential in modeling global semantic information and have been exploited in RS scene classification. In this letter, we propose a multi-level fusion Swin Transformer (MFST), which integrates a multi-level feature merging (MFM) module and an adaptive feature compression (AFC) module to further boost the performance for RS scene classification. The MFM module narrows the semantic gaps in multi-level features via patch merging in lower-level feature maps and lateral connections in the top-down pathway. The AFC module makes multi-level features have smaller dimensions and more coherent semantic information by adaptive channel reduction. We evaluate the proposed network on the aerial image dataset (AID) and NWPU-RESISC45 (NWPU) datasets, and the classification results reveal that the proposed network outperforms several state-of-the-art (SOTA) methods.

Deep learning (DL) has been introduced to change detection (CD) due to its powerful feature representation and robust generalization abilities. However, the application of large DL models has high computational complexity and massive storage requirements for achieving good performance. For disaster emergency response and other applications with high timeliness requirements, it is difficult to deploy large DL models on spaceborne edge devices with limited resources to achieve on-board CD processing. To address this limitation, a novel CD based on knowledge distillation (CDKD) method that combines prototypical contrastive (PC) distillation and channel-spatial-normalized (CSN) distillation is proposed. PC distillation represents the feature distribution by calculating the differences between the similarities of pixel features and their positive and negative prototypes, and improves the student model's detection ability in changed regions that have similar features to the background by mimicking the relative feature distribution. CSN distillation combines two distillation paradigms, channel normalization and spatial normalization, and guides the student model to comprehensively learn the knowledge contained in the output probabilities of the teacher model to accurately identify changed regions with complex shapes. The effectiveness and reliability of the proposed CDKD method are verified on three public remote sensing CD datasets, and extensive experiments and analyses show that the proposed CDKD method can be used to train lightweight models with comparable performance to that of large models.

In recent years, convolutional neural network (CNN)-based algorithms have been widely used in remote sensing image processing and show tremendous performance in a variety of application fields. However, large amounts of data and intensive computations make the deployment of CNN-based algorithms a challenging problem, especially for the spaceborne scenario where resources and power consumption are limited. To tackle this problem, this paper proposes an automatic CNN deployment solution on resource-limited field-programmable gate arrays (FPGAs) for spaceborne remote sensing applications. Firstly, a series of hardware-oriented optimization methods are proposed to reduce the complexity of the CNNs. Secondly, a hardware accelerator is designed. In this accelerator, a reconfigurable processing engine array with efficient convolutional computation architecture is used to accelerate CNN-based algorithms. Thirdly, to bridge the optimized CNNs and hardware accelerator, a compilation toolchain is introduced into the deployment solution. Through the automatic conversion from CNN models to hardware instructions, various networks can be deployed on hardware in real-time. Finally, we deployed an improved VGG16 network and an improved YOLOv2 network on Xilinx AC701 to evaluate the effectiveness of the proposed deployment solution. The experiments show that with only 3.407 W power consumption and 94 DSP consumption, our solution achieves 23.06 giga operations per second (GOPS) throughput in the improved VGG16 and 22.17 GOPS throughput in the improved YOLOv2. Compared to the related works, the DSP efficiency of our solution is improved by 1.3–2.7×.

Directly performing remote sensing scene classification (RSSC) on satellites can effectively relieve pressures on data downlinks. However, existing convolutional neural network (CNN)-based methods with dense multiplication operations require a vast resource overhead and can hardly be deployed on resource-limited space platforms. In this article, an all adder neural network (A ² NN) with a generative-based hybrid knowledge distillation (GHKD) training strategy is proposed to solve this problem. In contrast to traditional CNNs, the A ² NN discards all multiplication operations in the convolutional layers of the network and is instead completely composed of adder kernels; thus, this network has lower resource overheads than existing CNNs in hardware deployment. Compared with CNNs, the A ² NN has a considerably lower RSSC accuracy; fortunately, this can be mitigated with the proposed GHKD training strategy. This strategy combines a hybrid knowledge distillation method and a knowledge matching-based generative learning method. The hybrid knowledge distillation method utilizes the weight distribution knowledge in the adder neural network (ANN) to guide the high-performance classification knowledge in the CNN to be smoothly transferred to the A ² NN, which consists entirely of adder kernels and has a considerably different structure than the CNN. The knowledge matching-based generative learning method generates effective samples by matching the knowledge of the feature distribution of the real samples in the knowledge sources to augment the training database for hybrid knowledge distillation. Extensive experiments and analyses on six public RSSC datasets show that the proposed GHKD training strategy significantly improves the RSSC performance of the proposed A ² NN, achieving comparable performance to CNNs on most datasets.

Neural networks have been extensively applied across various Internet of Things (IoT) applications, such as drone- and satellite-based remote sensing and autonomous driving. With the increasing resolution and amount of data captured by sensors, the demand for real-time response in IoT applications is markedly increasing. However, it is difficult for existing convolutional neural network (CNN) accelerators for IoT applications on field-programmable gate array (FPGA) platforms to achieve high throughput because of the inherent dense multiplication operations of CNNs, memory bandwidth limitations and inefficient mapping mechanisms. In this article, a high-throughput and energy-efficient all adder neural network (A2NN) accelerator for IoT applications on FPGA platform is proposed to solve this problem. First, a series of hardware-oriented algorithm optimization methods are proposed to simplify the processing flow of A2NN and further minimize its deployment overhead. Second, a novel hardware architecture based on the idea of near-memory computation (NMC) is proposed to eliminate off-chip memory access completely and accelerate the reconstructed A2NN in the pipeline. Third, a set of quantitative analysis methods for the proposed accelerator is presented to balance throughput and energy consumption, allowing the accelerator to adapt to the varying demands of different IoT application scenarios. Extensive experimental results on the AMD-Xilinx VC709 board demonstrate that the proposed accelerator achieves state-of-the-art performance in terms of throughput, energy efficiency, and throughput efficiency. Moreover, experiments on the AMD-Xilinx KV260 board highlight the architecture’s exceptional scalability and energy efficiency, enabling a balance between speed and power consumption tailored to the specific requirements of IoT application scenarios.