Spiking Neural Networks (SNNs) have shown great promise in processing spatio-temporal information compared to Artificial Neural Networks (ANNs). However, there remains a performance gap between SNNs and ANNs, which impedes the practical application of SNNs. With intrinsic event-triggered property and temporal dynamics, SNNs have the potential to effectively extract spatio-temporal features from event streams. To leverage the temporal potential of SNNs, we propose a self-attention-based temporal-channel joint attention SNN (STCA-SNN) with end-to-end training, which infers attention weights along both temporal and channel dimensions concurrently. It models global temporal and channel information correlations with self-attention, enabling the network to learn ‘what’ and ‘when’ to attend simultaneously. Our experimental results show that STCA-SNNs achieve better performance on N-MNIST (99.67%), CIFAR10-DVS (81.6%), and N-Caltech 101 (80.88%) compared with the state-of-the-art SNNs. Meanwhile, our ablation study demonstrates that STCA-SNNs improve the accuracy of event stream classification tasks.
20 May 2024 in IEEE Geoscience and Remote Sensing Letters
Recent years have witnessed many research efforts for addressing the challenging difficulties for salient object detection in optical remote sensing images (ORSI-SOD). However, due to irregular imaging mechanism and complex scene properties, existing models suffer from a disproportion of performance and efficiency, yet remain much exploration room. We propose the parallel refinement network with group feature learning (PRNet) framework for ORSI-SOD. Specifically, we propose a parallel refinement module with three parallel and same blocks in which two proposed different branches aggregating features in a group feature learning strategy, one for fine-grained features aggregation from up to down, another for reversal features aggregation from down to up. Benefiting from the novel and efficient framework, PRNet outperforms over 15 state-of-the-art models on three public benchmark datasets (an average S-measure, mean E-measure, and MAE of 91.95%, 96.85% and 1.25%), runs up to real-time detection performance (36 FPS) on a single NIVIDIA 2080Ti GPU, achieving a better trade-off between performance and efficiency among deep comparison models. Project will be available at https://github.com/BIT-GuSY/PRNet-ORSI.
Although the performance of unmanned aerial vehicle (UAV) tracking has benefited from the successful application of discriminative correlation filters (DCF) and convolutional neural networks (CNNs), UAV tracking under occlusion and deformation remains a challenge. The main dilemma is that challenging scenes, such as occlusion or deformation, are very complex and changeable, making it difficult to obtain training data covering all situations, resulting in trained networks that may be confused by new contexts that differ from historical information. Data-driven strategies are the main direction of current solutions, but gathering large-scale datasets with object instances under various occlusion and deformation conditions is difficult and lacks diversity. This paper proposes an attention-based mask generation network (AMGN) for UAV-specific tracking, which combines the attention mechanism and adversarial learning to improve the tracker’s ability to handle occlusion and deformation. After the base CNN extracts the deep features of the candidate region, a series of masks are determined by the spatial attention module and sent to the generator, and the generator discards some features according to these masks to simulate the occlusion and deformation of the object, producing more hard positive samples. The discriminator seeks to distinguish these hard positive samples while guiding mask generation. Such adversarial learning can effectively complement occluded and deformable positive samples in the feature space, allowing to capture more robust features to distinguish objects from backgrounds. Comparative experiments show that our AMGN-based tracker achieves the highest area under curve (AUC) of 0.490 and 0.349, and the highest precision scores of 0.742 and 0.662, on the UAV123 tracking benchmark with partial and full occlusion attributes, respectively. It also achieves the highest AUC of 0.555 and the highest precision score of 0.797 on the DTB70 tracking benchmark with the deformation attribute. On the UAVDT tracking benchmark with the large occlusion attribute, it achieves the highest AUC of 0.407 and the highest precision score of 0.582.
Drone object detection faces numerous challenges such as dense clusters with overlapping, scale diversity, and long-tail distributions. Utilizing tiling inference through uniform sliding window is an effective way of enlarging tiny objects and meanwhile efficient for real-world applications. However, merely partitioning input images may result in heavy truncation and an unexpected performance drop in large objects. Therefore, in this work, we strive to develop an improved tiling detection framework with both competitive performance and high efficiency. First, we formulate the tiling inference and training pipeline with a mixed data strategy. To avoid truncation and handle objects at all scales, we simultaneously perform global detection on the original image and local detection on corresponding sub-patches, employing appropriate patch settings. Correspondingly, the training data includes both original images and the patches generated by random online anchor-cropping, which can ensure the effectiveness of patches and enrich the image scenarios. Furthermore, a scale filtering mechanism is applied to assign objects at diverse scales to global and local detection tasks to keep the scale invariance of a detector and obtain optimal fused predictions. As most of the additional operations are performed in parallel, the tiling inference remains highly efficient. Additionally, we devise two augmentations customized for tiling detection to effectively increase valid annotations, which can generate more challenging drone scenarios and simulate the practical cluster with overlapping, especially for rare categories. Comprehensive experiments on both public drone benchmarks and our customized real-world images demonstrate that, in comparison to other drone detection frameworks, the proposed tiling framework can significantly improve the performance of general detectors in drone scenarios with lower additional computational costs.
7 May 2024 in IEEE Transactions on Geoscience and Remote Sensing
The scarce onboard computational resources and real-time demand restrict the deployment of aerial trackers with sophisticated structures and customized operators. Meanwhile, aerial trackers need updating modules to adapt to continuous appearance variations in real-world long-term (LT) scenarios. However, frequent updating will introduce noisy templates and lead to tracking drifts and efficiency drops. Therefore, in this work, we develop a lightweight and highly efficient Siamese tracker for LT onboard aerial tracking applications. First, we build a compact, plain, and deployment-friendly Siamese network based on re-parameterization (Rep) as the baseline short-term (ST) tracker. Furthermore, we propose a tracking-specific decoupled knowledge distillation (KD) guided by strict teachers to unleash the appearance representation potential of the feature extractor without extra inference cost. Specifically, before distillation, the teacher conducts qualification verification to avoid misguiding the student. Then, hard negative background regions are mined and decoupled with the target region, encouraging the student to focus more on similar distractors and informative areas. Finally, to realize efficient and high-confidence LT tracking, we design two extensions and incorporate them into the boosted ST tracker: an initial-template-driven template updater with a corresponding pair-generating strategy to alleviate appearance pollution, and a confidence estimating branch to determine whether to update. Extensive results on large-scale drone benchmarks indicate that our proposed tracker significantly outperforms state-of-the-art (SOTA) aerial trackers. Real-world tests on our customized drone-captured LT dataset also validate its favorable practicability with a real-time speed of 44 fps on the Lynix KA200 chip.
1 April 2016 in International Journal of Engineering and Technology