28 January 2025 in IEEE Transactions on Emerging Topics in Computational Intelligence
Change detection (CD) is a crucial task in various real-world applications, aiming to identify regions of change between two images captured at different times. However, existing approaches mainly focus on designing advanced network architectures that map feature differences to change maps, overlooking the impact of feature difference quality. In this paper, we approach CD from a different perspective by exploring how to optimize feature differences to effectively highlight changes and suppress background regions. To achieve this, we propose a novel module called the iterative difference-enhanced transformers (IDET). IDET consists of three transformers: two for extracting long-range information from the bi-temporal images, and one for enhancing the feature difference. Unlike previous transformers, the third transformer utilizes the outputs of the first two transformers to guide iterative and dynamic enhancement of the feature difference. To further enhance refinement, we introduce the multi-scale IDET-based change detection approach, which utilizes multi-scale representations of the images to refine the feature difference at multiple scales. Additionally, we propose a coarse-to-fine fusion strategy to combine all refinements. Our final CD method surpasses nine state-of-the-art methods on six large-scale datasets across different application scenarios. This highlights the significance of feature difference enhancement and demonstrates the effectiveness of IDET. Furthermore, we demonstrate that our IDET can be seamlessly integrated into other existing CD methods, resulting in a substantial improvement in detection accuracy.
11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.
20 December 2024 in arXiv
Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.