In recent years, public security incidents caused by deepfake technology have occurred frequently around the world, which makes an efficient and accurate deepfake detection model crucial. The existing advanced methods use the manipulation features in the image to realize the binary classification of real and fake images by training complex neural network models. However, these models rely on a single manipulation feature, and the detection accuracy of these methods will be greatly reduced when the forgery technology or image quality of the training dataset and the validation dataset are different. Inspired by the existing work, we propose a two-stream collaborative learning framework that combines spatial texture differences and frequency information. The average difference convolution (ADC) is designed to extract the spatial texture difference information of the image, and the gray image frequency-aware decomposition (GFAD) is used to extract the artifact information of the image in the frequency domain. At the same time, the ViT idea is combined with cross-attention mechanism for feature fusion to comprehensively mine forged features in forged images. Experimental results show that the proposed model has good detection effects on three benchmark datasets. In terms of cross-dataset evaluation, the AUC on Celeb-DF dataset reaches 82.86%, which is better than the existing advanced methods.
Convolutional neural networks (CNNs) have been extensively used in numerous remote sensing image detection tasks owing to their exceptional performance. Nevertheless, CNNs are often vulnerable to adversarial examples, limiting the uses in different safety-critical scenarios. Recently, how to efficiently detect adversarial examples and improve the robustness of CNNs has drawn considerable focus. The existing adversarial example detection methods require modifying CNNs, which not only affects the model performance but also greatly enhances training cost. With the purpose of solving these problems, this study proposes a detection algorithm for adversarial examples that does not need modification of the CNN models and can simultaneously retain the classification accuracy of normal examples. Specifically, we design a method to detect adversarial examples using frequency domain reconstruction. After converting the input adversarial examples into the frequency domain by Fourier transform, the adversarial disturbance from adversarial attacks can be eliminated by modifying the frequency of the example. The inverse Fourier transform is then used to maximize the recovery of the original example. Firstly, we train a CNN to reconstruct input examples. Then, we insert Fourier transform, convolution operation, and inverse Fourier transform into the features of the input examples to automatically filter out adversarial frequencies. We refer to our proposed method as FDR (frequency domain reconstruction), which removes adversarial interference by converting input samples into frequency and reconstructing them back into the spatial domain to restore the image. In addition, we also introduce gradient masking into the proposed FDR method to enhance the detection accuracy of the model for complex adversarial examples. We conduct extensive experiments on five mainstream adversarial attacks on three benchmark datasets, and the experimental results show that FDR can outperform state-of-the-art solutions in detecting adversarial examples. Additionally, FDR does not require any modifications to the detector and can be integrated with other adversarial example detection methods to be installed in sensing devices to ensure detection safety.
6 August 2022 in Multimedia Tools and Applications
Image inpainting aims to repair the damaged region according to the known content in the damaged image. Recently, image inpainting methods have poor effects on high‐resolution damaged images, and the research on the inpainting of large‐area damaged images is limited. Therefore, this paper proposes an image inpainting method based on Generative Adversarial Networks (GAN) inversion and autoencoder. This work consists of two phases: first, the authors design an autoencoder‐based GAN, which learns the mapping from noise to low‐dimensional feature maps by training a generator, and then converts the generated feature maps into high‐resolution images. Thus, the difficulty of learning the mapping relationship is reduced. Second, the authors adopt the learning‐based GAN inversion to infer the closest latent code. The trained GAN is then used to reconstruct the complete image. Finally, the authors compare their method with other classical methods on the CelebAMask‐HQ, Flickr‐Faces‐HQ, and ImageNet datasets. According to the quantitative comparison, when the mask range is large, in other words, when the image has a large area of damage, the authors’ method is superior to the comparison methods. According to the qualitative comparison, the structure of the high‐resolution image inpainted by the authors’ method is more reasonable and the texture details are more realistic.
31 May 2023 in KSII Transactions on Internet and Information Systems
The primary challenge of the multimodal sentiment analysis (MSA) task is the modal fusion, and the lack of modalities may exist in the fusion process, which leads to poor prediction results. Most of the previous research on multimodal fusion is single‐stage fusion, disregarding how various modality subsets interact, as well as rarely considering relative position relationship of modality sequences, causing the fragmentation of context info. Considering the aforementioned issues, this study introduces an MSA method based on the multi‐stage graph fusion network (MSGFN) under random missing modality conditions to improve the robustness of the model to MSA under the random missing modality conditions. Firstly, for each modality, its inter‐modal and intra‐modal multi‐head attention are used to learn robust representation of the modality sequence. Meanwhile, the relative position encoding (RPE) is introduced into mechanism of attention that enables model to perceive and learn the relative position before and after the modality sequence when calculating attention, thereby better understanding the contextual info of the sequence. After that, the transformer encoder receives the learned modality features and uses the pre‐trained model to supervise the reconstruction of the missing modality information. Finally, the feature representations of different modalities have effectively fused using multi‐stage graph fusion network, and the output is used for the ultimate sentiment classification. Wide experiments are conducted on two publicly available datasets, CMU‐MOSI and IEMOCAP, and the findings indicate that the proposed method can better handle the challenges caused by modality fusion and modality missing compared with several baseline methods.