2 March 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Image manipulation localization (IML), which seeks to accurately segment tampered regions that are artfully fastened into a normal image, is a fundamental yet challenging computer vision task. Despite that impressive results have been achieved by some progressive deep learning methods, they usually fail in capturing the subtle manipulation artifacts at different object scales, which are not competent to generate a perfect segmentation mask with complete and fine object structures. Besides, the problem of coarse boundaries also occurs frequently. To this end, in this paper, we propose a Transformer-Auxiliary by operator-induced neural Network (TANet) to localize forged regions for IML. Specifically, a stacked multi-scale transformer (SMT) branch is first introduced as a compensation for feature representations of the mainstream convolutional neural network branch. SMT can detect structured abnormalities of the input image at multi-levels by operating on patches of different sizes. Then TANet explicitly exploits an operator induction module (OIM) to excavate valuable and manipulated region-related boundary semantics to guide the representative learning of the mainstream branch. The OIM encourages the network to generate features that highlight object structure, thereby promoting precise boundary localization of forged regions. We conduct extensive experiments on various datasets and settings to validate the effectiveness of TANet. Results show that TANet outperforms the state-of-the-art methods by a large margin under widely-used evaluation metrics.
30 November 2018 in IEEE Access
In multimedia forensics, many efforts have been made to detect whether an image is pristine or manipulated with high enough accuracies based on specially designed features and classifiers in the past decade. However, the important task for localizing the tampering regions in a fake image still faces more challenges compared to manipulation detection and relatively few algorithms attempt to tackle it. With this in mind, a technique that utilizes the dual domain-based convolutional neural networks (D-CNNs) taking different kinds of input into consideration is proposed in this paper. In the proposed framework, two sub-networks, named spatial-domain CNN model (Sub-SCNN) and frequency domain-based CNN model (Sub-FCNN), are designed and trained respectively. With the well-trained parameters, a transfer policy is applied to the training process of D-CNN. While CNNs are capable of learning classification features directly from data, in their standard form they tend to learn features related to the image’s content. To overcome this issue in image forensics tasks, a new image pre-processing layer is proposed to jointly suppress image’s content and adaptively learn manipulation detection and localization features. After investigating the properties of datasets, two post-processing operations are finally proposed and compared to obtain the final results of the pixel-wise manipulation region localization. The D-CNNs is trained and validated using 75 percent of images in the CASIA v2.0 and tested using the remaining images in CASIA v2.0, all images in Columbia Uncompressed and Carvalho datasets. The extensive experiments show that the proposed post-processing operations optimize the final tamper probability map, and our framework with the combination of Sub-SCNN and Sub-FCNN significantly outperforms the state-of-art techniques with the best F1 scores on the datasets.
25 September 2020 in IEEE Signal Processing Letters
This letter focuses on image manipulation detection which aims to recognize the manipulated regions under the contextual semantic information. Existing approaches usually overlook the semantic discrepancy between different levels of feature maps, and directly fuse (e.g., addition, or concatenation) them for detection. In this letter, we argue that the semantic gap is the main reason for the low effectiveness of feature fusion in manipulation predictions. To address this problem, we propose a Global Semantic Consistency Network (GSCNet) for image manipulation detection, which is based on an encoder-decoder structure. Specifically, to make GSCNet include more global texture information which has been empirically confirmed to be beneficial to manipulation detection, gram block is first deployed on each level of feature maps in the encoding stage. Based on that, bi-directional convolutional LSTM is further implemented on the decoding stage, such that feature maps of the same level have semantic consistency. Experimental results on NIST16, and CASIA v1.0 declare that GSCNet can accurately locate the manipulated regions. Furthermore, compared to the existing models, GSCNet can achieve new state-of-the-art results.
14 January 2022 in International Journal of Intelligent Systems
26 June 2022 in Multimedia Systems
12 March 2025 in ACM Transactions on Multimedia Computing, Communications, and Applications
With the continuous development of deep counterfeiting technology, the information security in our daily life is under serious threat. While existing face forgery detection methods exhibit impressive accuracy when applied to datasets such as FaceForensics++ and Celeb-DF, they falter significantly when confronted with out-of-domain scenarios. This causes specialization of learned representations to known forgery patterns presented in the training set, rendering it difficult to detect forgeries with unknown patterns. To address this challenge, we propose a novel end-to-end Face Reconstruction-Based Generalized Deepfake Detection (FRG2D) model with Residual Outlook Attention (ROA) , which emphasizes the robust visual representations of genuine faces and discerns the subtle differences between authentic and manipulated facial images. Our methodology entails reconstructing authentic face images using an encoder–decoder architecture based on U-net, facilitating a deeper understanding of disparities between genuine and manipulated facial images. Furthermore, we integrate the convolutional block attention module (CBAM) and channel attention block (CAB) to selectively focus the network’s attention on salient features within real face images. Furthermore, we employ ROA to guide the network’s focus towards precise features within manipulated facial images. Simultaneously, the computed reconstruction differences obtained through ROA serves as the ultimate representation fed into the classifier for face forgery detection. Both the reconstruction and classification learning processes are optimized end-to-end. Through extensive experimentation, our model demonstrated a substantial improvement in deepfake detection across unknown domains, while maintaining a high accuracy within the known domain.