8 December 2023 in International Journal of Computer Vision
30 April 2024 in IEEE Transactions on Multimedia
Recently, the source-free domain adaptation (SFDA) problem has attracted much attention, where the pre-trained model for the source domain is adapted to the target domain in the absence of source data. However, due to domain shift, the negative alignment usually exists between samples from the same class, which may lower intra-class feature similarity. To address this issue, we present a self-supervised representation learning strategy for SFDA, named as neighborhood-aware mutual information (NAMI), which maximizes the mutual information (MI) between the representations of target samples and their corresponding neighbors. Moreover, we theoretically demonstrate that NAMI can be decomposed into a weighted sum of local MI, which suggests that the weighted terms can better estimate NAMI. To this end, we introduce neighborhood consensus score over the set of weakly and strongly augmented views and point-wise density based on neighborhood, both of which determine the weights of local MI for NAMI by leveraging the neighborhood information of samples. The proposed method can significantly handle domain shift and adaptively reduce the noise in the neighborhood of each target sample. In combination with the consistency loss over views, NAMI leads to consistent improvement over existing state-of-the-art methods on three popular SFDA benchmarks.
3 January 2024 in IEEE Transactions on Cognitive and Developmental Systems
Unsupervised person re-identification (re-ID) aims to perform person retrieval in an unsupervised manner across multiple cameras. Most recent works usually adopted contrastive learning with clustering strategy to guide learning process. However, the domain gap between cameras easily causes the high intra-class variance and prohibits an effective contrastive learning. To overcome this problem, we propose dual-graph contrastive learning (Dual-GCL) to fully utilize the structures between the instances as well as the relationship between the instances and the centroids generated by the clustering strategy. Firstly, a dual-graph is constructed by utilizing the affinities of instances in the feature space and the pseudo-label space derived from clustering strategy with the node-to-node contrastive learning. Then, to capture the relations between the dual-graph and the centroids, we perform dual-graph contrastive learning at both the node level and the super-node level. The contrastive learning based on the two levels can encode multi-level structural representations between the instances corresponding to the nodes in dual-graph and align the centroids between different cameras in the shared embedding space. Moreover, a camera-aware augmentation scheme that exploits the semantic relationship is proposed to alleviate the domain discrepancy between cameras. Experimental results demonstrate that our approach outperforms the state-of-the-art methods.
30 August 2021 in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Generative adversarial networks (GANs) are subject to catastrophic forgetting when learning stream of data. Inspired by the knowledge of neuroscience, this article develops a memory formation system (MFS) to establish memory for GANs. MFS is composed of three modules, including the identifier, weights upgrade (WU), and weights reactivate (WR). These modules simulate the function of encoding, consolidating, and retrieving in memory formation of human. Identifier creates indexes to label continuous tasks and these indexes are used as a cue when the corresponding tasks are recalled. WU and WR work as synaptic consolidation and system consolidation, respectively. In WU, a novel method, weight saliency measure (WSM) is proposed to measure the saliency of weight. Valuable weights are protected and invaluable weights are released for update when GAN is trained for the new task. In WR, traditional data-replay methods are improved by selecting ${k}$ representatives in each class and their feature vectors are restored. When pseudo data of this class are regenerated, only the samples with distance less than a threshold can be used in future training. Experimental results based on the testing of continual image generation and continual 3-D reconstruction show that MFS can establish a memory system to handle the catastrophic forgetting problem effectively.