8 August 2024 in IEEE Transactions on Image Processing
Multi-view clustering aims to learn discriminative representations from multi-view data. Although existing methods show impressive performance by leveraging contrastive learning to tackle the representation gap between every two views, they share the common limitation of not performing semantic alignment from a global perspective, resulting in the undermining of semantic patterns in multi-view data. This paper presents CSOT, namely Common Semantics via Optimal Transport, to boost contrastive multi-view clustering via semantic learning in a common space that integrates all views. Through optimal transport, the samples in multiple views are mapped to the joint clusters which represent the multi-view semantic patterns in the common space. With the semantic assignment derived from the optimal transport plan, we design a semantic learning module where the soft assignment vector works as a global supervision to enforce the model to learn consistent semantics among all views. Moreover, we propose a semantic-aware re-weighting strategy to treat samples differently according to their semantic significance, which improves the effectiveness of cross-view contrastive representation learning. Extensive experimental results demonstrate that CSOT achieves the state-of-the-art clustering performance.
16 February 2023 in IEEE Transactions on Intelligent Transportation Systems
Re-identification (re-ID) of maritime vessels plays an important role in marine surveillance, but remains highly unexplored due to the lack of large-scale annotated datasets. In vessel re-ID, contrastive methods are supposed to learn discriminative representation from unlabeled vessel images in an unsupervised manner. However, directly introducing classical instance-level contrastive methods to maritime vessel re-ID suffers from the difficulty of finding vessel images with the same pseudo label as positive images, which potentially leads to inefficient training and unsatisfactory performance. This paper proposes a simple but effective method to solve such a hard positive problem. Our method takes all images in an intra-batch cluster as positives and excludes them from the set of negative samples when computing instance-level contrastive loss. Based on this strategy, we construct a multi-level contrastive learning (MCL) framework for vessel re-ID trained with the specifically designed intra-batch cluster-level contrastive loss along with the instance-level one. Experiments on a newly proposed dataset consisting of $1,248$ vessel identities show that MCL achieves the state-of-the-art performance compared with other unsupervised methods.
25 January 2022 in ACM Transactions on Multimedia Computing, Communications, and Applications
To cope with the problem caused by inadequate training data, many person re-identification (re-id) methods exploit generative adversarial networks (GAN) for data augmentation, where the training of GAN is typically independent of that of the re-id model. The coupling relation between them that probably brings in a performance gain of re-id is thus ignored. In this work, we propose a general framework, namely JoT-GAN, to jointly train GAN and the re-id model. It can simultaneously achieve the optima of both the generator and the re-id model, where the training is guided by each other through a discriminator. The re-id model is boosted for two reasons: (1) the adversarial training encourages it to fool the discriminator, and (2) the generated samples augment the training data. Extensive results on benchmark datasets show that for the re-id model trained with the identification loss as well as the triplet loss, the proposed joint training framework outperforms existing methods with separate training and achieves state-of-the-art re-id performance.
4 December 2020 in IEEE Transactions on Multimedia
The task of spatial layout estimation of monocular image is to segment an RGB image of indoor scenes with semantic surface labels (i.e., ceiling, floor, front wall, left wall, and right wall). Most recent methods have to produce layout hypotheses based on the estimated edge map or semantic labels, and then rank the layout hypotheses. In this paper, we present an end-to-end framework that can directly output the layout type and keypoint coordinates (defined in the LSUN challenge). The proposed method takes advantage of transfer learning via learning on the fake samples, i.e., plenty of artificial {type, keypoints, edge map} triplets are generated to learn the mapping from edge maps to keypoint coordinates. Generative adversarial network (GAN) is implemented in this work for domain adaptation of the edge maps. Experimental results show that the proposed method can achieve state-of-the-art layout estimation performance on benchmark datasets.
3 January 2024 in IEEE Transactions on Cognitive and Developmental Systems
Unsupervised person re-identification (re-ID) aims to perform person retrieval in an unsupervised manner across multiple cameras. Most recent works usually adopted contrastive learning with clustering strategy to guide learning process. However, the domain gap between cameras easily causes the high intra-class variance and prohibits an effective contrastive learning. To overcome this problem, we propose dual-graph contrastive learning (Dual-GCL) to fully utilize the structures between the instances as well as the relationship between the instances and the centroids generated by the clustering strategy. Firstly, a dual-graph is constructed by utilizing the affinities of instances in the feature space and the pseudo-label space derived from clustering strategy with the node-to-node contrastive learning. Then, to capture the relations between the dual-graph and the centroids, we perform dual-graph contrastive learning at both the node level and the super-node level. The contrastive learning based on the two levels can encode multi-level structural representations between the instances corresponding to the nodes in dual-graph and align the centroids between different cameras in the shared embedding space. Moreover, a camera-aware augmentation scheme that exploits the semantic relationship is proposed to alleviate the domain discrepancy between cameras. Experimental results demonstrate that our approach outperforms the state-of-the-art methods.
5 June 2024 in IEEE Transactions on Intelligent Transportation Systems
Visual perception of ships has been attracting increasing attention in the fields of computer vision and ocean engineering. Despite the extensive work related to landmark detection of common objects, the role of landmarks in ship perception has been overlooked. In this paper, we aim to fill this gap by focusing on ship landmarks. Specifically, we give a comprehensive analysis of both the physical structure and deep features of ships, which finds that highlighted areas in feature maps correspond with structurally significant parts of ships. By summarizing the locations of such areas in ships, we define 20 ship landmarks and build the Ship Landmark Dataset (SLAD), the first ship dataset with landmark annotations. We also provide a benchmark for ship landmark detection by evaluating state-of-the-art landmark detection methods on the newly built SLAD. Moreover, we showcased several applications of ship landmarks, including ship recognition, ship image generation, key area detection for ships, and ship detection. Project web page: https://vsislab.github.io/Ships_VSIS/.