24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Due to the efficiency of integrating semantic consensus and complementary information across different views, multi-view classification methods have attracted much attention in recent years. However, multi-view data often suffers from both the miss of view features and insufficient label information, which significantly decrease the performance of traditional multi-view classification methods in practice. Learning for such simultaneous lack of feature and label is crucial but rarely studied. To tackle these problems, we propose a novel Deep Incomplete Multi-view Learning Network (DIMvLN) by incorporating graph networks and semi-supervised learning in this paper. Specifically, DIMvLN firstly designs the deep graph networks to effectively recover missing data with assigning pseudo-labels of large amounts of unlabeled instances and refine the incomplete feature information. Meanwhile, to enhance the label information, a novel pseudo-label generation strategy with the similarity constraints of unlabeled instances is proposed to exploit additional supervisory information and guide the completion module to preserve more semantic information of absent multi-view data. Besides, we design view-specific representation extractors with the autoencoder structure and contrastive loss to learn high-level semantic representations for each view, promote cross-view consistencies and augment the separability between different categories. Finally, extensive experimental results demonstrate the effectiveness of our DIMvLN, attaining noteworthy performance improvements compared to state-of-the-art competitors on several public benchmark datasets. Code will be available at GitHub.
16 August 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Scene graph generation (SGG) is one of the hottest topics in computer vision and has attracted many interests since it provides rich semantic information between objects. In practice, the SGG datasets are often dual imbalanced, presented as a large number of backgrounds and rarely few foregrounds, and highly skewed foreground relationships categories (i.e. the long-tailed distribution). How to tackle this dual imbalanced problem is crucial but rarely studied in literature. Existing methods only consider the long-tailed distribution of foregrounds classes and ignore the background-foreground imbalance in SGG, which results in a biased model and prevents it from being applied in the downstream tasks widely. To reduce its side effect and make the contributions of different categories equally, we propose a novel debiased SGG method (named DSDI) by incorporating biased resistance loss and causal intervention tree. We first deeply analyze the potential causes of dual imbalanced problem in SGG. Then, to learn more discriminate representation of the foreground by expanding the foreground features space, the biased resistance loss decouples the background classification from foreground relationship recognition. Meanwhile, a causal graph of content and context is designed to remove the context bias and learn unbiased relationship features via casual intervention tree. Extensive experimental results on two extremely imbalanced datasets: VG150 and VrR-VG, demonstrate our DSDI outperforms other state-of-the-art methods. All our models will be available in https://github.com/zhouhao0515/unbiasedSGG-DSDI.
10 April 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Scene graph generation (SGG) establishes a structured representation between multiple objects by exploring their relationship for visual perception and reasoning tasks. Existing SGG methods often fit the relationships’ distribution by introducing language prior or statistical knowledge. However, the relationships should be the semantic reflection of the interaction between objects, rather than the statistical dependency between their categories. To solve this problem, we propose a novel Causal Features Enhancement Network (CFEN) to mine the essential semantic features between objects and relationships. Specifically, by decomposing the object features into class-generic and object-specific components, the causal graph framework is designed to analyze these existing SGG methods. To measure the influence of object-specific features for relationship recognition, we construct the counterfactual training framework for computing the difference between fact and counterfactual logits. Besides, to strengthen the role of object-specific features and learn the interaction between objects, a distribution matching loss is proposed to compute the KL divergence between counterfactual outputs and standard difference distributions and modulate the relations predictions. Finally, compared with the current state-of-the-art methods, the extensive experimental results on VG150 and VrR-VG datasets demonstrate the effectiveness and superiority of our proposed CFEN.
12 March 2026 in IEEE Transactions on Image Processing
Transfer subspace learning plays a critical role in unsupervised domain adaptation by establishing a shared embedding space where source domain data can be linearly reconstructed to match target domain distributions. While existing methods exploit the low-rank structures of reconstruction matrix, they frequently overlook the alignment of cross-domain joint probability distributions in the learned low-dimensional subspace. To address these challenges, we propose a novel non-convex transfer learning method named DATSL, which employs embedded distribution alignment. Our DATSL incorporates a non-convex regularizer to approximate low-rank constraints, capturing the complex characteristics of the rank function by minimizing top k smallest singular values of reconstruction matrix. To align the joint distributions across domains, a category-aware joint distribution alignment mechanism extracts more discriminative representations and enhances subspace discriminability through label-informed covariance matching. Besides, DATSL is extended to a graph-based variant GDATSL, which incorporates manifold-preserving constraints via Laplacian regularization to maintain intrinsic data topology during knowledge transfer. Furthermore, we develop an efficient iterative optimization algorithm to solve our formulated nonconvex minimization problems with proved convergence. Extensive experimental results on several public datasets demonstrate the effectiveness of our proposed methods in comparison to other state-of-the-art approaches.
11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Multi-view multi-label learning has become a research focus for describing objects with rich expressions and annotations. However, real-world data often contains numerous unlabeled instances, due to the high cost and technical limitations of manual labeling. This crucial problem involves three main challenges: i) How to extract advanced semantics from available views? ii) How to build a refined classification framework with limited labeled space? iii) How to provide more high-quality supervisory information? To address these problems, we propose a Semi-Supervised Multi-View Multi-Label Learning Method with View-Specific Transformer and Enhanced Pseudo-Label named SMVTEP. Specifically, Generative Adversarial Networks are employed to extract informative shared and specific representations and their consistency and distinctiveness are ensured through the adversarial mechanism and information theory based contrastive learning. Then we build specific classifiers for each extracted feature and apply instance-level manifold constraints to reduce bias across classifiers. Moreover, we design a transformer-style fusion approach that simultaneously captures the imbalance of expressive power among views, mapping effects on specific labels, and label dependencies by incorporating confidence scores and category semantics into the self-attention mechanism. Furthermore, after using Mixup for data augmentation, category-enhanced pseudo-labels are leveraged to improve the reliability of additional annotations by aligning the label distribution of unlabeled samples with the true distribution. Finally, extensive experimental results validate the effectiveness of SMVTEP against state-of-the-art methods.